tcpdump locks up kvm host for a while.

tcpdump locks up kvm host for a while.

[Robin Lee Powell]

I can essentially test this at will, so feel free to ask for
debugging steps.

My host and VMs are all Fedora 15.

Every time I run tcpdump on a VM, it hangs for a while.  It uses all
available CPU, "virsh list" hangs trying to show it, and I can't
shut it down except by killing the qemu-kvm process.

After a few minutes, which amount of time seems to be proportionate
to how long it's been since I last ran tcpdump (the longer it's
been, the longer I have to wait) everything goes back to normal if I
don't kill the VM.

Any idea what's going on there?

-Robin

Re: tcpdump locks up kvm host for a while.

[Robin Lee Powell]

On Tue, Sep 27, 2011 at 07:01:37PM -0700, Robin Lee Powell wrote:
> 
> I can essentially test this at will, so feel free to ask for
> debugging steps.
> 
> My host and VMs are all Fedora 15.
> 
> Every time I run tcpdump on a VM, it hangs for a while.  It uses
> all available CPU, "virsh list" hangs trying to show it, and I
> can't shut it down except by killing the qemu-kvm process.
> 
> After a few minutes, which amount of time seems to be
> proportionate to how long it's been since I last ran tcpdump (the
> longer it's been, the longer I have to wait) everything goes back
> to normal if I don't kill the VM.
> 
> Any idea what's going on there?

Nobody else has this problem?  How odd.

-Robin

Re: tcpdump locks up kvm host for a while.

[Nikola Ciprich]

[-- Attachment #1: Type: text/plain, Size: 819 bytes --]

Hello Robin,

actually I'm stil hitting problems with tcpdump in KVM virtuals..
what is the content of /sys/devices/system/clocksource/clocksource0/current_clocksource?
I guess it'll be kvm-clock, try using clocksource=tsc or hpet kernel parameter.
does it help?
n.


> 
> -Robin
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

-- 
-------------------------------------
Ing. Nikola CIPRICH
LinuxBox.cz, s.r.o.
28. rijna 168, 709 01 Ostrava

tel.:   +420 596 603 142
fax:    +420 596 621 273
mobil:  +420 777 093 799

www.linuxbox.cz

mobil servis: +420 737 238 656
email servis: servis@linuxbox.cz
-------------------------------------

[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]

Re: Re: tcpdump locks up kvm host for a while.

[Robin Lee Powell]

On Fri, Sep 30, 2011 at 09:03:21PM +0200, Nikola Ciprich wrote:
> Hello Robin,
> 
> actually I'm stil hitting problems with tcpdump in KVM virtuals..
> what is the content of /sys/devices/system/clocksource/clocksource0/current_clocksource?
> I guess it'll be kvm-clock, try using clocksource=tsc or hpet kernel parameter.
> does it help?

[    1.911056] Override clocksource tsc is not HRT compatible. Cannot switch while in HRT/NOHZ mode

Will hpet do any better?

-Robin

Re: Re: tcpdump locks up kvm host for a while.

[Nikola Ciprich]

[-- Attachment #1: Type: text/plain, Size: 642 bytes --]

well, give it a try ;)
I still haven't fully resolved this, but I'm sure it has to do something with
the timesource - with kvm-clock, it got much worse...
n.

> [    1.911056] Override clocksource tsc is not HRT compatible. Cannot switch while in HRT/NOHZ mode
> 
> Will hpet do any better?
> 
> -Robin
> 

-- 
-------------------------------------
Ing. Nikola CIPRICH
LinuxBox.cz, s.r.o.
28. rijna 168, 709 01 Ostrava

tel.:   +420 596 603 142
fax:    +420 596 621 273
mobil:  +420 777 093 799

www.linuxbox.cz

mobil servis: +420 737 238 656
email servis: servis@linuxbox.cz
-------------------------------------

[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]

Re: Re: [kvm] Re: tcpdump locks up kvm host for a while.

[Robin Lee Powell]

I'm afraid clocksource=hpet didn't help much; after about 24 hours
up, tcpdump hung the machine for for about 10 seconds.  That may or
may not be better than usual; I haven't been timing things
precisely.  My suspicion is that it's slightly better, but it's
still not great.

Also:

rlpowell@vrici> dmesg | grep -i clock
[    0.000000] Command line: ro root=/dev/vda2 rd_NO_LUKS rd_NO_LVM rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYTABLE=us console=ttyS0,115200 clocksource=hpet
[    0.000000] kvm-clock: Using msrs 12 and 11
[    0.000000] kvm-clock: cpu 0, msr 0:1b567c1, boot clock
[    0.000000] kvm-clock: cpu 0, msr 0:5fc137c1, primary cpu clock
[    0.000000] Kernel command line: ro root=/dev/vda2 rd_NO_LUKS rd_NO_LVM rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYTABLE=us console=ttyS0,115200 clocksource=hpet
[    0.000000] hpet clockevent registered
[    0.260136] Switching to clocksource hpet
[    1.009393] rtc_cmos 00:01: setting system clock to 2011-10-01 07:49:50 UTC (1317455390)
[    1.821905] Refined TSC clocksource calibration: 2394.030 MHz.
[60555.473797] Clocksource tsc unstable (delta = -42950328292 ns)

(note the last line)

-Robin


On Sat, Oct 01, 2011 at 09:36:43AM +0200, Nikola Ciprich wrote:
> well, give it a try ;)
> I still haven't fully resolved this, but I'm sure it has to do something with
> the timesource - with kvm-clock, it got much worse...
> n.
> 
> > [    1.911056] Override clocksource tsc is not HRT compatible. Cannot switch while in HRT/NOHZ mode
> > 
> > Will hpet do any better?
> > 
> > -Robin
> > 
> 
> -- 
> -------------------------------------
> Ing. Nikola CIPRICH
> LinuxBox.cz, s.r.o.
> 28. rijna 168, 709 01 Ostrava
> 
> tel.:   +420 596 603 142
> fax:    +420 596 621 273
> mobil:  +420 777 093 799
> 
> www.linuxbox.cz
> 
> mobil servis: +420 737 238 656
> email servis: servis@linuxbox.cz
> -------------------------------------

Re: Re: [kvm] Re: tcpdump locks up kvm host for a while.

[Avi Kivity]

On 10/02/2011 02:45 AM, Robin Lee Powell wrote:
> I'm afraid clocksource=hpet didn't help much; after about 24 hours
> up, tcpdump hung the machine for for about 10 seconds.  That may or
> may not be better than usual; I haven't been timing things
> precisely.  My suspicion is that it's slightly better, but it's
> still not great.
>
> Also:
>
> rlpowell@vrici>  dmesg | grep -i clock
> [    0.000000] Command line: ro root=/dev/vda2 rd_NO_LUKS rd_NO_LVM rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYTABLE=us console=ttyS0,115200 clocksource=hpet
> [    0.000000] kvm-clock: Using msrs 12 and 11
> [    0.000000] kvm-clock: cpu 0, msr 0:1b567c1, boot clock
> [    0.000000] kvm-clock: cpu 0, msr 0:5fc137c1, primary cpu clock
> [    0.000000] Kernel command line: ro root=/dev/vda2 rd_NO_LUKS rd_NO_LVM rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYTABLE=us console=ttyS0,115200 clocksource=hpet
> [    0.000000] hpet clockevent registered
> [    0.260136] Switching to clocksource hpet
> [    1.009393] rtc_cmos 00:01: setting system clock to 2011-10-01 07:49:50 UTC (1317455390)
> [    1.821905] Refined TSC clocksource calibration: 2394.030 MHz.
> [60555.473797] Clocksource tsc unstable (delta = -42950328292 ns)
>
> (note the last line)
>

Do you get a similar error with kvmclock?

-- 
error compiling committee.c: too many arguments to function

Re: Re: [kvm] Re: [kvm] Re: tcpdump locks up kvm host for a while.

[Robin Lee Powell]

On Sun, Oct 02, 2011 at 07:06:35PM +0200, Avi Kivity wrote:
> On 10/02/2011 02:45 AM, Robin Lee Powell wrote:
> >I'm afraid clocksource=hpet didn't help much; after about 24 hours
> >up, tcpdump hung the machine for for about 10 seconds.  That may or
> >may not be better than usual; I haven't been timing things
> >precisely.  My suspicion is that it's slightly better, but it's
> >still not great.
> >
> >Also:
> >
> >rlpowell@vrici>  dmesg | grep -i clock
> >[    0.000000] Command line: ro root=/dev/vda2 rd_NO_LUKS rd_NO_LVM rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYTABLE=us console=ttyS0,115200 clocksource=hpet
> >[    0.000000] kvm-clock: Using msrs 12 and 11
> >[    0.000000] kvm-clock: cpu 0, msr 0:1b567c1, boot clock
> >[    0.000000] kvm-clock: cpu 0, msr 0:5fc137c1, primary cpu clock
> >[    0.000000] Kernel command line: ro root=/dev/vda2 rd_NO_LUKS rd_NO_LVM rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYTABLE=us console=ttyS0,115200 clocksource=hpet
> >[    0.000000] hpet clockevent registered
> >[    0.260136] Switching to clocksource hpet
> >[    1.009393] rtc_cmos 00:01: setting system clock to 2011-10-01 07:49:50 UTC (1317455390)
> >[    1.821905] Refined TSC clocksource calibration: 2394.030 MHz.
> >[60555.473797] Clocksource tsc unstable (delta = -42950328292 ns)
> >
> >(note the last line)
> >
> 
> Do you get a similar error with kvmclock?

I don't think I did, no.  I'm not completely certain, but pretty
sure.

-Robin

Re: Re: [kvm] Re: [kvm] Re: tcpdump locks up kvm host for a while.

[Avi Kivity]

On 10/02/2011 07:07 PM, Robin Lee Powell wrote:
> On Sun, Oct 02, 2011 at 07:06:35PM +0200, Avi Kivity wrote:
> >  On 10/02/2011 02:45 AM, Robin Lee Powell wrote:
> >  >I'm afraid clocksource=hpet didn't help much; after about 24 hours
> >  >up, tcpdump hung the machine for for about 10 seconds.  That may or
> >  >may not be better than usual; I haven't been timing things
> >  >precisely.  My suspicion is that it's slightly better, but it's
> >  >still not great.
> >  >
> >  >Also:
> >  >
> >  >rlpowell@vrici>   dmesg | grep -i clock
> >  >[    0.000000] Command line: ro root=/dev/vda2 rd_NO_LUKS rd_NO_LVM rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYTABLE=us console=ttyS0,115200 clocksource=hpet
> >  >[    0.000000] kvm-clock: Using msrs 12 and 11
> >  >[    0.000000] kvm-clock: cpu 0, msr 0:1b567c1, boot clock
> >  >[    0.000000] kvm-clock: cpu 0, msr 0:5fc137c1, primary cpu clock
> >  >[    0.000000] Kernel command line: ro root=/dev/vda2 rd_NO_LUKS rd_NO_LVM rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYTABLE=us console=ttyS0,115200 clocksource=hpet
> >  >[    0.000000] hpet clockevent registered
> >  >[    0.260136] Switching to clocksource hpet
> >  >[    1.009393] rtc_cmos 00:01: setting system clock to 2011-10-01 07:49:50 UTC (1317455390)
> >  >[    1.821905] Refined TSC clocksource calibration: 2394.030 MHz.
> >  >[60555.473797] Clocksource tsc unstable (delta = -42950328292 ns)
> >  >
> >  >(note the last line)
> >  >
> >
> >  Do you get a similar error with kvmclock?
>
> I don't think I did, no.  I'm not completely certain, but pretty
> sure.
>

Then let's try to understand why it doesn't work with kvmclock.

-- 
error compiling committee.c: too many arguments to function

Re: Re: [kvm] Re: [kvm] Re: [kvm] Re: tcpdump locks up kvm host for a while.

[Robin Lee Powell]

On Sun, Oct 02, 2011 at 07:12:42PM +0200, Avi Kivity wrote:
> On 10/02/2011 07:07 PM, Robin Lee Powell wrote:
> >On Sun, Oct 02, 2011 at 07:06:35PM +0200, Avi Kivity wrote:
> >>  On 10/02/2011 02:45 AM, Robin Lee Powell wrote:
> >>  >I'm afraid clocksource=hpet didn't help much; after about 24 hours
> >>  >up, tcpdump hung the machine for for about 10 seconds.  That may or
> >>  >may not be better than usual; I haven't been timing things
> >>  >precisely.  My suspicion is that it's slightly better, but it's
> >>  >still not great.
> >>  >
> >>  >Also:
> >>  >
> >>  >rlpowell@vrici>   dmesg | grep -i clock
> >>  >[    0.000000] Command line: ro root=/dev/vda2 rd_NO_LUKS rd_NO_LVM rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYTABLE=us console=ttyS0,115200 clocksource=hpet
> >>  >[    0.000000] kvm-clock: Using msrs 12 and 11
> >>  >[    0.000000] kvm-clock: cpu 0, msr 0:1b567c1, boot clock
> >>  >[    0.000000] kvm-clock: cpu 0, msr 0:5fc137c1, primary cpu clock
> >>  >[    0.000000] Kernel command line: ro root=/dev/vda2 rd_NO_LUKS rd_NO_LVM rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYTABLE=us console=ttyS0,115200 clocksource=hpet
> >>  >[    0.000000] hpet clockevent registered
> >>  >[    0.260136] Switching to clocksource hpet
> >>  >[    1.009393] rtc_cmos 00:01: setting system clock to 2011-10-01 07:49:50 UTC (1317455390)
> >>  >[    1.821905] Refined TSC clocksource calibration: 2394.030 MHz.
> >>  >[60555.473797] Clocksource tsc unstable (delta = -42950328292 ns)
> >>  >
> >>  >(note the last line)
> >>  >
> >>
> >>  Do you get a similar error with kvmclock?
> >
> >I don't think I did, no.  I'm not completely certain, but pretty
> >sure.
> >
> 
> Then let's try to understand why it doesn't work with kvmclock.

I'd love to.  Suggestions?

-Robin

Re: Re: [kvm] Re: [kvm] Re: [kvm] Re: tcpdump locks up kvm host for a while.

[Avi Kivity]

On 10/02/2011 07:13 PM, Robin Lee Powell wrote:
> On Sun, Oct 02, 2011 at 07:12:42PM +0200, Avi Kivity wrote:
> >  On 10/02/2011 07:07 PM, Robin Lee Powell wrote:
> >  >On Sun, Oct 02, 2011 at 07:06:35PM +0200, Avi Kivity wrote:
> >  >>   On 10/02/2011 02:45 AM, Robin Lee Powell wrote:
> >  >>   >I'm afraid clocksource=hpet didn't help much; after about 24 hours
> >  >>   >up, tcpdump hung the machine for for about 10 seconds.  That may or
> >  >>   >may not be better than usual; I haven't been timing things
> >  >>   >precisely.  My suspicion is that it's slightly better, but it's
> >  >>   >still not great.
> >  >>   >
> >  >>   >Also:
> >  >>   >
> >  >>   >rlpowell@vrici>    dmesg | grep -i clock
> >  >>   >[    0.000000] Command line: ro root=/dev/vda2 rd_NO_LUKS rd_NO_LVM rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYTABLE=us console=ttyS0,115200 clocksource=hpet
> >  >>   >[    0.000000] kvm-clock: Using msrs 12 and 11
> >  >>   >[    0.000000] kvm-clock: cpu 0, msr 0:1b567c1, boot clock
> >  >>   >[    0.000000] kvm-clock: cpu 0, msr 0:5fc137c1, primary cpu clock
> >  >>   >[    0.000000] Kernel command line: ro root=/dev/vda2 rd_NO_LUKS rd_NO_LVM rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYTABLE=us console=ttyS0,115200 clocksource=hpet
> >  >>   >[    0.000000] hpet clockevent registered
> >  >>   >[    0.260136] Switching to clocksource hpet
> >  >>   >[    1.009393] rtc_cmos 00:01: setting system clock to 2011-10-01 07:49:50 UTC (1317455390)
> >  >>   >[    1.821905] Refined TSC clocksource calibration: 2394.030 MHz.
> >  >>   >[60555.473797] Clocksource tsc unstable (delta = -42950328292 ns)
> >  >>   >
> >  >>   >(note the last line)
> >  >>   >
> >  >>
> >  >>   Do you get a similar error with kvmclock?
> >  >
> >  >I don't think I did, no.  I'm not completely certain, but pretty
> >  >sure.
> >  >
> >
> >  Then let's try to understand why it doesn't work with kvmclock.
>
> I'd love to.  Suggestions?
>

Well, any messages?  Alternatively, detailed instructions to reproduce?

-- 
error compiling committee.c: too many arguments to function

Re: Re: [kvm] Re: [kvm] Re: [kvm] Re: [kvm] Re: tcpdump locks up kvm host for a while.

[Robin Lee Powell]

On Sun, Oct 02, 2011 at 07:21:08PM +0200, Avi Kivity wrote:
> On 10/02/2011 07:13 PM, Robin Lee Powell wrote:
> >On Sun, Oct 02, 2011 at 07:12:42PM +0200, Avi Kivity wrote:
> >>  On 10/02/2011 07:07 PM, Robin Lee Powell wrote:
> >>  >On Sun, Oct 02, 2011 at 07:06:35PM +0200, Avi Kivity wrote:
> >>  >>   On 10/02/2011 02:45 AM, Robin Lee Powell wrote:
> >>  >>   >I'm afraid clocksource=hpet didn't help much; after about 24 hours
> >>  >>   >up, tcpdump hung the machine for for about 10 seconds.  That may or
> >>  >>   >may not be better than usual; I haven't been timing things
> >>  >>   >precisely.  My suspicion is that it's slightly better, but it's
> >>  >>   >still not great.
> >>  >>   >
> >>  >>   >Also:
> >>  >>   >
> >>  >>   >rlpowell@vrici>    dmesg | grep -i clock
> >>  >>   >[    0.000000] Command line: ro root=/dev/vda2 rd_NO_LUKS rd_NO_LVM rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYTABLE=us console=ttyS0,115200 clocksource=hpet
> >>  >>   >[    0.000000] kvm-clock: Using msrs 12 and 11
> >>  >>   >[    0.000000] kvm-clock: cpu 0, msr 0:1b567c1, boot clock
> >>  >>   >[    0.000000] kvm-clock: cpu 0, msr 0:5fc137c1, primary cpu clock
> >>  >>   >[    0.000000] Kernel command line: ro root=/dev/vda2 rd_NO_LUKS rd_NO_LVM rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYTABLE=us console=ttyS0,115200 clocksource=hpet
> >>  >>   >[    0.000000] hpet clockevent registered
> >>  >>   >[    0.260136] Switching to clocksource hpet
> >>  >>   >[    1.009393] rtc_cmos 00:01: setting system clock to 2011-10-01 07:49:50 UTC (1317455390)
> >>  >>   >[    1.821905] Refined TSC clocksource calibration: 2394.030 MHz.
> >>  >>   >[60555.473797] Clocksource tsc unstable (delta = -42950328292 ns)
> >>  >>   >
> >>  >>   >(note the last line)
> >>  >>   >
> >>  >>
> >>  >>   Do you get a similar error with kvmclock?
> >>  >
> >>  >I don't think I did, no.  I'm not completely certain, but pretty
> >>  >sure.
> >>  >
> >>
> >>  Then let's try to understand why it doesn't work with kvmclock.
> >
> >I'd love to.  Suggestions?
> >
> 
> Well, any messages?  

None I can find.  I'll try again later today and double check.

> Alternatively, detailed instructions to reproduce?

Start a VM.  Wait a few days.  Run tcpdump.  The system locks up for
30+ seconds.

That's all I have.  There are, of course, any number of details that
make my VMs different from yours, but they're basically straight
Fedora 15.  I could give arbitrary amounts of detail, but I've no
idea what to concentrate on.

-Robin

Re: Re: [kvm] Re: [kvm] Re: [kvm] Re: [kvm] Re: tcpdump locks up kvm host for a while.

[Avi Kivity]

> > 
> > Well, any messages?
> 
> None I can find.  I'll try again later today and double check.


Please do.


> 
> > Alternatively, detailed instructions to reproduce?
> 
> Start a VM.  Wait a few days.  Run tcpdump.  The system locks up for
> 30+ seconds.

How do you detect the lockup?  Are you at the console when this happens and everything just freezes?  The entire host, right, not guests?

Is the lockup immediate after starting tcpdump?


> 
> That's all I have.  There are, of course, any number of details that
> make my VMs different from yours, but they're basically straight
> Fedora 15.  I could give arbitrary amounts of detail, but I've no
> idea what to concentrate on.
> 

host /proc/cpuinfo
qemu command line

Re: Re: [kvm] Re: [kvm] Re: [kvm] Re: [kvm] Re: [kvm] Re: tcpdump locks up kvm host for a while.

[Robin Lee Powell]

On Sun, Oct 02, 2011 at 03:38:29PM -0400, Avi Kivity wrote:
> 
> > > 
> > > Well, any messages?
> > 
> > None I can find.  I'll try again later today and double check.
> 
> 
> Please do.

Oct  2 19:53:01 vrici dbus: [system] Activating service name='net.reactivated.Fprint' (using servicehelper)
Oct  2 19:53:01 vrici dbus: [system] Successfully activated service 'net.reactivated.Fprint'
Oct  2 19:53:55 vrici kernel: [155001.460802] device eth0 entered promiscuous mode
Oct  2 19:53:56 vrici kernel: [155002.367313] device eth0 left promiscuous mode

type=SYSCALL msg=audit(10/02/2011 19:53:14.731:48050) : arch=x86_64 syscall=execve success=yes exit=0 a0=7fff41959fd0 a1=7fc2f48d5150 a2=1474800 a3=6a6222 items=3 ppid=18453 pid=18456 auid=rlpowell uid=rlpowell gid=rlpowell euid=rlpowell suid=rlpowell fsuid=rlpowell egid=rlpowell sgid=rlpowell fsgid=rlpowell tty=pts0 ses=4 comm=date exe=/bin/date subj=staff_u:staff_r:staff_t:s0 key=64bit_execs
----
type=SYSCALL msg=audit(10/02/2011 19:53:56.454:48059) : arch=x86_64 syscall=close success=yes exit=0 a0=3 a1=0 a2=1c11220 a3=7fff2eb675b0 items=0 ppid=18457 pid=18458 auid=rlpowell uid=tcpdump gid=tcpdump euid=tcpdump suid=tcpdump fsuid=tcpdump egid=tcpdump sgid=tcpdump fsgid=tcpdump tty=pts0 ses=4 comm=tcpdump exe=/usr/sbin/tcpdump subj=staff_u:unconfined_r:unconfined_t:s0 key=(null)
type=ANOM_PROMISCUOUS msg=audit(10/02/2011 19:53:56.454:48059) : dev=eth0 prom=no old_prom=yes auid=rlpowell uid=tcpdump gid=tcpdump ses=4

That's all I could find that's even remotely interesting.

> > > Alternatively, detailed instructions to reproduce?
> > 
> > Start a VM.  Wait a few days.  Run tcpdump.  The system locks up
> > for 30+ seconds.
> 
> How do you detect the lockup?  Are you at the console when this
> happens and everything just freezes?  The entire host, right, not
> guests?
> 
> Is the lockup immediate after starting tcpdump?

Ooooh.  Oh, we're on the wrong page entirely.  This *only* happens
on the guests.  tcpdump on the master host causes no trouble of any
kind whatsoever.

I guess I never specified that, maybe?  Shame on me, if so.

If it was the *host*, I wouldn't be complaining on the KVM list,
though, I'd be complaining on the main kernel list.  :)

When I run tcpdump on a *guest*, the entire guest completely freezes
up; no response even to hitting enter on the console.  "virsh list"
also locks up whenever it tries to print state about that VM (but
the others work fine), as does any other operation that touches the
state of that VM.  The VM takes up 100% of CPU on one core while
this is happening.  Eventually it gets better.

> > That's all I have.  There are, of course, any number of details that
> > make my VMs different from yours, but they're basically straight
> > Fedora 15.  I could give arbitrary amounts of detail, but I've no
> > idea what to concentrate on.
> > 
> 
> host /proc/cpuinfo

processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 15
model name      : Intel(R) Core(TM)2 CPU          6600  @ 2.40GHz
stepping        : 6
cpu MHz         : 2393.999
cache size      : 4096 KB
physical id     : 0
siblings        : 2
core id         : 0
cpu cores       : 2
apicid          : 0
initial apicid  : 0
fpu             : yes
fpu_exception   : yes
cpuid level     : 10
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc arch_perfmon pebs bts rep_good nopl aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm lahf_lm dts tpr_shadow
bogomips        : 4787.99
clflush size    : 64
cache_alignment : 64
address sizes   : 36 bits physical, 48 bits virtual
power management:

processor       : 1
vendor_id       : GenuineIntel
cpu family      : 6
model           : 15
model name      : Intel(R) Core(TM)2 CPU          6600  @ 2.40GHz
stepping        : 6
cpu MHz         : 2393.999
cache size      : 4096 KB
physical id     : 0
siblings        : 2
core id         : 1
cpu cores       : 2
apicid          : 1
initial apicid  : 1
fpu             : yes
fpu_exception   : yes
cpuid level     : 10
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc arch_perfmon pebs bts rep_good nopl aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm lahf_lm dts tpr_shadow
bogomips        : 4787.78
clflush size    : 64
cache_alignment : 64
address sizes   : 36 bits physical, 48 bits virtual
power management:


> qemu command line

qemu     11811     1  1 Oct01 ?        00:42:13 /usr/bin/qemu-kvm -S -M pc-0.14 -enable-kvm -m 1536 -smp 1,sockets=1,cores=1,threads=1 -name vrici -uuid 20fc34fc-438e-3f51-d53e-406a68f96cbc -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/vrici.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -boot c -drive file=/dev/mapper/local-vrici_root,if=none,id=drive-virtio-disk0,boot=on,format=raw,cache=none -device virtio-blk-pci,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0 -drive file=/dev/mapper/local-vrici_srv,if=none,id=drive-virtio-disk1,format=raw,cache=none -device virtio-blk-pci,bus=pci.0,addr=0x5,drive=drive-virtio-disk1,id=virtio-disk1 -drive file=/dev/mapper/local-vrici_swap,if=none,id=drive-virtio-disk
 2,format=raw,cache=none -device virtio-blk-pci,bus=pci.0,addr=0x6,drive=drive-virtio-disk2,id=virtio-disk2 -drive file=/dev/mapper/local-vrici_home,if=none,id=drive-virtio-disk3,format=raw,cache=none -device virtio-blk-pci,bus=pci.0,addr=0x8,drive=drive-virtio-disk3,id=virtio-disk3 -netdev tap,fd=32,id=hostnet0 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:34:91:67,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -usb -device usb-tablet,id=input0 -vnc 0.0.0.0:1 -vga cirrus -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x7

-Robin

Re: Re: [kvm] Re: [kvm] Re: [kvm] Re: [kvm] Re: [kvm] Re: tcpdump locks up kvm host for a while.

[Nikola Ciprich]

[-- Attachment #1: Type: text/plain, Size: 656 bytes --]

> When I run tcpdump on a *guest*, the entire guest completely freezes
> up; no response even to hitting enter on the console.  "virsh list"
> also locks up whenever it tries to print state about that VM (but
> the others work fine), as does any other operation that touches the
> state of that VM.  The VM takes up 100% of CPU on one core while
> this is happening.  Eventually it gets better.
> 
exactly the same I was describing here some time ago, with the difference
I get a lot of ugly backtraces and sometimes the guest doesn't get to usable state
at all (filesystems switches to read only due to errors or the whole guest locks up)

n.

[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]

Re: Re: [kvm] Re: [kvm] Re: [kvm] Re: [kvm] Re: [kvm] Re: tcpdump locks up kvm host for a while.

[Avi Kivity]

On 10/03/2011 05:03 AM, Robin Lee Powell wrote:
> >  >  >  Alternatively, detailed instructions to reproduce?
> >  >
> >  >  Start a VM.  Wait a few days.  Run tcpdump.  The system locks up
> >  >  for 30+ seconds.
> >
> >  How do you detect the lockup?  Are you at the console when this
> >  happens and everything just freezes?  The entire host, right, not
> >  guests?
> >
> >  Is the lockup immediate after starting tcpdump?
>
> Ooooh.  Oh, we're on the wrong page entirely.  This *only* happens
> on the guests.  tcpdump on the master host causes no trouble of any
> kind whatsoever.
>
> I guess I never specified that, maybe?  Shame on me, if so.

Well, the subject line does say "kvm host".

> If it was the *host*, I wouldn't be complaining on the KVM list,
> though, I'd be complaining on the main kernel list.  :)

Be sure to copy kvm@ too if it happens - certainly kvm bugs can affect 
the host.

>
> When I run tcpdump on a *guest*, the entire guest completely freezes
> up; no response even to hitting enter on the console.  "virsh list"
> also locks up whenever it tries to print state about that VM (but
> the others work fine), as does any other operation that touches the
> state of that VM.  The VM takes up 100% of CPU on one core while
> this is happening.  Eventually it gets better.

You can use 'perf kvm' to figure out where the guest is spinning.

>
> >  >  That's all I have.  There are, of course, any number of details that
> >  >  make my VMs different from yours, but they're basically straight
> >  >  Fedora 15.  I could give arbitrary amounts of detail, but I've no
> >  >  idea what to concentrate on.
> >  >
> >
> >  host /proc/cpuinfo
>
> model name      : Intel(R) Core(TM)2 CPU          6600  @ 2.40GHz
> flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc arch_perfmon pebs bts rep_good nopl aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm lahf_lm dts tpr_shadow

constant_tsc indicates tsc should be good.


>
> >  qemu command line
>
> qemu     11811     1  1 Oct01 ?        00:42:13 /usr/bin/qemu-kvm -S -M pc-0.14 -enable-kvm -m 1536 -smp 1,sockets=1,cores=1,threads=1 -name vrici -uuid 20fc34fc-438e-3f51-d53e-406a68f96cbc -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/vrici.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -boot c -drive file=/dev/mapper/local-vrici_root,if=none,id=drive-virtio-disk0,boot=on,format=raw,cache=none -device virtio-blk-pci,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0 -drive file=/dev/mapper/local-vrici_srv,if=none,id=drive-virtio-disk1,format=raw,cache=none -device virtio-blk-pci,bus=pci.0,addr=0x5,drive=drive-virtio-disk1,id=virtio-disk1 -drive file=/dev/mapper/local-vrici_swap,if=none,id=drive-virtio-di
 sk2,format=raw,cache=none -device virtio-blk-pci,bus=pci.0,addr=0x6,drive=drive-virtio-disk2,id=virtio-disk2 -drive file=/dev/mapper/local-vrici_home,if=none,id=drive-virtio-disk3,format=raw,cache=none -device virtio-blk-pci,bus=pci.0,addr=0x8,drive=drive-virtio-disk3,id=virtio-disk3 -netdev tap,fd=32,id=hostnet0 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:34:91:67,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -usb -device usb-tablet,id=input0 -vnc 0.0.0.0:1 -vga cirrus -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x7
>

Nothing suspicious here.

-- 
error compiling committee.c: too many arguments to function

Re: tcpdump locks up kvm host for a while.

[Robin Lee Powell]

On Tue, Oct 04, 2011 at 08:15:12PM +0200, Avi Kivity wrote:
> On 10/03/2011 05:03 AM, Robin Lee Powell wrote:
> >>  >  >  Alternatively, detailed instructions to reproduce?
> >>  >
> >>  >  Start a VM.  Wait a few days.  Run tcpdump.  The system
> >>  >  locks up for 30+ seconds.
> >>
> >>  How do you detect the lockup?  Are you at the console when
> >>  this happens and everything just freezes?  The entire host,
> >>  right, not guests?
> >>
> >>  Is the lockup immediate after starting tcpdump?
> >
> >Ooooh.  Oh, we're on the wrong page entirely.  This *only*
> >happens on the guests.  tcpdump on the master host causes no
> >trouble of any kind whatsoever.
> >
> >I guess I never specified that, maybe?  Shame on me, if so.
> 
> Well, the subject line does say "kvm host".

*mega-face-palm*

(the subject line is also filled with crap of my own devising; I
think I've fixed the script in question)

> >When I run tcpdump on a *guest*, the entire guest completely
> >freezes up; no response even to hitting enter on the console.
> >"virsh list" also locks up whenever it tries to print state about
> >that VM (but the others work fine), as does any other operation
> >that touches the state of that VM.  The VM takes up 100% of CPU
> >on one core while this is happening.  Eventually it gets better.
> 
> You can use 'perf kvm' to figure out where the guest is spinning.

OK, gathered with:

sudo  perf kvm --guest --host record -o /tmp/kvm_perf -a

I don't know how to read it at all, so it's at
http://users.digitalkingdom.org/~rlpowell/media/public/kvm_perf

> >>  >  That's all I have.  There are, of course, any number of details that
> >>  >  make my VMs different from yours, but they're basically straight
> >>  >  Fedora 15.  I could give arbitrary amounts of detail, but I've no
> >>  >  idea what to concentrate on.
> >>  >
> >>
> >>  host /proc/cpuinfo
> >
> >model name      : Intel(R) Core(TM)2 CPU          6600  @ 2.40GHz
> >flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc arch_perfmon pebs bts rep_good nopl aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm lahf_lm dts tpr_shadow
> 
> constant_tsc indicates tsc should be good.

Yeah, that's what hte host is on.

-Robin

Re: tcpdump locks up kvm host for a while.

[Avi Kivity]

On 10/04/2011 09:40 PM, Robin Lee Powell wrote:
> >  >When I run tcpdump on a *guest*, the entire guest completely
> >  >freezes up; no response even to hitting enter on the console.
> >  >"virsh list" also locks up whenever it tries to print state about
> >  >that VM (but the others work fine), as does any other operation
> >  >that touches the state of that VM.  The VM takes up 100% of CPU
> >  >on one core while this is happening.  Eventually it gets better.
> >
> >  You can use 'perf kvm' to figure out where the guest is spinning.
>
> OK, gathered with:
>
> sudo  perf kvm --guest --host record -o /tmp/kvm_perf -a
>
> I don't know how to read it at all, so it's at
> http://users.digitalkingdom.org/~rlpowell/media/public/kvm_perf
>

Not accessible.

Please post the output of 'perf kvm report > log'.

-- 
error compiling committee.c: too many arguments to function

Re: Re: tcpdump locks up kvm host for a while.

[Robin Lee Powell]

On Wed, Oct 05, 2011 at 11:36:53AM -0700, Robin Lee Powell wrote:
> On Wed, Oct 05, 2011 at 08:21:53PM +0200, Avi Kivity wrote:
> > On 10/04/2011 09:40 PM, Robin Lee Powell wrote:
> > >>  >When I run tcpdump on a *guest*, the entire guest completely
> > >>  >freezes up; no response even to hitting enter on the console.
> > >>  >"virsh list" also locks up whenever it tries to print state
> > >>  >about that VM (but the others work fine), as does any other
> > >>  >operation that touches the state of that VM.  The VM takes up
> > >>  >100% of CPU on one core while this is happening.  Eventually
> > >>  >it gets better.
> > >>
> > >>  You can use 'perf kvm' to figure out where the guest is
> > >>  spinning.
> > >
> > >OK, gathered with:
> > >
> > >sudo  perf kvm --guest --host record -o /tmp/kvm_perf -a
> > >
> > >I don't know how to read it at all, so it's at
> > >http://users.digitalkingdom.org/~rlpowell/media/public/kvm_perf
> > >
> > 
> > Not accessible.
> 
> -_-  Fixed.
> 
> > Please post the output of 'perf kvm report > log'.
> 
> # Events: 42K
> #
> # Overhead   Command     Shared Object  Symbol
> # ........  ........  ................  ......
> #
>    100.00%  qemu-kvm  [unknown]         [g] 0xffffffff8111c7a5
> 
> 
> #
> # (For a higher level overview, try: perf report --sort comm,dso)
> #
> 
> How helpful is that?  -_-
> 
> I'm guessing I need --guestkallsyms= ; since they're all the same
> kernel I thought it'd figure it out.  I'll redo.

OK, here's a "better" version.

# Events: 46K cycles
#
# Overhead   Command            Shared Object                   Symbol
# ........  ........  .......................  .......................
#
    74.81%  qemu-kvm  [unknown]                [u] 0x7fbdffd4c18a
    25.14%  qemu-kvm  [guest.kernel.kallsyms]  [g] 0xffffffff811112f0
     0.03%  qemu-kvm  [virtio_net]             [g] 0x83e8
     0.01%  qemu-kvm  [virtio_balloon]         [g] 0x103b
     0.00%  qemu-kvm  [ip6_tables]             [g] compat_standard_to_user
     0.00%  qemu-kvm  [ipv6]                   [g] icmpv6_send
     0.00%  qemu-kvm  [virtio_blk]             [g] 0x7783
     0.00%  qemu-kvm  [ipv6]                   [g] raw6_seq_show
     0.00%  qemu-kvm  [ipv6]                   [g] icmpv6_rcv
     0.00%  qemu-kvm  [virtio_net]             [g] fini
     0.00%  qemu-kvm  [ip6table_filter]        [g] 0x9b5


#
# (For a higher level overview, try: perf report --sort comm,dso)
#

The file is also updated.

-Robin

Re: Re: tcpdump locks up kvm host for a while.

[Avi Kivity]

On 10/05/2011 10:29 PM, Robin Lee Powell wrote:
> >
> >  #
> >  # (For a higher level overview, try: perf report --sort comm,dso)
> >  #
> >
> >  How helpful is that?  -_-
> >
> >  I'm guessing I need --guestkallsyms= ; since they're all the same
> >  kernel I thought it'd figure it out.  I'll redo.
>
> OK, here's a "better" version.
>
> # Events: 46K cycles
> #
> # Overhead   Command            Shared Object                   Symbol
> # ........  ........  .......................  .......................
> #
>      74.81%  qemu-kvm  [unknown]                [u] 0x7fbdffd4c18a

This is in userspace, so it seems the guest wasn't completely stuck.

Try 'top -b' inside the guest to record what happens, let's see what 
processes this is and go from there.

>      25.14%  qemu-kvm  [guest.kernel.kallsyms]  [g] 0xffffffff811112f0

This doesn't resolve, please make sure the kernel-debuginfo package is 
installed in the guest and use the guestmount option.  (or you can 
install it in the host, I think)


-- 
error compiling committee.c: too many arguments to function