kvm causing memory corruption? ~2.6.25-rc6

kvm causing memory corruption? ~2.6.25-rc6

[Dave Hansen]

I was getting some kvm userspace crashes trying to run a Windows guest.
So, I decided to try a recent kernel (2.6.25-rc6-00333-ga4083c9)  with
the kvm kernel code that shipped with that kernel.

I've had some lockups doing similar things over the last month or two,
but figured it was something really stupid I was doing, and never really
connected the dots.  Now, I've hooked up a serial console and reproduced
it with a fresh boot and not much else going on at all on the machine.

Machine is a Thinkpad T61.  .config is here:

http://sr71.net/~dave/linux/config-2.6.25-rc6-00333-ga4083c9

To trigger it, I first run kvm and see an error (-no-kvm works fine,
btw):

$ ~/src/kvm-userspace/qemu/x86_64-softmmu/qemu-system-x86_64 -hda ~/projects/qemu/windows-xp-base-runme.img 
kvm_run: Cannot allocate memory
kvm_run returned -12

Then, run it again.  I usually get an oops.  But, the weird part is that
the oops isn't *in* kvm.  It's in some other part of the kernel and in
some *OTHER* process.  One in bash is below.  That's what leads me to
believe it is memory corruption.  The machine also becomes increasingly
unstable after the original oops so there's definitely collateral
damage.
        
        $ addr2line -e vmlinux c01795e4
        /home/dave/kernels/linux-2.6.git/mm/filemap.c:1327
        
        int filemap_fault(struct vm_area_struct *vma, struct vm_fault
        *vmf)
        {
                int error;
                struct file *file = vma->vm_file;
                struct address_space *mapping = file->f_mapping;
                struct file_ra_state *ra = &file->f_ra;
        HERE--->struct inode *inode = mapping->host;

Which is a line of code that literally hasn't touched since the
beginning of time (in git terms :).  Full oops is below:

[  435.057922] BUG: unable to handle kernel NULL pointer dereference at 00000048
[  435.067275] IP: [<c01795e4>] filemap_fault+0x34/0x310
[  435.072815] *pdpt = 000000002a4a7001 *pde = 0000000000000000 
[  435.081272] Oops: 0000 [#2] SMP 
[  435.084812] Modules linked in: nls_iso8859_1 vfat fat rfcomm l2cap tun ppdev acpi_cpufreq cpufreq_ondemand cpufreq_conservative cpufreq_stats freq_table cpufreq_userspace cpufreq_powersave sbs container sbshc af_packet sbp2 lp loop usb_storage arc4 ecb crypto_blkcipher pcmcia usbhid libusual hid snd_hda_intel snd_pcm_oss snd_mixer_oss snd_pcm snd_seq_dummy snd_seq_oss snd_seq_midi_event snd_seq snd_timer snd_seq_device joydev iwl4965 snd serio_raw mac80211 yenta_socket parport_pc sdhci uhci_hcd ehci_hcd ricoh_mmc ohci1394 rsrc_nonstatic soundcore cfg80211 parport psmouse mmc_core ieee1394 pcmcia_core usbcore snd_page_alloc e1000 button thinkpad_acpi nvram evdev thermal processor fan fuse
[  435.084812] 
[  435.084812] Pid: 7691, comm: bash Tainted: G      D  (2.6.25-rc6-00333-ga4083c9 #144)
[  435.084812] EIP: 0060:[<c01795e4>] EFLAGS: 00010286 CPU: 0
[  435.084812] EIP is at filemap_fault+0x34/0x310
[  435.084812] EAX: ef83bf48 EBX: 00000012 ECX: 00000000 EDX: ef83c7e8
[  435.084812] ESI: c04cc248 EDI: 00000000 EBP: ef96ee40 ESP: ef96ee00
[  435.084812]  DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
[  435.084812] Process bash (pid: 7691, ti=ef96e000 task=ef8a2e00 task.ti=ef96e000)
[  435.084812] Stack: ef96ee2c c0130cbc ef96ee28 c01870bb ef96ee28 00000000 00000000 00000000 
[  435.084812]        ef83bf48 ef83c7e8 ef83bf00 ef96ee9c ea49f7e8 00000012 c04cc248 00000000 
[  435.084812]        ef96eeb8 c018ab57 80000001 00000001 00000001 00000000 00000000 eacb6314 
[  435.084812] Call Trace:
[  435.084812]  [<c0130cbc>] ? kmap_atomic_prot+0x12c/0x150
[  435.084812]  [<c01870bb>] ? vm_normal_page+0x2b/0xa0
[  435.084812]  [<c018ab57>] ? __do_fault+0x67/0x4e0
[  435.084812]  [<c01a8a70>] ? pipe_read+0x1f0/0x290
[  435.084812]  [<c018b03d>] ? do_linear_fault+0x6d/0x80
[  435.084812]  [<c018b570>] ? handle_mm_fault+0x1c0/0x4d0
[  435.084812]  [<c014d58e>] ? do_sigaction+0x16e/0x190
[  435.084812]  [<c03b3419>] ? do_page_fault+0x169/0x4d0
[  435.084812]  [<c01a38b9>] ? fput+0x19/0x20
[  435.084812]  [<c03b32b0>] ? do_page_fault+0x0/0x4d0
[  435.084812]  [<c03b187a>] ? error_code+0x72/0x78
[  435.084812]  [<c03b0000>] ? wait_for_completion_killable+0x10/0x30
[  435.084812]  =======================
[  435.084812] Code: 89 45 f0 89 55 ec 8b 40 4c 89 45 e8 8b 50 7c 83 c0 48 89 45 e0 89 55 e4 8b 0a c7 45 d8 00 00 00 00 c7 45 d4 00 00 00 00 89 4d dc <8b> 49 48 89 f6 8d bc 27 00 00 00 00 89 c8 8b 7d dc 8b 5f 40 8b 
[  435.084812] EIP: [<c01795e4>] filemap_fault+0x34/0x310 SS:ESP 0068:ef96ee00
[  435.084870] ---[ end trace addcd60623916614 ]---

~/src/kvm-userspace$ git describe
kvm-63-118-g52be1a1

/proc/cpuinfo:

processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 15
model name      : Intel(R) Core(TM)2 Duo CPU     T7300  @ 2.00GHz
stepping        : 10
cpu MHz         : 800.000
cache size      : 4096 KB
physical id     : 0
siblings        : 2
core id         : 0
cpu cores       : 2
fdiv_bug        : no
hlt_bug         : no
f00f_bug        : no
coma_bug        : no
fpu             : yes
fpu_exception   : yes
cpuid level     : 10
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge
mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe nx lm
constant_tsc arch_perfmon pebs bts pni monitor ds_cpl vmx est tm2 ssse3
cx16 xtpr lahf_lm ida
bogomips        : 3996.38
clflush size    : 64
processor       : 1
vendor_id       : GenuineIntel
cpu family      : 6
model           : 15
model name      : Intel(R) Core(TM)2 Duo CPU     T7300  @ 2.00GHz
stepping        : 10
cpu MHz         : 800.000
cache size      : 4096 KB
physical id     : 0
siblings        : 2
core id         : 1
cpu cores       : 2
fdiv_bug        : no
hlt_bug         : no
f00f_bug        : no
coma_bug        : no
fpu             : yes
fpu_exception   : yes
cpuid level     : 10
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge
mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe nx lm
constant_tsc arch_perfmon pebs bts pni monitor ds_cpl vmx est tm2 ssse3
cx16 xtpr lahf_lm ida
bogomips        : 3990.03
clflush size    : 64



-- Dave

Re: [kvm-devel] kvm causing memory corruption? ~2.6.25-rc6

[Avi Kivity]

Dave Hansen wrote:
> I was getting some kvm userspace crashes trying to run a Windows guest.
> So, I decided to try a recent kernel (2.6.25-rc6-00333-ga4083c9)  with
> the kvm kernel code that shipped with that kernel.
>
>   

This is fixed in 2.6.25-rc7.

-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.

Re: [kvm-devel] kvm causing memory corruption? ~2.6.25-rc6

[Dave Hansen]

On Wed, 2008-03-26 at 11:50 +0200, Avi Kivity wrote:
> Dave Hansen wrote:
> > I was getting some kvm userspace crashes trying to run a Windows guest.
> > So, I decided to try a recent kernel (2.6.25-rc6-00333-ga4083c9)  with
> > the kvm kernel code that shipped with that kernel.
> >
> 
> This is fixed in 2.6.25-rc7.

I just updated to -rc7 and re-tested.  Same symptoms:

[  751.033545] BUG: unable to handle kernel paging request at 0096b848
[  751.040082] IP: [<c01a0636>] d_instantiate+0x26/0x50
[  751.048065] Oops: 0002 [#1] SMP
[  751.052057] Modules linked in: kvm_intel kvm nls_iso8859_1 vfat fat rfcomm l2cap tun ppdev acpi_cpufreq cpufreq_ondemand cpe
[  751.052057]
[  751.052057] Pid: 8743, comm: evolution Not tainted (2.6.25-rc7 #146)
[  751.052057] EIP: 0060:[<c01a0636>] EFLAGS: 00210286 CPU: 0
[  751.052057] EIP is at d_instantiate+0x26/0x50
[  751.052057] EAX: 0096b844 EBX: e65d7d48 ECX: 00000000 EDX: e65d7d60
[  751.052057] ESI: e67a7d00 EDI: e67a7cc0 EBP: e802ce48 ESP: e802ce3c
[  751.052057]  DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
[  751.052057] Process evolution (pid: 8743, ti=e802c000 task=f3c8ce00 task.ti=e802c000)
[  751.052057] Stack: e65d7d48 00000000 f4c191f8 e802ce60 c01e2fa4 e67a7cc0 f4c191f8 e65d7d48
[  751.052057]        e660c280 e802ce80 c01e30c3 00008180 e67a7cc0 00000000 c03b4a00 e660c280
[  751.052057]        00008180 e802cea0 c0197964 e802cf24 c03b4a00 e67a7cc0 e660c280 e802cf24
[  751.052057] Call Trace:
[  751.052057]  [<c01e2fa4>] ? ext3_add_nondir+0x34/0x60
[  751.052057]  [<c01e30c3>] ? ext3_create+0xf3/0x100
[  751.052057]  [<c0197964>] ? vfs_create+0x74/0x100
[  751.052057]  [<c0197c8f>] ? open_namei_create+0x4f/0xa0
[  751.052057]  [<c01981f3>] ? open_namei+0x513/0x560
[  751.052057]  [<c018db2c>] ? do_filp_open+0x2c/0x60
[  751.052057]  [<c018dd29>] ? get_unused_fd_flags+0x39/0xd0
[  751.052057]  [<c018dec4>] ? do_sys_open+0x54/0xe0
[  751.052057]  [<c018df6c>] ? sys_open+0x1c/0x20
[  751.052057]  [<c0104e2c>] ? sysenter_past_esp+0x6d/0xa5
[  751.052057]  [<c0390000>] ? quirk_vt8235_acpi+0x90/0xa0
[  751.052057]  =======================
[  751.052057] Code: 27 00 00 00 00 55 89 e5 57 89 c7 56 8d 70 40 53
89 d3 39 70 40 75 37 b8 40 15 4e c0 e8 14 d1 1f 00 85 db
[  751.052057] EIP: [<c01a0636>] d_instantiate+0x26/0x50 SS:ESP 0068:e802ce3c
[  751.052103] ---[ end trace 514c1de750400319 ]---


-- Dave

Re: [kvm-devel] kvm causing memory corruption? ~2.6.25-rc6

[Avi Kivity]

Dave Hansen wrote:
> On Wed, 2008-03-26 at 11:50 +0200, Avi Kivity wrote:
>   
>> Dave Hansen wrote:
>>     
>>> I was getting some kvm userspace crashes trying to run a Windows guest.
>>> So, I decided to try a recent kernel (2.6.25-rc6-00333-ga4083c9)  with
>>> the kvm kernel code that shipped with that kernel.
>>>
>>>       
>> This is fixed in 2.6.25-rc7.
>>     
>
> I just updated to -rc7 and re-tested.  Same symptoms:
>   

Bad.  Which kvm userspace are you running?

-- 
error compiling committee.c: too many arguments to function

Re: [kvm-devel] kvm causing memory corruption? ~2.6.25-rc6

[Dave Hansen]

On Wed, 2008-03-26 at 18:58 +0200, Avi Kivity wrote:
> Dave Hansen wrote:
> > On Wed, 2008-03-26 at 11:50 +0200, Avi Kivity wrote:
> >> Dave Hansen wrote:
> >>     
> >>> I was getting some kvm userspace crashes trying to run a Windows guest.
> >>> So, I decided to try a recent kernel (2.6.25-rc6-00333-ga4083c9)  with
> >>> the kvm kernel code that shipped with that kernel.
> >>>
> >>>       
> >> This is fixed in 2.6.25-rc7.
> >>     
> >
> > I just updated to -rc7 and re-tested.  Same symptoms:
> 
> Bad.  Which kvm userspace are you running?

~/src/kvm-userspace$ git describe
kvm-63-118-g52be1a1

-- Dave

Re: [kvm-devel] kvm causing memory corruption? ~2.6.25-rc6

[Avi Kivity]

Dave Hansen wrote:
> On Wed, 2008-03-26 at 18:58 +0200, Avi Kivity wrote:
>   
>> Dave Hansen wrote:
>>     
>>> On Wed, 2008-03-26 at 11:50 +0200, Avi Kivity wrote:
>>>       
>>>> Dave Hansen wrote:
>>>>     
>>>>         
>>>>> I was getting some kvm userspace crashes trying to run a Windows guest.
>>>>> So, I decided to try a recent kernel (2.6.25-rc6-00333-ga4083c9)  with
>>>>> the kvm kernel code that shipped with that kernel.
>>>>>
>>>>>       
>>>>>           
>>>> This is fixed in 2.6.25-rc7.
>>>>     
>>>>         
>>> I just updated to -rc7 and re-tested.  Same symptoms:
>>>       
>> Bad.  Which kvm userspace are you running?
>>     
>
> ~/src/kvm-userspace$ git describe
> kvm-63-118-g52be1a1
>
>   

I dug out my i386 install and tried it.  Doesn't reproduce for me on 
either kvm.git or -rc7.

Do you have a working setup that we can bisect?


-- 
error compiling committee.c: too many arguments to function

Re: [kvm-devel] kvm causing memory corruption? ~2.6.25-rc6

[Avi Kivity]

Avi Kivity wrote:
> Dave Hansen wrote:
>> On Wed, 2008-03-26 at 18:58 +0200, Avi Kivity wrote:
>>  
>>> Dave Hansen wrote:
>>>    
>>>> On Wed, 2008-03-26 at 11:50 +0200, Avi Kivity wrote:
>>>>      
>>>>> Dave Hansen wrote:
>>>>>            
>>>>>> I was getting some kvm userspace crashes trying to run a Windows 
>>>>>> guest.
>>>>>> So, I decided to try a recent kernel (2.6.25-rc6-00333-ga4083c9)  
>>>>>> with
>>>>>> the kvm kernel code that shipped with that kernel.
>
[...]


btw, is this with >= 4GB RAM on the host?

-- 
error compiling committee.c: too many arguments to function

Re: [kvm-devel] kvm causing memory corruption? ~2.6.25-rc6

[Dave Hansen]

On Thu, 2008-03-27 at 12:10 +0200, Avi Kivity wrote:
> btw, is this with >= 4GB RAM on the host?

Well, are you asking whether I have PAE on or not? :)

The host has 4GB of RAM exactly, but there are some serious BIOS holes,
so I needed PAE because about a gig of it is mapped >4GB.

[    0.000000] BIOS-provided physical RAM map:
[    0.000000]  BIOS-e820: 0000000000000000 - 000000000009d800 (usable)
[    0.000000]  BIOS-e820: 000000000009d800 - 00000000000a0000 (reserved)
[    0.000000]  BIOS-e820: 00000000000d2000 - 00000000000d4000 (reserved)
[    0.000000]  BIOS-e820: 00000000000e0000 - 0000000000100000 (reserved)
[    0.000000]  BIOS-e820: 0000000000100000 - 00000000bf6b0000 (usable)
[    0.000000]  BIOS-e820: 00000000bf6b0000 - 00000000bf6cc000 (ACPI data)
[    0.000000]  BIOS-e820: 00000000bf6cc000 - 00000000bf700000 (ACPI NVS)
[    0.000000]  BIOS-e820: 00000000bf700000 - 00000000c0000000 (reserved)
[    0.000000]  BIOS-e820: 00000000f0000000 - 00000000f4000000 (reserved)
[    0.000000]  BIOS-e820: 00000000fec00000 - 00000000fec10000 (reserved)
[    0.000000]  BIOS-e820: 00000000fed00000 - 00000000fed00400 (reserved)
[    0.000000]  BIOS-e820: 00000000fed14000 - 00000000fed1a000 (reserved)
[    0.000000]  BIOS-e820: 00000000fed1c000 - 00000000fed90000 (reserved)
[    0.000000]  BIOS-e820: 00000000fee00000 - 00000000fee01000 (reserved)
[    0.000000]  BIOS-e820: 00000000ff000000 - 0000000100000000 (reserved)
[    0.000000]  BIOS-e820: 0000000100000000 - 000000013c000000 (usable)


-- Dave

Re: [kvm-devel] kvm causing memory corruption? ~2.6.25-rc6

[Avi Kivity]

Dave Hansen wrote:
> On Thu, 2008-03-27 at 12:10 +0200, Avi Kivity wrote:
>   
>> btw, is this with >= 4GB RAM on the host?
>>     
>
> Well, are you asking whether I have PAE on or not? :)
>
>   

No, I'm asking whether there is a possibility of address truncation :)

PAE by itself doesn't affect kvm much, as it always runs the guest in 
pae mode.

Can you try running with mem=2000M or something?

-- 
error compiling committee.c: too many arguments to function

Re: [kvm-devel] kvm causing memory corruption? ~2.6.25-rc6

[Dave Hansen]

On Thu, 2008-03-27 at 11:36 +0200, Avi Kivity wrote:
> I dug out my i386 install and tried it.  Doesn't reproduce for me on 
> either kvm.git or -rc7.
> 
> Do you have a working setup that we can bisect?

I don't really have a working revision to bisect against.  I'm not sure
that it ever worked.  It's also on my actual laptop, so it's a bit of a
pain to get any other work done while I'm bisecting. :)

I'll move the Windows image over to another machine today and see if I
can reproduce elsewhere.  I'll also check some older versions of KVM to
see if any of those work.  If I do that, should I keep the kvm
userspace, modules and BIOSes all synchronized from each version that I
test?

-- Dave

Re: [kvm-devel] kvm causing memory corruption? ~2.6.25-rc6

[Avi Kivity]

Dave Hansen wrote:
> On Thu, 2008-03-27 at 11:36 +0200, Avi Kivity wrote:
>   
>> I dug out my i386 install and tried it.  Doesn't reproduce for me on 
>> either kvm.git or -rc7.
>>
>> Do you have a working setup that we can bisect?
>>     
>
> I don't really have a working revision to bisect against.  I'm not sure
> that it ever worked.  

I'm fairly sure Windows works on kvm...

How did you generate the image?

> It's also on my actual laptop, so it's a bit of a
> pain to get any other work done while I'm bisecting. :)
>
> I'll move the Windows image over to another machine today and see if I
> can reproduce elsewhere.  I'll also check some older versions of KVM to
> see if any of those work.  If I do that, should I keep the kvm
> userspace, modules and BIOSes all synchronized from each version that I
> test?
>   

You can keep the userspace (qemu + bios) fixed and change the kernel, or 
vice versa.

-- 
error compiling committee.c: too many arguments to function

Re: [kvm-devel] kvm causing memory corruption? ~2.6.25-rc6

[Dave Hansen]

On Thu, 2008-03-27 at 16:59 +0200, Avi Kivity wrote:
> Dave Hansen wrote:
> > On Thu, 2008-03-27 at 12:10 +0200, Avi Kivity wrote:
> >> btw, is this with >= 4GB RAM on the host?
> >>     
> > Well, are you asking whether I have PAE on or not? :)
> 
> No, I'm asking whether there is a possibility of address truncation :)
> 
> PAE by itself doesn't affect kvm much, as it always runs the guest in 
> pae mode.
> 
> Can you try running with mem=2000M or something?

Oh, sure.  I'll give that a shot.

-- Dave

Re: [kvm-devel] kvm causing memory corruption? ~2.6.25-rc6

[Dave Hansen]

On Thu, 2008-03-27 at 17:53 +0200, Avi Kivity wrote:
> Dave Hansen wrote:
> > On Thu, 2008-03-27 at 11:36 +0200, Avi Kivity wrote:
> >   
> >> I dug out my i386 install and tried it.  Doesn't reproduce for me on 
> >> either kvm.git or -rc7.
> >>
> >> Do you have a working setup that we can bisect?
> >>     
> > I don't really have a working revision to bisect against.  I'm not sure
> > that it ever worked.  
> 
> I'm fairly sure Windows works on kvm...

Oh, I didn't mean to imply that Windows doesn't work, just that the
particular perverted way in which I'm poking it may have never
worked. :)

> How did you generate the image?

The original install was done in a kqemu-accelerated host. 

> > It's also on my actual laptop, so it's a bit of a
> > pain to get any other work done while I'm bisecting. :)
> >
> > I'll move the Windows image over to another machine today and see if I
> > can reproduce elsewhere.  I'll also check some older versions of KVM to
> > see if any of those work.  If I do that, should I keep the kvm
> > userspace, modules and BIOSes all synchronized from each version that I
> > test?
> >   
> 
> You can keep the userspace (qemu + bios) fixed and change the kernel, or 
> vice versa.
> 
-- Dave

Re: kvm causing memory corruption? now 2.6.26-rc4

[Dave Hansen]

On Thu, 2008-03-27 at 16:59 +0200, Avi Kivity wrote:
> Dave Hansen wrote:
> > On Thu, 2008-03-27 at 12:10 +0200, Avi Kivity wrote:
> >> btw, is this with >= 4GB RAM on the host?
> >
> > Well, are you asking whether I have PAE on or not? :)  
> 
> No, I'm asking whether there is a possibility of address truncation :)
> 
> PAE by itself doesn't affect kvm much, as it always runs the guest in 
> pae mode.
> 
> Can you try running with mem=2000M or something?

I have a few more data points on this.  Sorry for the massive delay from
the last report -- I'm being a crappy bug reporter.  But, this is on my
one and only laptop which makes it a serious pain to diagnose.  I also
didn't have a hardware serial console on it before, which I do now.
This is all on 2.6.26-rc4-01549-g1beee8d.

Adding the mem= does not help at all.  But, it is all a bit more
diagnosable now than a month or two ago.  I turned on all of the kernel
debugging that I could get my grubby little hands on.  It now oopses
quite consistently when kvm runs instead of after.  Here's a collection
of oopses that I captured after setting up a serial line:

	http://sr71.net/~dave/kvm-oops1.txt

After collecting all those, I turned on CONFIG_DEBUG_HIGHMEM and the
oopses miraculously stopped.  But, the guest hung (for at least 5
minutes or so) during windows bootup, pegging my host CPU.  Most of the
CPU was going to klogd, so I checked dmesg.

I was seeing messages like this

[  428.918108] kvm_handle_exit: unexpected, valid vectoring info and exit reason is 0x9

And quite a few of them, like 100,000/sec.  That's why klogd was pegging
the CPU.  Any idea on a next debugging step?

-- Dave

Re: kvm causing memory corruption? now 2.6.26-rc4

[Dave Hansen]

On Mon, 2008-06-02 at 15:30 -0700, Dave Hansen wrote:
> On Thu, 2008-03-27 at 16:59 +0200, Avi Kivity wrote:
> > Dave Hansen wrote:
> > > On Thu, 2008-03-27 at 12:10 +0200, Avi Kivity wrote:
> > >> btw, is this with >= 4GB RAM on the host?
> > >
> > > Well, are you asking whether I have PAE on or not? :)  
> > 
> > No, I'm asking whether there is a possibility of address truncation :)
> > 
> > PAE by itself doesn't affect kvm much, as it always runs the guest in 
> > pae mode.
> > 
> > Can you try running with mem=2000M or something?
> 
> I have a few more data points on this.  Sorry for the massive delay from
> the last report -- I'm being a crappy bug reporter.  But, this is on my
> one and only laptop which makes it a serious pain to diagnose.  I also
> didn't have a hardware serial console on it before, which I do now.
> This is all on 2.6.26-rc4-01549-g1beee8d.
> 
> Adding the mem= does not help at all.  But, it is all a bit more
> diagnosable now than a month or two ago.  I turned on all of the kernel
> debugging that I could get my grubby little hands on.  It now oopses
> quite consistently when kvm runs instead of after.  Here's a collection
> of oopses that I captured after setting up a serial line:
> 
> 	http://sr71.net/~dave/kvm-oops1.txt
> 
> After collecting all those, I turned on CONFIG_DEBUG_HIGHMEM and the
> oopses miraculously stopped.  But, the guest hung (for at least 5
> minutes or so) during windows bootup, pegging my host CPU.  Most of the
> CPU was going to klogd, so I checked dmesg.
> 
> I was seeing messages like this
> 
> [  428.918108] kvm_handle_exit: unexpected, valid vectoring info and exit reason is 0x9
> 
> And quite a few of them, like 100,000/sec.  That's why klogd was pegging
> the CPU.  Any idea on a next debugging step?

I followed these steps, and can now boot a vm.  But, causing the host
crashes is still a pretty bad bug.  I would imagine turning ACPI back on
will let me reproduce if necessary.

http://kvm.qumranet.com/kvmwiki/Windows_ACPI_Workaround

-- Dave

Re: kvm causing memory corruption? now 2.6.26-rc4

[Avi Kivity]

Dave Hansen wrote:
> On Thu, 2008-03-27 at 16:59 +0200, Avi Kivity wrote:
>   
>> Dave Hansen wrote:
>>     
>>> On Thu, 2008-03-27 at 12:10 +0200, Avi Kivity wrote:
>>>       
>>>> btw, is this with >= 4GB RAM on the host?
>>>>         
>>> Well, are you asking whether I have PAE on or not? :)  
>>>       
>> No, I'm asking whether there is a possibility of address truncation :)
>>
>> PAE by itself doesn't affect kvm much, as it always runs the guest in 
>> pae mode.
>>
>> Can you try running with mem=2000M or something?
>>     
>
> I have a few more data points on this.  Sorry for the massive delay from
> the last report -- I'm being a crappy bug reporter.  But, this is on my
> one and only laptop which makes it a serious pain to diagnose.  I also
> didn't have a hardware serial console on it before, which I do now.
> This is all on 2.6.26-rc4-01549-g1beee8d.
>
> Adding the mem= does not help at all.  But, it is all a bit more
> diagnosable now than a month or two ago.  I turned on all of the kernel
> debugging that I could get my grubby little hands on.  It now oopses
> quite consistently when kvm runs instead of after.  Here's a collection
> of oopses that I captured after setting up a serial line:
>
> 	http://sr71.net/~dave/kvm-oops1.txt
>
> After collecting all those, I turned on CONFIG_DEBUG_HIGHMEM and the
> oopses miraculously stopped.  But, the guest hung (for at least 5
> minutes or so) during windows bootup, pegging my host CPU.  Most of the
> CPU was going to klogd, so I checked dmesg.
>
>   

Can you check with mem=900 (and CONFIG_HIGHMEM_DEBUG=n)?  That will 
confirm that the problems are highmem related, but not physical address 
truncation related.

> I was seeing messages like this
>
> [  428.918108] kvm_handle_exit: unexpected, valid vectoring info and exit reason is 0x9
>
> And quite a few of them, like 100,000/sec.  That's why klogd was pegging
> the CPU.  Any idea on a next debugging step?
>
>   

That's a task switch.  Newer kvms handle them.


-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.

Re: kvm causing memory corruption? now 2.6.26-rc4

[Dave Hansen]

On Wed, 2008-06-04 at 16:42 +0300, Avi Kivity wrote:
> Dave Hansen wrote:
...
> > After collecting all those, I turned on CONFIG_DEBUG_HIGHMEM and the
> > oopses miraculously stopped.  But, the guest hung (for at least 5
> > minutes or so) during windows bootup, pegging my host CPU.  Most of the
> > CPU was going to klogd, so I checked dmesg.
> >
> Can you check with mem=900 (and CONFIG_HIGHMEM_DEBUG=n)?  That will 
> confirm that the problems are highmem related, but not physical address 
> truncation related.

Do you mean 800M? ;) Highmem begins at 896MB if I remember correctly.

Anyway, it still oopses on current git with mem=800M

> > I was seeing messages like this
> >
> > [  428.918108] kvm_handle_exit: unexpected, valid vectoring info and exit reason is 0x9
> >
> > And quite a few of them, like 100,000/sec.  That's why klogd was pegging
> > the CPU.  Any idea on a next debugging step?
> >
> That's a task switch.  Newer kvms handle them.

Newer userspace?  I'm running current kvm-git userspace as of a day or
two ago.

-- Dave

Re: kvm causing memory corruption? now 2.6.26-rc4

[Avi Kivity]

Dave Hansen wrote:
> On Wed, 2008-06-04 at 16:42 +0300, Avi Kivity wrote:
>   
>> Dave Hansen wrote:
>>     
> ...
>   
>>> After collecting all those, I turned on CONFIG_DEBUG_HIGHMEM and the
>>> oopses miraculously stopped.  But, the guest hung (for at least 5
>>> minutes or so) during windows bootup, pegging my host CPU.  Most of the
>>> CPU was going to klogd, so I checked dmesg.
>>>
>>>       
>> Can you check with mem=900 (and CONFIG_HIGHMEM_DEBUG=n)?  That will 
>> confirm that the problems are highmem related, but not physical address 
>> truncation related.
>>     
>
> Do you mean 800M? ;) Highmem begins at 896MB if I remember correctly.
>
> Anyway, it still oopses on current git with mem=800M
>
>   

Stumped.  Please post .config, will try to reproduce.

>>> I was seeing messages like this
>>>
>>> [  428.918108] kvm_handle_exit: unexpected, valid vectoring info and exit reason is 0x9
>>>
>>> And quite a few of them, like 100,000/sec.  That's why klogd was pegging
>>> the CPU.  Any idea on a next debugging step?
>>>
>>>       
>> That's a task switch.  Newer kvms handle them.
>>     
>
> Newer userspace?  I'm running current kvm-git userspace as of a day or
> two ago.
>   

No, it's kernel code.

-- 
I have a truly marvellous patch that fixes the bug which this
signature is too narrow to contain.

Re: kvm causing memory corruption? now 2.6.26-rc4

[Dave Hansen]

On Thu, 2008-06-12 at 16:10 +0300, Avi Kivity wrote:
> Stumped.  Please post .config, will try to reproduce.

http://sr71.net/~dave/linux/config-2.6.26-rc6

-- Dave

Re: kvm causing memory corruption? now 2.6.26-rc8

[Dave Hansen]

So, just a continuation of what we were talking about before...

I just had a bug triggered on my system because I'm running sparsemem
(it was in show_mem()).  I wonder if sparsemem is contributing to the
bug.  Does kvm ever do any arithmetic that you can think of with 'struct
page's?

To summarize my config: 
* 4GB of RAM
* hole from 3-4GB (so ram ranges from 0-5GB)
* PAE enabled
* SPARSEMEM=y

http://sr71.net/~dave/linux/config-2.6.26-rc8-00089-ge1441b9

I'm compiling a kernel right now to try without sparsemem.

-- Dave

Re: kvm causing memory corruption? now 2.6.26

[Dave Hansen]

On a suggestion of Anthony's, I tried a defconfig kernel.

It is now bombing out on an assertion in the lapic code:

	http://sr71.net/~dave/linux/2.6.26-oops1.txt



-- Dave

KVM overflows the stack

[Dave Hansen]

On Wed, 2008-07-16 at 14:44 -0700, Dave Hansen wrote:
> On a suggestion of Anthony's, I tried a defconfig kernel.
> 
> It is now bombing out on an assertion in the lapic code:
> 
> 	http://sr71.net/~dave/linux/2.6.26-oops1.txt

I think I found it!!!

$ (objdump -d kvm.ko ; objdump -d kvm-intel.ko ) | egrep 'sub.*0x...,.*esp|>:'  | egrep sub -B1
00001a90 <kvm_vcpu_ioctl>:
    1a9a:	81 ec 60 06 00 00    	sub    $0x660,%esp
--
00004e90 <kvm_arch_vcpu_ioctl>:
    4e9d:	81 ec 6c 08 00 00    	sub    $0x86c,%esp
--
00005900 <kvm_arch_vm_ioctl>:
    5903:	81 ec 34 05 00 00    	sub    $0x534,%esp
--
0000d4f0 <paging64_prefetch_page>:
    d4f8:	81 ec 1c 01 00 00    	sub    $0x11c,%esp
--
0000dfd0 <paging32_prefetch_page>:
    dfd8:	81 ec 1c 01 00 00    	sub    $0x11c,%esp
--
0000f390 <kvm_pv_mmu_op>:
    f3a1:	81 ec 28 02 00 00    	sub    $0x228,%esp

We're simply overflowing the stack.  I changed all of the large on-stack
allocations to 'static', and it actually boots now.  I know 'static'
isn't safe, but it was good for a quick test.

A 'make stackcheck' confirms this:

dave@nimitz:~/kernels/linux-2.6.git$ make checkstack
objdump -d vmlinux $(find . -name '*.ko') | \
	perl /home/dave/kernels/linux-2.6.git-t61/scripts/checkstack.pl i386
0x000042d3 kvm_arch_vcpu_ioctl [kvm]:			2148
0x000012e3 kvm_vcpu_ioctl [kvm]:			1620
0x00004a83 kvm_arch_vm_ioctl [kvm]:			1332
0x00009a26 airo_get_aplist [airo]:			1140
0x00009b76 airo_get_aplist [airo]:			1140
0x00009c82 airo_get_aplist [airo]:			1140
...

In other words, kvm has the top 3 stack users in my kernel.  As you can
see from my trace above, these things also get called with super-long
stacks already.  Man.  That sucked to find.

Avi, how would you like this fixed?  I'd be happy to prepare some
patches.  Do you have a particular approach that you think we should
use?  Just make the big objects dynamically allocated?

-- Dave

Re: kvm causing memory corruption? now 2.6.26

[Avi Kivity]

Dave Hansen wrote:
> On a suggestion of Anthony's, I tried a defconfig kernel.
>
> It is now bombing out on an assertion in the lapic code:
>
> 	http://sr71.net/~dave/linux/2.6.26-oops1.txt
>
>
>   

Well that assert is plain wrong:

static int apic_match_dest(struct kvm_vcpu *vcpu, struct kvm_lapic *source,
               int short_hand, int dest, int dest_mode)
{
    int result = 0;
    struct kvm_lapic *target = vcpu->arch.apic;

    apic_debug("target %p, source %p, dest 0x%x, "
           "dest_mode 0x%x, short_hand 0x%x",
           target, source, dest, dest_mode, short_hand);

    ASSERT(!target);


It should be ASSERT(target), if anything.

-- 
I have a truly marvellous patch that fixes the bug which this
signature is too narrow to contain.

Re: KVM overflows the stack

[Avi Kivity]

Dave Hansen wrote:
> On Wed, 2008-07-16 at 14:44 -0700, Dave Hansen wrote:
>   
>> On a suggestion of Anthony's, I tried a defconfig kernel.
>>
>> It is now bombing out on an assertion in the lapic code:
>>
>> 	http://sr71.net/~dave/linux/2.6.26-oops1.txt
>>     
>
> I think I found it!!!
>
> $ (objdump -d kvm.ko ; objdump -d kvm-intel.ko ) | egrep 'sub.*0x...,.*esp|>:'  | egrep sub -B1
> 00001a90 <kvm_vcpu_ioctl>:
>     1a9a:	81 ec 60 06 00 00    	sub    $0x660,%esp
> --
> 00004e90 <kvm_arch_vcpu_ioctl>:
>     4e9d:	81 ec 6c 08 00 00    	sub    $0x86c,%esp
> --
> 00005900 <kvm_arch_vm_ioctl>:
>     5903:	81 ec 34 05 00 00    	sub    $0x534,%esp
> --
> 0000d4f0 <paging64_prefetch_page>:
>     d4f8:	81 ec 1c 01 00 00    	sub    $0x11c,%esp
> --
> 0000dfd0 <paging32_prefetch_page>:
>     dfd8:	81 ec 1c 01 00 00    	sub    $0x11c,%esp
> --
> 0000f390 <kvm_pv_mmu_op>:
>     f3a1:	81 ec 28 02 00 00    	sub    $0x228,%esp
>
> We're simply overflowing the stack.  I changed all of the large on-stack
> allocations to 'static', and it actually boots now.  I know 'static'
> isn't safe, but it was good for a quick test.
>
>   

Yes!   It's obvious, once you know it...

> A 'make stackcheck' confirms this:
>
> dave@nimitz:~/kernels/linux-2.6.git$ make checkstack
> objdump -d vmlinux $(find . -name '*.ko') | \
> 	perl /home/dave/kernels/linux-2.6.git-t61/scripts/checkstack.pl i386
> 0x000042d3 kvm_arch_vcpu_ioctl [kvm]:			2148
> 0x000012e3 kvm_vcpu_ioctl [kvm]:			1620
> 0x00004a83 kvm_arch_vm_ioctl [kvm]:			1332
> 0x00009a26 airo_get_aplist [airo]:			1140
> 0x00009b76 airo_get_aplist [airo]:			1140
> 0x00009c82 airo_get_aplist [airo]:			1140
> ...
>
> In other words, kvm has the top 3 stack users in my kernel.  As you can
> see from my trace above, these things also get called with super-long
> stacks already.  Man.  That sucked to find.
>
> Avi, how would you like this fixed?  I'd be happy to prepare some
> patches.  Do you have a particular approach that you think we should
> use?  Just make the big objects dynamically allocated?
>   

Yes, things like kvm_lapic_state are way too big to be on the stack.  
There's an additional problem here, that apparently your gcc (which 
version?) doesn't fold objects in a switch statement into the same stack 
slot:

switch (...) {
    case x: {
         struct medium a;
         ...
    }
    case y:
          struct medium b;
          ...
    }
};

These could be solved either by stack allocation, or by moving into 
functions marked noinline.  Whichever is easier.

-- 
I have a truly marvellous patch that fixes the bug which this
signature is too narrow to contain.

Re: KVM overflows the stack

[Roland Dreier]

 > Yes, things like kvm_lapic_state are way too big to be on the stack.

I had a quick look at the code, and my worry about dynamic allocation
would be that handling allocation failure seems like it might get
tricky.  Eg for handling struct kvm_pv_mmu_op_buffer (which is 528 bytes
on the stack in kvm_pv_mmu_op()) can you deal with an mmu op failing?
(maybe in that case you can easily by just setting *ret to 0?)

 > There's an additional problem here, that apparently your gcc (which
 > version?) doesn't fold objects in a switch statement into the same
 > stack slot:
 > 
 > switch (...) {
 >    case x: {
 >         struct medium a;
 >         ...
 >    }
 >    case y:
 >          struct medium b;
 >          ...
 >    }
 > };

A trick for this is to do:

	union {
		struct medium1 a;
		struct medium2 b;
	} u;

	switch (...) {
	case x:
		use u.a;
		...

	case y:
		use u.b;
		...
	}

Re: KVM overflows the stack

[Dave Hansen]

On Thu, 2008-07-17 at 08:52 +0300, Avi Kivity wrote:
> Dave Hansen wrote:
> > Avi, how would you like this fixed?  I'd be happy to prepare some
> > patches.  Do you have a particular approach that you think we should
> > use?  Just make the big objects dynamically allocated?
> >   
> 
> Yes, things like kvm_lapic_state are way too big to be on the stack.  
> There's an additional problem here, that apparently your gcc (which 
> version?) doesn't fold objects in a switch statement into the same stack 
> slot:

$ gcc -v
gcc version 3.4.6 (Ubuntu 3.4.6-6ubuntu5)

> switch (...) {
>     case x: {
>          struct medium a;
>          ...
>     }
>     case y:
>           struct medium b;
>           ...
>     }
> };
> 
> These could be solved either by stack allocation, or by moving into 
> functions marked noinline.  Whichever is easier.

Did you mean dynamic allocation? :)

-- Dave

Re: KVM overflows the stack

[Dave Hansen]

A newer gcc (4.2) makes this a wee bit better, but probably still
worrisome.

dave@nimitz:~/src/kvm-userspace-virgin/kernel$ objdump -d *.ko | perl /home/dave/kernels/linux-2.6.git-t61/scripts/checkstack.pl i386
0x00007b33 kvm_arch_vm_ioctl [kvm]:			1164
0x000072e8 kvm_arch_vcpu_ioctl [kvm]:			1060
0x00007497 kvm_arch_vcpu_ioctl [kvm]:			1060
0x0000adad kvm_pv_mmu_op [kvm]:				564
0x0000ae6b kvm_pv_mmu_op [kvm]:				564
0x00002553 kvm_vcpu_ioctl [kvm]:			468
0x0000d7e8 paging64_prefetch_page [kvm]:		304
0x0000d926 paging64_prefetch_page [kvm]:		304
0x0000d936 paging64_prefetch_page [kvm]:		304
0x0000c118 paging32_prefetch_page [kvm]:		300
0x0000c146 paging32_prefetch_page [kvm]:		300
0x0000c286 paging32_prefetch_page [kvm]:		300
0x0000f1eb x86_emulate_insn [kvm]:			192
0x0000f372 x86_emulate_insn [kvm]:			192
0x0000f5f2 x86_emulate_insn [kvm]:			192
0x00005373 kvm_task_switch [kvm]:			172
0x00001466 __kvm_set_memory_region [kvm]:		164
0x000014da __kvm_set_memory_region [kvm]:		164
0x0000dae8 paging64_page_fault [kvm]:			156
0x0000db06 paging64_page_fault [kvm]:			156
0x0000db67 paging64_page_fault [kvm]:			156
0x0000dcd5 paging64_page_fault [kvm]:			156
0x0000dcf1 paging64_page_fault [kvm]:			156
0x0000c356 kvm_mmu_pte_write [kvm]:			152
0x0000c50a kvm_mmu_pte_write [kvm]:			152
0x0000df28 paging32_page_fault [kvm]:			144
0x0000df46 paging32_page_fault [kvm]:			144
0x0000dfa7 paging32_page_fault [kvm]:			144
0x0000e115 paging32_page_fault [kvm]:			144
0x0000e131 paging32_page_fault [kvm]:			144
0x00014ed3 apic_mmio_read [kvm]:			108

-- Dave

Re: KVM overflows the stack

[Avi Kivity]

Dave Hansen wrote:
>   
>> switch (...) {
>>     case x: {
>>          struct medium a;
>>          ...
>>     }
>>     case y:
>>           struct medium b;
>>           ...
>>     }
>> };
>>
>> These could be solved either by stack allocation, or by moving into 
>> functions marked noinline.  Whichever is easier.
>>     
>
> Did you mean dynamic allocation? :)
>
>   

Yeah.  Too early in the morning for me.

Roland's trick might be easier than either, though.

-- 
I have a truly marvellous patch that fixes the bug which this
signature is too narrow to contain.

[PATCH] update kvm's anon_inodes.c for r/o bind mounts

[Dave Hansen]

That also reminds me.  kvm somehow has an outdated copy of
anon_inodes.c.  It needs to be updated for the r/o bind mount patches to
add a proper mnt_want/drop_write().  Otherwise, you'll run into warnings
about imbalanced mount writer counts.  Something like this will do, but
it would be best to just stay up to date with mainline for this kind of
stuff.

diff --git a/kernel/anon_inodes.c b/kernel/anon_inodes.c
index 74630ea..a47bc55 100644
--- a/kernel/anon_inodes.c
+++ b/kernel/anon_inodes.c
@@ -90,9 +90,6 @@ int anon_inode_getfd(const char *name, const struct file_operations *fops,
 
 	if (IS_ERR(anon_inode_inode))
 		return -ENODEV;
-	file = get_empty_filp();
-	if (!file)
-		return -ENFILE;
 
 	inode = igrab(anon_inode_inode);
 	if (IS_ERR(inode)) {
@@ -121,14 +118,15 @@ int anon_inode_getfd(const char *name, const struct file_operations *fops,
 	dentry->d_flags &= ~DCACHE_UNHASHED;
 	d_instantiate(dentry, inode);
 
-	file->f_vfsmnt = mntget(anon_inode_mnt);
-	file->f_dentry = dentry;
-	file->f_mapping = inode->i_mapping;
+	file = alloc_file(anon_inode_mnt, dentry, FMODE_READ | FMODE_WRITE,
+			(struct file_operations *)fops);
+	if (!file) {
+		error = -ENFILE;
+		goto err_dput;
+	}
 
 	file->f_pos = 0;
 	file->f_flags = O_RDWR;
-	file->f_op = (struct file_operations *)fops;
-	file->f_mode = FMODE_READ | FMODE_WRITE;
 	file->f_version = 0;
 	file->private_data = priv;
 
@@ -136,12 +134,13 @@ int anon_inode_getfd(const char *name, const struct file_operations *fops,
 
 	return fd;
 
+err_dput:
+	dput(dentry);
 err_put_unused_fd:
 	put_unused_fd(fd);
 err_iput:
 	iput(inode);
 err_put_filp:
-	fput(file);
 	return error;
 }
 


-- Dave

Re: KVM overflows the stack

[Dave Hansen]

On Wed, 2008-07-16 at 23:08 -0700, Roland Dreier wrote:
>  > Yes, things like kvm_lapic_state are way too big to be on the
> stack.
> 
> I had a quick look at the code, and my worry about dynamic allocation
> would be that handling allocation failure seems like it might get
> tricky.  Eg for handling struct kvm_pv_mmu_op_buffer (which is 528 bytes
> on the stack in kvm_pv_mmu_op()) can you deal with an mmu op failing?

Well, you *better* be able to deal with it.  :)

This code is also doing a *ton* of copy_to/from_user().  If userspace
had one of its input buffers swapped out (or one of its output buffers
not faulted in yet) and we're out of memory enough to be failing
kmallocs() then we're sure as heck also going to failing the user
copies.

I think it's a non-issue.

-- Dave

Re: KVM overflows the stack

[Avi Kivity]

Dave Hansen wrote:
> On Wed, 2008-07-16 at 23:08 -0700, Roland Dreier wrote:
>   
>>  > Yes, things like kvm_lapic_state are way too big to be on the
>> stack.
>>
>> I had a quick look at the code, and my worry about dynamic allocation
>> would be that handling allocation failure seems like it might get
>> tricky.  Eg for handling struct kvm_pv_mmu_op_buffer (which is 528 bytes
>> on the stack in kvm_pv_mmu_op()) can you deal with an mmu op failing?
>>     
>
> Well, you *better* be able to deal with it.  :)
>
> This code is also doing a *ton* of copy_to/from_user().  If userspace
> had one of its input buffers swapped out (or one of its output buffers
> not faulted in yet) and we're out of memory enough to be failing
> kmallocs() then we're sure as heck also going to failing the user
> copies.
>
> I think it's a non-issue.
>
>   

Yes, it's designed to be restartable.  Returning 0 should be fine.

We can reduce the buffer size to 256 though.  I wouldn't want an 
allocation in this hot path.

-- 
error compiling committee.c: too many arguments to function

Re: [PATCH] update kvm's anon_inodes.c for r/o bind mounts

[Avi Kivity]

Dave Hansen wrote:
> That also reminds me.  kvm somehow has an outdated copy of
> anon_inodes.c.  It needs to be updated for the r/o bind mount patches to
> add a proper mnt_want/drop_write().  Otherwise, you'll run into warnings
> about imbalanced mount writer counts.  Something like this will do, but
> it would be best to just stay up to date with mainline for this kind of
> stuff.
>
>   

Instead I changed the code to use the host's anon_inodes when 
available.  I also introduced a couple of bugs but the list fixed these 
quickly.  Thanks list!

-- 
I have a truly marvellous patch that fixes the bug which this
signature is too narrow to contain.