linux-2.6.29-git1 + kubuntu-8.04 kvm = kernel oops

linux-2.6.29-git1 + kubuntu-8.04 kvm = kernel oops

[Adam Richter]

Hi everyone,

I have a qcow2 image that runs fine under kubuntu 8.04 with kvm for kernels up to 2.6.29.  However, for 2.6.29-git1 and every kernel that I have tried thereafter including 2.6.30-rc8-git1, I get a kernel oops when I try to run kvm on this image.

>From the stack traces that I see, it is possible that the bug is not in kvm, but rather in some filesystem code.  However, fs/ext3/inode.c was the only file in fs/ext3 updated between 2.6.29 and 2.6.29-git1, and the problem persisisted after I reverted that change in 2.6.29-git1.

I should also add that, on another Linux computer, which is not running Kubuntu, I got a kernel oops when trying to rsync an image I use with kvm when I was also using a post-2.6.29 kernel.  It is possible that I had run kvm on that file since booting the computer and before doing the rsync, but I am not sure.  So, it is possible that there might be bug where kvm somehow breaks dentry or inode information, which results in an oops later in the file system code, or it may still be possible that the bug is purely a file system bug.

I have done a few iterations of git bisect, but I do not think I will have time to do the ~10 more that will be necessary for it to converge (assuming no versions in the middle with serious compilation problems).  So, I am posting this information now.

The rest of this message is just information to help anyone who thinks they recognize this bug to determine if this is likely the same bug.  I'll post a follow-up if and when I complete the git bisect, assuming that I do not learn that this problem has already been solved.  If anyone recognizes this problem as having a known fix, please let me know so that I can stop duplicating your efforts.

Thanks in advance for any feedback.

Adam Richter


I know I have observed the problem on all kernels after 2.6.29-git1 that I have tried, including at least these:

2.6.29-git1
2.6.29-git2
2.6.29-git4
2.6.29-git8
2.6.30-rc1
...some other 2.6.30-rc's...
2.6.30-rc8
2.6.30-rc8-git1

The qcow2 disk image is approximately 19 gigabytes:

$ ls -lh kubuntu-9.04.qcow2
-rw-r--r-- 1 adam adam 19G 2009-05-31 02:10 kubuntu-9.04.qcow2
$ ls -l kubuntu-9.04.qcow2
-rw-r--r-- 1 adam adam 19714072576 2009-05-31 02:10 kubuntu-9.04.qcow2

Here is the shell script that I use to invoke kvm:

#!/bin/sh

#if [ -z "$*" ] ; then
#       set -- -cdrom kubuntu-9.04-desktop-i386.iso
#fi

if tty --silent ; then
        sudo=sudo
else
        sudo=
fi

$sudo modprobe kqemu
$sudo modprobe kvm-intel
$sudo modprobe kvm
$sudo chmod a+rw /dev/kqemu /dev/kvm

#disk_hw='-hda /home/adam/virtualization/kubuntu-9.04.qcow2'
disk_hw='-drive file=/home/adam/virtualization/kubuntu-9.04.qcow2,if=virtio,boot=on'
net_hw='-net user -net nic,model=virtio'

export QEMU_AUDIO_DRV=sdl
exec kvm $disk_hw $net_hw \
        -m 1024 \
        -soundhw es1370 \
        -usb -usbdevice tablet \
        $@



And, finally, here is an oops log:


[  111.781369] type=1804 audit(1244170344.844:8): integrity: pid=9045 uid=0 auid=4294967295 ses=4294967295 op=invalid_pcr cause=open_writers comm="update_auth" name="sh-thd-1244179930" dev=sda2 ino=828773 res=0
[  131.790166] kubuntu-9.04.qcow2 dentry_open failed
[  131.790193] BUG: unable to handle kernel paging request at ffffffcd
[  131.790199] IP: [<c01d66da>] fput+0xa/0x30
[  131.790207] *pdpt = 00000000005b4001 *pde = 00000000005b7067 *pte = 0000000000000000
[  131.790214] Oops: 0002 [#1] SMP
[  131.790217] last sysfs file: /sys/class/net/eth0/carrier
[  131.790221] Modules linked in: af_packet rfcomm l2cap bluetooth nfsd exportfs kvm_intel kvm radeon drm agpgart ppdev ipv6 video output pci_slot container sbs sbshc dgcusbdcp battery acpi_cpufreq cpufreq_stats cpufreq_conservative cpufreq_powersave cpufreq_userspace cpufreq_ondemand freq_table nfs lockd fscache nfs_acl auth_rpcgss sunrpc iptable_filter ip_tables x_tables dm_crypt dm_mod ac sbp2 parport_pc lp parport snd_hda_codec_atihdmi snd_hda_codec_realtek snd_hda_intel snd_hda_codec snd_seq_oss snd_pcm_oss snd_seq_midi snd_mixer_oss snd_pcm snd_rawmidi snd_seq_midi_event snd_seq snd_timer snd_seq_device snd serio_raw psmouse shpchp soundcore snd_page_alloc pci_hotplug wmi pcspkr rtc_cmos rtc_core i2c_nforce2 i2c_core rtc_lib processor button evdev ext3 jbd mbcache sg sr_mod cdrom sd_
 mod crc_t10dif pata_amd usb_storage pata_acpi usb_libusual ata_generic ahci ohci1394 ohci_hcd ieee1394 libata ehci_hcd forcedeth scsi_mod usbcore thermal fan
 thermal_sys fuse fbcon tileblit font bitblit softcursor
[  131.790319]
[  131.790323] Pid: 9078, comm: kvm Not tainted (2.6.30-rc8-git1 #1) GX613AA-ABA a6300f
[  131.790326] EIP: 0060:[<c01d66da>] EFLAGS: 00010282 CPU: 0
[  131.790330] EIP is at fput+0xa/0x30
[  131.790333] EAX: ffffffb5 EBX: f215e7cc ECX: 00000082 EDX: ffffffb5
[  131.790336] ESI: ffffffb5 EDI: f1dedf00 EBP: f1dede80 ESP: f1dede80
[  131.790339]  DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068
[  131.790343] Process kvm (pid: 9078, ti=f1dec000 task=f1d42c00 task.ti=f1dec000)
[  131.790345] Stack:
[  131.790347]  f1dedea8 c0250a00 c04a0fcf f215e7cc f2df6160 002360df f2df6140 f216be9c
[  131.790356]  00000004 00000000 f1deded0 c01ddf2c ffffff9c f1deded0 00008001 f1dedf00
[  131.790364]  f215e770 00000000 ffffffe9 00000000 f1dedf6c c01e0ad4 f1dedf00 f1dedee4
[  131.790374] Call Trace:
[  131.790376]  [<c0250a00>] ? ima_path_check+0x180/0x1b0
[  131.790383]  [<c01ddf2c>] ? may_open+0x9c/0x2d0
[  131.790387]  [<c01e0ad4>] ? do_filp_open+0x164/0x770
[  131.790392]  [<c01264d6>] ? kunmap_atomic+0x36/0x90
[  131.790397]  [<c01266c0>] ? kmap_atomic+0x0/0x30
[  131.790402]  [<c01b4606>] ? handle_mm_fault+0x796/0xd60
[  131.790409]  [<c01d3186>] ? do_sys_open+0x56/0xf0
[  131.790413]  [<c01d328e>] ? sys_open+0x2e/0x40
[  131.790417]  [<c0103023>] ? sysenter_do_call+0x12/0x28
[  131.790423] Code: 8b 45 0c 89 44 24 04 8b 45 08 89 04 24 89 f8 e8 ad 66 f6 ff 8b 5d f4 8b 75 f8 8b 7d fc 89 ec 5d c3 55 89 e5 0f 1f 44 00 00 89 c2 <f0> ff 48 18 0f 94 c0 84 c0 75 02 5d c3 89 d0 e8 82 fb ff ff 5d
[  131.790470] EIP: [<c01d66da>] fput+0xa/0x30 SS:ESP 0068:f1dede80
[  131.790477] CR2: 00000000ffffffcd
[  131.790480] ---[ end trace d5ce23f6e60b0afc ]---





      

Re: linux-2.6.29-git1 + kubuntu-8.04 kvm = kernel oops

[Avi Kivity]

Adam Richter wrote:
> Hi everyone,
>
> I have a qcow2 image that runs fine under kubuntu 8.04 with kvm for kernels up to 2.6.29.  However, for 2.6.29-git1 and every kernel that I have tried thereafter including 2.6.30-rc8-git1, I get a kernel oops when I try to run kvm on this image.
>
> From the stack traces that I see, it is possible that the bug is not in kvm, but rather in some filesystem code.  However, fs/ext3/inode.c was the only file in fs/ext3 updated between 2.6.29 and 2.6.29-git1, and the problem persisisted after I reverted that change in 2.6.29-git1.
>
> I should also add that, on another Linux computer, which is not running Kubuntu, I got a kernel oops when trying to rsync an image I use with kvm when I was also using a post-2.6.29 kernel.  It is possible that I had run kvm on that file since booting the computer and before doing the rsync, but I am not sure.  So, it is possible that there might be bug where kvm somehow breaks dentry or inode information, which results in an oops later in the file system code, or it may still be possible that the bug is purely a file system bug.
>
> I have done a few iterations of git bisect, but I do not think I will have time to do the ~10 more that will be necessary for it to converge (assuming no versions in the middle with serious compilation problems).  So, I am posting this information now.
>
> The rest of this message is just information to help anyone who thinks they recognize this bug to determine if this is likely the same bug.  I'll post a follow-up if and when I complete the git bisect, assuming that I do not learn that this problem has already been solved.  If anyone recognizes this problem as having a known fix, please let me know so that I can stop duplicating your efforts.
>
>   

This is a guest kernel oops, right?

When you change the kernel, do you mean the host kernel or guest kernel?

What arch and pae-ness are the guest and host running?


-- 
error compiling committee.c: too many arguments to function

Re: linux-2.6.29-git1 + kubuntu-8.04 kvm = kernel oops

[Adam Richter]

On Mon, 6/8/09, Avi Kivity <avi@redhat.com> wrote:
> Turns out this is a non-kvm bug; see http://bugzilla.kernel.org/show_bug.cgi?id=13473. 
> Disable CONFIG_IMA or wait for 2.6.30.1.

Right you are!

I confirm that disabling CONFIG_IMA in linux-2.6.30-rc8-git6 eliminated the kernel oops I was seeing.

Thank you!

Adam Richter



      

Re: linux-2.6.29-git1 + kubuntu-8.04 kvm = kernel oops

[Avi Kivity]

adam_richter2004@yahoo.com wrote:
>> When you change the kernel, do you mean the host kernel or
>> guest kernel?
>>     
>
> Host kernel.
>  

Turns out this is a non-kvm bug; see 
http://bugzilla.kernel.org/show_bug.cgi?id=13473.  Disable CONFIG_IMA or 
wait for 2.6.30.1.

-- 
I have a truly marvellous patch that fixes the bug which this
signature is too narrow to contain.

Re: linux-2.6.29-git1 + kubuntu-8.04 kvm = kernel oops

[adam_richter2004]

On Mon, 6/8/09, Avi Kivity wrote:
> Adam Richter wrote:
> > Hi everyone,
> >
> > I have a qcow2 image that runs fine under kubuntu 8.04
> with kvm for kernels up to 2.6.29.  However, for
> 2.6.29-git1 and every kernel that I have tried thereafter
> including 2.6.30-rc8-git1, I get a kernel oops when I try to
> run kvm on this image.
> >
> > From the stack traces that I see, it is possible that
> the bug is not in kvm, but rather in some filesystem
> code.  However, fs/ext3/inode.c was the only file in
> fs/ext3 updated between 2.6.29 and 2.6.29-git1, and the
> problem persisisted after I reverted that change in
> 2.6.29-git1.
> >
> > I should also add that, on another Linux computer,
> which is not running Kubuntu, I got a kernel oops when
> trying to rsync an image I use with kvm when I was also
> using a post-2.6.29 kernel.  It is possible that I had
> run kvm on that file since booting the computer and before
> doing the rsync, but I am not sure.  So, it is possible
> that there might be bug where kvm somehow breaks dentry or
> inode information, which results in an oops later in the
> file system code, or it may still be possible that the bug
> is purely a file system bug.
> >
> > I have done a few iterations of git bisect, but I do
> not think I will have time to do the ~10 more that will be
> necessary for it to converge (assuming no versions in the
> middle with serious compilation problems).  So, I am
> posting this information now.
> >
> > The rest of this message is just information to help
> anyone who thinks they recognize this bug to determine if
> this is likely the same bug.  I'll post a follow-up if
> and when I complete the git bisect, assuming that I do not
> learn that this problem has already been solved.  If
> anyone recognizes this problem as having a known fix, please
> let me know so that I can stop duplicating your efforts.
> >
> >   
> 
> This is a guest kernel oops, right?

No.  The host kernel gets the oops, causing the the "kvm" program to exit with a fake kill signal from the kernel.  I believe that the oops occurs before the grub bootloader on the guest has started running, much less booted any operating system.

> When you change the kernel, do you mean the host kernel or
> guest kernel?

Host kernel.
 
> What arch and pae-ness are the guest and host running?

The guest kernel that the target OS uses happens to be non-PAE, but I do not believe that the guest kernel is even copied into memory on the guest virtual machine before the kernel oops on the host computer.

I have reproduced the problem with both PAE and non-PAE host configurations of the same linux-2.6.30-rc8-git5 source tree.

By the way, I apparently twice botched the git bisect search I was doing on this problem, and am in the process of trying for a third time.  I will let you know if and when I have a result from that.

Thank you for considering this problem.

Adam Richter