Re: BUG at mmu.c:615 from localhost migration using ept+hugetlbfs

[Marcelo Tosatti <mtosatti@redhat.com>]

Ryan,

On Fri, May 29, 2009 at 11:43:26AM -0500, Ryan Harper wrote:
> Testing latest qemu-kvm.git and kvm-kmod.git, ept enabled and backing
> guests with large pages trips a BUG in the mmu code.  If I disable ept,
> but still use large pages, migration succeeds.  Reproduce with:
> 
> hugetlbfs setup:
> % mkdir -p /hugetlbfs && mount -t hugetlbfs hugetlbfs /hugetlbfs
> % echo 10000 > /proc/sys/vm/nr_hugepages
> 
> qemu commands:
> 
> guest a:
> sudo x86_64-softmmu/qemu-system-x86_64 -L pc-bios -m 2048 -mempath /hugetlbfs -net nic -net tap -vnc :12 -monitor stdio -hda /scratch/images/rharper/rhel4u8-32-ide.raw
> 
> guest b:
> sudo x86_64-softmmu/qemu-system-x86_64 -L pc-bios -m 2048 -mempath /hugetlbfs -net nic -net tap -vnc :13 -monitor stdio -hda /scratch/images/rharper/rhel4u8-32-ide.raw -incoming tcp:0:4444
> 
> Once the guest a is up, issued migrate command:
> (qemu) migrate -d tcp:localhost:444
> 
> rmap_remove: ffff880a08e00098 c0336e65c0336e5b 0->BUG
				^^^^^^^^^^^^^^^^

This value looks very strange (bits 5:3 contain invalid value, for one).
Don't have access to HW at the very moment, so it would be great if you
had time to do a change equivalent to this and reproduce:

diff --git a/arch/x86/kvm/mmu.c b/arch/x86/kvm/mmu.c
index 809cce0..ceb70b0 100644
--- a/arch/x86/kvm/mmu.c
+++ b/arch/x86/kvm/mmu.c
@@ -1759,7 +1764,7 @@ static void mmu_set_spte(struct kvm_vcpu *vcpu, u64 *shadow_pte,
 			child = page_header(pte & PT64_BASE_ADDR_MASK);
 			mmu_page_remove_parent_pte(child, shadow_pte);
 		} else if (pfn != spte_to_pfn(*shadow_pte)) {
-			pgprintk("hfn old %lx new %lx\n",
+			printk(KERN_ERR "hfn old %lx new %lx\n",
 				 spte_to_pfn(*shadow_pte), pfn);
 			rmap_remove(vcpu->kvm, shadow_pte);
 		} else

Avi, any hints?

> ------------[ cut here ]------------
> kernel BUG at /home/rharper/work/git/kvm-kmod/x86/mmu.c:615!
> invalid opcode: 0000 [1] SMP 
> last sysfs file: /sys/devices/system/cpu/cpu15/cache/index2/shared_cpu_map
> CPU 6 
> Modules linked in: kvm_intel(N) kvm(N) tun nfs lockd nfs_acl sunrpc ipv6 bridge stp cpufreq_conservative cpufreq_userspace cpufreq_powersave acpi_cpufreq microcode fuse loop sr_mod cdrom dm_mod sg rtc_cmos thermal cdc_ether i2c_i801 rtc_core usbnet usb_storage shpchp i2c_core rtc_lib processor bnx2 pcspkr button pci_hotplug mii mptctl joydev usbhid hid ff_memless uhci_hcd ehci_hcd usbcore sd_mod crc_t10dif edd fan thermal_sys hwmon ext3 mbcache jbd mptsas mptscsih mptbase scsi_transport_sas scsi_mod [last unloaded: kvm]
> Supported: No
> Pid: 17635, comm: qemu-system-x86 Tainted: G          2.6.27.19-5-default #1
> RIP: 0010:[<ffffffffa012d8dc>]  [<ffffffffa012d8dc>] rmap_remove+0xc9/0x19e [kvm]
> RSP: 0018:ffff880c7a1cbba8  EFLAGS: 00010296
> RAX: 0000000000000039 RBX: 00000036e65c0336 RCX: ffff880c7b405e60
> RDX: ffffffff806e0d08 RSI: 0000000000000092 RDI: ffffffff806e0d00
> RBP: ffff880a08e00098 R08: ffffffff806e0cf0 R09: 0000000100000000
> R10: 0000000000000046 R11: 000000000000000a R12: ffff880c7b066a20
> R13: ffff8806778e0000 R14: 0000000000000000 R15: 0000000000000007
> FS:  00007f298b4ad950(0000) GS:ffff880c7cd83f40(0000) knlGS:0000000000000000
> CS:  0010 DS: 002b ES: 002b CR0: 000000008005003b
> CR2: 0000000000879ba0 CR3: 0000000679da8000 CR4: 00000000000026e0
> DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> Process qemu-system-x86 (pid: 17635, threadinfo ffff880c7a1ca000, task ffff8809ebce4880)
> Stack:  ffff88069822b888 0000000000000000 ffff8803f1000040 ffff880a08e00098
>  0000000000000001 ffffffffa012f5d3 ffff880c7a1cbc58 ffffffff8023661a
>  0000000000000000 8000000a08e000e7 ffffffff80228db7 00007f298e413fff
> Call Trace:
>  [<ffffffffa012f5d3>] mmu_set_spte+0x98/0x302 [kvm]
>  [<ffffffffa012ffa3>] __direct_map+0xee/0x1b8 [kvm]
>  [<ffffffffa013014b>] tdp_page_fault+0xde/0x114 [kvm]
>  [<ffffffffa0130f16>] kvm_mmu_page_fault+0x19/0x81 [kvm]
>  [<ffffffffa012a64b>] kvm_arch_vcpu_ioctl_run+0x89b/0xaf2 [kvm]
>  [<ffffffffa0123540>] kvm_vcpu_ioctl+0xf1/0x46b [kvm]
>  [<ffffffff802bd249>] vfs_ioctl+0x21/0x6c
>  [<ffffffff802bd4b6>] do_vfs_ioctl+0x222/0x231
>  [<ffffffff802bd516>] sys_ioctl+0x51/0x73
>  [<ffffffff8020bfbb>] system_call_fastpath+0x16/0x1b
>  [<00007f298c3c3b77>] 0x7f298c3c3b77
> 
> 
> Code: 80 00 00 00 48 8b 34 c1 e8 0c ff ff ff 49 89 c1 48 8b 00 48 85 c0 75 17 48 8b 55 00 48 89 ee 48 c7 c7 2f db 13 a0 e8 6d cc 36 e0 <0f> 0b eb fe a8 01 75 2a 48 39 c5 74 19 48 8b 55 00 48 89 ee 48 
> RIP  [<ffffffffa012d8dc>] rmap_remove+0xc9/0x19e [kvm]
>  RSP <ffff880c7a1cbba8>
> ---[ end trace 91e1d7963caa34a7 ]---
> 
> hugepage info:
> HugePages_Total: 10000
> HugePages_Free:   7944
> HugePages_Rsvd:      0
> HugePages_Surp:      0
> Hugepagesize:     2048 kB
> 
> module info:
> filename:       /lib/modules/2.6.27.19-5-default/extra/kvm-intel.ko
> license:        GPL
> author:         Qumranet
> version:        kvm-devel
> srcversion:     9F14ECEFD8109654DFA20D2
> depends:        kvm
> vermagic:       2.6.27.19-5-default SMP mod_unload modversions 
> parm:           bypass_guest_pf:bool
> parm:           vpid:bool
> parm:           flexpriority:bool
> parm:           ept:bool
> parm:           emulate_invalid_guest_state:bool
> 
> filename:       /lib/modules/2.6.27.19-5-default/extra/kvm.ko
> license:        GPL
> author:         Qumranet
> version:        kvm-devel
> srcversion:     157F8CB48FC31BC2F44847B
> depends:        
> vermagic:       2.6.27.19-5-default SMP mod_unload modversions 
> parm:           oos_shadow:bool
> 
> 
> 
> -- 
> Ryan Harper
> Software Engineer; Linux Technology Center
> IBM Corp., Austin, Tx
> ryanh@us.ibm.com
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html