Re: BUG at mmu.c:615 from localhost migration using ept+hugetlbfs

From: Ryan Harper <ryanh@us.ibm.com>
To: Marcelo Tosatti <mtosatti@redhat.com>
Cc: Ryan Harper <ryanh@us.ibm.com>, Avi Kivity <avi@redhat.com>,
	kvm@vger.kernel.org
Subject: Re: BUG at mmu.c:615 from localhost migration using ept+hugetlbfs
Date: Tue, 9 Jun 2009 13:31:40 -0500	[thread overview]
Message-ID: <20090609183140.GM9872@us.ibm.com> (raw)
In-Reply-To: <20090609164036.GA10828@amt.cnet>

* Marcelo Tosatti <mtosatti@redhat.com> [2009-06-09 11:45]:
> Ryan,

Marcelo, thanks for taking a look.  Applied patch and reproduced,
included the new debug output.

> 
> On Fri, May 29, 2009 at 11:43:26AM -0500, Ryan Harper wrote:
> > Testing latest qemu-kvm.git and kvm-kmod.git, ept enabled and backing
> > guests with large pages trips a BUG in the mmu code.  If I disable ept,
> > but still use large pages, migration succeeds.  Reproduce with:
> > 
> > hugetlbfs setup:
> > % mkdir -p /hugetlbfs && mount -t hugetlbfs hugetlbfs /hugetlbfs
> > % echo 10000 > /proc/sys/vm/nr_hugepages
> > 
> > qemu commands:
> > 
> > guest a:
> > sudo x86_64-softmmu/qemu-system-x86_64 -L pc-bios -m 2048 -mempath /hugetlbfs -net nic -net tap -vnc :12 -monitor stdio -hda /scratch/images/rharper/rhel4u8-32-ide.raw
> > 
> > guest b:
> > sudo x86_64-softmmu/qemu-system-x86_64 -L pc-bios -m 2048 -mempath /hugetlbfs -net nic -net tap -vnc :13 -monitor stdio -hda /scratch/images/rharper/rhel4u8-32-ide.raw -incoming tcp:0:4444
> > 
> > Once the guest a is up, issued migrate command:
> > (qemu) migrate -d tcp:localhost:444
> > 
> > rmap_remove: ffff880a08e00098 c0336e65c0336e5b 0->BUG
> 				^^^^^^^^^^^^^^^^
> 
> This value looks very strange (bits 5:3 contain invalid value, for one).
> Don't have access to HW at the very moment, so it would be great if you
> had time to do a change equivalent to this and reproduce:
> 
> diff --git a/arch/x86/kvm/mmu.c b/arch/x86/kvm/mmu.c
> index 809cce0..ceb70b0 100644
> --- a/arch/x86/kvm/mmu.c
> +++ b/arch/x86/kvm/mmu.c
> @@ -1759,7 +1764,7 @@ static void mmu_set_spte(struct kvm_vcpu *vcpu, u64 *shadow_pte,
>  			child = page_header(pte & PT64_BASE_ADDR_MASK);
>  			mmu_page_remove_parent_pte(child, shadow_pte);
>  		} else if (pfn != spte_to_pfn(*shadow_pte)) {
> -			pgprintk("hfn old %lx new %lx\n",
> +			printk(KERN_ERR "hfn old %lx new %lx\n",
>  				 spte_to_pfn(*shadow_pte), pfn);
>  			rmap_remove(vcpu->kvm, shadow_pte);
>  		} else

hfn old 36e65c0336 new 472213
BUG: unable to handle kernel NULL pointer dereference at 0000000000000000
IP: [<ffffffffa012ca5e>] gfn_to_rmap+0x17/0x49 [kvm]
PGD 676517067 PUD 2de5cd067 PMD 0
Oops: 0000 [1] SMP
last sysfs file: /sys/devices/system/cpu/cpu15/cache/index2/shared_cpu_map
CPU 5
Modules linked in: kvm_intel(N) kvm(N) nls_iso8859_1 nls_cp437 vfat fat crc32c libcrc32c ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi scsi_transport_iscsi iscsi_ibft tun nfs lockd nfs_acl sunrpc ipv6 bridge stp cpufreq_conservative cpufreq_userspace cpufreq_powersave acpi_cpufreq microcode fuse loop dm_mod sr_mod cdrom cdc_ether thermal usb_storage usbnet processor sg rtc_cmos i2c_i801 shpchp rtc_core rtc_lib button mii i2c_core pcspkr pci_hotplug joydev bnx2 mptctl usbhid hid ff_memless uhci_hcd ehci_hcd usbcore sd_mod crc_t10dif edd fan thermal_sys hwmon ext3 mbcache jbd mptsas mptscsih mptbase scsi_transport_sas scsi_mod [last unloaded: kvm]
Supported: No
Pid: 31785, comm: qemu-system-x86 Tainted: G          2.6.27.19-5-default #1
RIP: 0010:[<ffffffffa012ca5e>]  [<ffffffffa012ca5e>] gfn_to_rmap+0x17/0x49 [kvm]
RSP: 0018:ffff880677499ba8  EFLAGS: 00010246
RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
RDX: 00000000000fffbd RSI: ffff8803808e08c0 RDI: 730000434950415f
RBP: 730000434950415f R08: 0000000000000023 R09: 0000000000000000
R10: 0000000000000046 R11: 0000000000000006 R12: ffff88067a971c60
R13: ffff8803808e0000 R14: 00000000c0336e5b R15: 0000000000000000
FS:  00007fee7dfbf950(0000) GS:ffff880c7cd938c0(0000) knlGS:0000000000000000
CS:  0010 DS: 002b ES: 002b CR0: 000000008005003b
CR2: 0000000000000000 CR3: 0000000674d09000 CR4: 00000000000026e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process qemu-system-x86 (pid: 31785, threadinfo ffff880677498000, task ffff8803425186c0)
Stack:  ffff880677499bb8 00000036e65c0336 ffff880472200098 ffffffffa012cb3b
 0000000000472213 0000000000000000 ffff880c7a148080 ffff880472200098
 ffff880c7a148080 ffffffffa012e5b2 ffff880677499ce8 00000005425186c0
Call Trace:
 [<ffffffffa012cb3b>] rmap_remove+0xab/0x19e [kvm]
 [<ffffffffa012e5b2>] mmu_set_spte+0xb0/0x316 [kvm]
 [<ffffffffa012ef2b>] direct_map_entry+0x7b/0x104 [kvm]
 [<ffffffffa012c1b1>] walk_shadow+0x8d/0xb7 [kvm]
 [<ffffffffa012deb1>] tdp_page_fault+0xf8/0x137 [kvm]
 [<ffffffffa012f802>] kvm_mmu_page_fault+0x19/0x80 [kvm]
 [<ffffffffa01be9b5>] handle_ept_violation+0xe0/0x17a [kvm_intel]
 [<ffffffffa012a012>] kvm_arch_vcpu_ioctl_run+0x4dd/0x6e5 [kvm]
 [<ffffffffa0123457>] kvm_vcpu_ioctl+0xf1/0x46b [kvm]
 [<ffffffff802bd249>] vfs_ioctl+0x21/0x6c
 [<ffffffff802bd4b6>] do_vfs_ioctl+0x222/0x231
 [<ffffffff802bd516>] sys_ioctl+0x51/0x73
 [<ffffffff8020bfbb>] system_call_fastpath+0x16/0x1b
 [<00007fee7eed5b77>] 0x7fee7eed5b77

Code: 31 ed 0f 18 08 eb a1 41 58 5b 5d 41 5c 41 5d 41 5e 41 5f c3 55 48 89 f5 53 89 d3 48 83 ec 08 e8 85 7a ff ff 85 db 48 89 c1 75 11 <48> 2b 28 48 8d 14 ed 00 00 00 00 48 03 50 18 eb 19 48 8b 00 48
RIP  [<ffffffffa012ca5e>] gfn_to_rmap+0x17/0x49 [kvm]
 RSP <ffff880677499ba8>
CR2: 0000000000000000
---[ end trace 6127eb9ebc2e7fb6 ]---

-- 
Ryan Harper
Software Engineer; Linux Technology Center
IBM Corp., Austin, Tx
ryanh@us.ibm.com