Re: kernel BUG at arch/x86/xen/mmu.c:1860!

Re: kernel BUG at arch/x86/xen/mmu.c:1860!

[MaoXiaoyun]

[-- Attachment #1.1: Type: text/plain, Size: 2223 bytes --]


Hi: 
     Unfortunately I met the exactly same bug today. With pvops kernel 2.6.32.36, and xen 4.0.1.
     Kernel Panic and serial log attached. 
 
     Our test cases is quite simple, on a single physical host, we start 12 HVMS(windows 2003),
each of the HVM reboot every 10minutes. 
 
     The bug is easy to hit on our 48G machine(in hours). But We haven't hit the bug in our 24G 
machine(we have three 24G machine, all works fine.)  -----Is is possible related to Memory capacity?
 
Taking a look at the serial output,  the Dom0 code is attempting to pin what it thins 
is a "PGT_l3_page_table", however the hypervisor returns -EINVAL because it actually  is a "PGT_writable_page". 
 
(XEN) mm.c:2364:d0 Bad type (saw 7400000000000001 != exp 4000 0000 0000 0000) for mfn 898a41 (pfn 9ca41)
(XEN) mm.c:2733:d0 Error while pinning mfn 898a41
 
And  before that quite a lot abnormal grant table log like :
 
(XEN) grant_table.c:1717:d0 Bad grant reference 4294965983
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) grant_table.c:1717:d0 Bad grant reference 4294965888
(XEN) grant_table.c:1717:d0 Bad grant reference 4294965983
 
It looks like something wrong with grant table.
 
Many thanks.
 
> From: Jeremy Fitzhardinge <jeremy@goop.org>
> Subject: Re: [Xen-devel] [SPAM] Re: kernel BUG at
> arch/x86/xen/mmu.c:1860! - ideas.
> To: Ian Campbell <Ian.Campbell@citrix.com>
> Cc: Dave Hunter <dave@ivt.com.au>, Teck Choon Giam
> <giamteckchoon@gmail.com>, "xen-devel@lists.xensource.com"
> xen-devel@lists.xensource.com
 
> On 04/06/2011 12:53 AM, Ian Campbell wrote:
> > Please don't top post.
> >
> > On Wed, 2011-04-06 at 00:20 +0100, Dave Hunter wrote:
> >> Is it likely that Debian would release an updated kernel in squeeze with
> >> this configuration? (sorry, this might not be the place to ask).
> > I doubt they will, enabling DEBUG_PAGEALLOC seems very much like a
> > workaround not a solution to me.
> 
> Yes, it will impose a pretty large performance overhead.
> 
> J
> 
> 

 		 	   		  

[-- Attachment #1.2: Type: text/html, Size: 3064 bytes --]

[-- Attachment #2: kernel.txt --]
[-- Type: text/plain, Size: 4390 bytes --]

Apr  8 12:19:47 r14a11017 kernel: ------------[ cut here ]------------
Apr  8 12:19:47 r14a11017 kernel: kernel BUG at arch/x86/xen/mmu.c:1872!
Apr  8 12:19:47 r14a11017 kernel: invalid opcode: 0000 [#1] SMP
Apr  8 12:19:47 r14a11017 kernel: last sysfs file: /sys/hypervisor/properties/capabilities
Apr  8 12:19:47 r14a11017 kernel: CPU 0
Apr  8 12:19:47 r14a11017 kernel: Modules linked in: 8021q garp blktap xen_netback xen_blkback blkback_pagemap nbd bridge stp llc autofs4 ipmi_devintf ipmi_si ipmi_msghandler lockd sunrpc bonding ipv6 xenfs dm_multipath video output sbs sbshc parport_pc lp parport ses enclosure snd_seq_dummy snd_seq_oss bnx2 snd_seq_midi_event serio_raw snd_seq snd_seq_device snd_pcm_oss snd_mixer_oss snd_pcm snd_timer i2c_i801 iTCO_wdt i2c_core snd soundcore snd_page_alloc iTCO_vendor_support pata_acpi ata_generic pcspkr ata_piix shpchp mptsas mptscsih mptbase [last unloaded: freq_table]
Apr  8 12:19:47 r14a11017 kernel: Pid: 15769, comm: sh Not tainted 2.6.32.36xen #1 Tecal RH2285
Apr  8 12:19:47 r14a11017 kernel: RIP: e030:[<ffffffff8100cebc>]  [<ffffffff8100cebc>] pin_pagetable_pfn+0x36/0x3c
Apr  8 12:19:47 r14a11017 kernel: RSP: e02b:ffff88001eb7baa8  EFLAGS: 00010282
Apr  8 12:19:47 r14a11017 kernel: RAX: 00000000ffffffea RBX: 000000000007b307 RCX: 0000000000000001
Apr  8 12:19:47 r14a11017 kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88001eb7baa8
Apr  8 12:19:47 r14a11017 kernel: RBP: ffff88001eb7bac8 R08: 0000000000000420 R09: ffff880000000000
Apr  8 12:19:47 r14a11017 kernel: R10: 0000000000007ff0 R11: ffff88008fc97248 R12: ffff88002840b000
Apr  8 12:19:47 r14a11017 kernel: R13: 000000000007b484 R14: 0000000000000003 R15: ffff88009b090000
Apr  8 12:19:47 r14a11017 kernel: FS:  00007fe8bbc656e0(0000) GS:ffff88002803b000(0000) knlGS:0000000000000000
Apr  8 12:19:47 r14a11017 kernel: CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
Apr  8 12:19:47 r14a11017 kernel: CR2: 00000000006bb338 CR3: 000000007b307000 CR4: 0000000000002660
Apr  8 12:19:47 r14a11017 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Apr  8 12:19:47 r14a11017 kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Apr  8 12:19:47 r14a11017 kernel: Process sh (pid: 15769, threadinfo ffff88001eb7a000, task ffff88009b090000)
Apr  8 12:19:47 r14a11017 kernel: Stack:
Apr  8 12:19:47 r14a11017 kernel:  0000000000000000 00000000004b7484 000000011eb7bac8 000000000007b307
Apr  8 12:19:47 r14a11017 kernel: <0> ffff88001eb7baf8 ffffffff8100e8ef ffff88012e4fb100 ffff88000fb5e018
Apr  8 12:19:47 r14a11017 kernel: <0> 000000000007b484 00000000006bb338 ffff88001eb7bb08 ffffffff8100e935
Apr  8 12:19:47 r14a11017 kernel: Call Trace:
Apr  8 12:19:47 r14a11017 kernel:  [<ffffffff8100e8ef>] xen_alloc_ptpage+0x8d/0x96
Apr  8 12:19:47 r14a11017 kernel:  [<ffffffff8100e935>] xen_alloc_pte+0x13/0x15
Apr  8 12:19:47 r14a11017 kernel:  [<ffffffff810eb702>] __pte_alloc+0x7f/0xdc
Apr  8 12:19:47 r14a11017 kernel:  [<ffffffff810e90bd>] ? pmd_offset+0x13/0x3c
Apr  8 12:19:47 r14a11017 kernel:  [<ffffffff810eb818>] handle_mm_fault+0xb9/0x771
Apr  8 12:19:47 r14a11017 kernel:  [<ffffffff810f08fd>] ? vma_link+0x7c/0xa4
Apr  8 12:19:47 r14a11017 kernel:  [<ffffffff810f13b0>] ? mmap_region+0x322/0x42b
Apr  8 12:19:47 r14a11017 kernel:  [<ffffffff8100f169>] ? xen_force_evtchn_callback+0xd/0xf
Apr  8 12:19:47 r14a11017 kernel:  [<ffffffff81449701>] do_page_fault+0x21c/0x288
Apr  8 12:19:47 r14a11017 kernel:  [<ffffffff81447695>] page_fault+0x25/0x30
Apr  8 12:19:47 r14a11017 kernel:  [<ffffffff81222a39>] ? __clear_user+0x33/0x55
Apr  8 12:19:47 r14a11017 kernel:  [<ffffffff81222a1d>] ? __clear_user+0x17/0x55
Apr  8 12:19:47 r14a11017 kernel:  [<ffffffff81222a8b>] clear_user+0x30/0x38
Apr  8 12:19:47 r14a11017 kernel:  [<ffffffff8115139a>] load_elf_binary+0x5d5/0x17ef
Apr  8 12:19:47 r14a11017 kernel:  [<ffffffff811f4648>] ? process_measurement+0xc0/0xd7
Apr  8 12:19:47 r14a11017 kernel:  [<ffffffff81150dc5>] ? load_elf_binary+0x0/0x17ef
Apr  8 12:19:47 r14a11017 kernel:  [<ffffffff81113094>] search_binary_handler+0xc8/0x255
Apr  8 12:19:47 r14a11017 kernel:  [<ffffffff81114362>] do_execve+0x1c3/0x29e
Apr  8 12:19:47 r14a11017 kernel:  [<ffffffff8101155d>] sys_execve+0x43/0x5d
Apr  8 12:19:47 r14a11017 kernel:  [<ffffffff810131ca>] stub_execve+0x6a/0xc0

[-- Attachment #3: serial.txt --]
[-- Type: text/plain, Size: 4360 bytes --]

(XEN) grant_table.c:1717:d0 Bad grant reference 4294965983
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) grant_table.c:1717:d0 Bad grant reference 4294965888
(XEN) grant_table.c:1717:d0 Bad grant reference 4294965983
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) grant_table.c:1717:d0 Bad grant reference 4294965983
(XEN) grant_table.c:1717:d0 Bad grant reference 4294965983
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) printk: 1 messages suppressed.
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) printk: 1 messages suppressed.
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) printk: 5 messages suppressed.
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) printk: 14 messages suppressed.
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) printk: 7 messages suppressed.
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) printk: 1 messages suppressed.
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) grant_table.c:1717:d0 Bad grant reference 4294901760
(XEN) printk: 13 messages suppressed.
(XEN) grant_table.c:1717:d0 Bad grant reference 4294901760
(XEN) printk: 59 messages suppressed.
(XEN) grant_table.c:1717:d0 Bad grant reference 4294901760
(XEN) printk: 81 messages suppressed.
(XEN) grant_table.c:1717:d0 Bad grant reference 4294901760
(XEN) printk: 75 messages suppressed.
(XEN) grant_table.c:1717:d0 Bad grant reference 4294901760
(XEN) printk: 79 messages suppressed.
(XEN) grant_table.c:1717:d0 Bad grant reference 4294901760
(XEN) printk: 81 messages suppressed.
(XEN) grant_table.c:1717:d0 Bad grant reference 4294901760
(XEN) printk: 33 messages suppressed.
(XEN) grant_table.c:1717:d0 Bad grant reference 4294901760
(XEN) printk: 9 messages suppressed.
(XEN) grant_table.c:1717:d0 Bad grant reference 4294901765
(XEN) printk: 7 messages suppressed.
(XEN) grant_table.c:1717:d0 Bad grant reference 4294901760
(XEN) printk: 1 messages suppressed.
(XEN) grant_table.c:1717:d0 Bad grant reference 4294901760
(XEN) grant_table.c:1717:d0 Bad grant reference 4294901765
(XEN) grant_table.c:1717:d0 Bad grant reference 4294901760
(XEN) grant_table.c:1717:d0 Bad grant reference 4294901765
(XEN) grant_table.c:1717:d0 Bad grant reference 4294901760
(XEN) grant_table.c:1717:d0 Bad grant reference 4294901765
(XEN) grant_table.c:1717:d0 Bad grant reference 4294901760
(XEN) grant_table.c:1717:d0 Bad grant reference 4294901765
(XEN) printk: 10 messages suppressed.
(XEN) mm.c:2364:d0 Bad type (saw 7400000000000001 != exp 4000 0000 0000 0000) for mfn 898a41 (pfn 9ca41)
(XEN) mm.c:2733:d0 Error while pinning mfn 898a41
                                 8000000000000000 
(XEN) mm.c:2364:d0 Bad type (saw 7400000000000001 != exp 4000000000000000) for mfn 871443 (pfn 75443)
(XEN) mm.c:2733:d0 Error while pinning mfn 871443
(XEN) mm.c:2364:d0 Bad type (saw 7400000000000001 != exp 4000000000000000) for mfn 898a41 (pfn 9ca41)
(XEN) mm.c:2500:d0 Error while installing new baseptr 898a41
(XEN) mm.c:2364:d0 Bad type (saw 7400000000000001 != exp 4000000000000000) for mfn 871443 (pfn 75443)
(XEN) mm.c:2825:d0 Error while installing new mfn 871443
(XEN) mm.c:2364:d0 Bad type (saw 4400000000000001 != exp 7000000000000000) for mfn 899551 (pfn 9d551)
(XEN) mm.c:860:d0 Error getting mfn 899551 (pfn 9d551) from L1 entry 8000000899551063 for l1e_owner=0, pg_owner=0

[-- Attachment #4: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

RE: kernel BUG at arch/x86/xen/mmu.c:1860!

[MaoXiaoyun]

[-- Attachment #1.1: Type: text/plain, Size: 3144 bytes --]


HI:
 
     As I go through the code with log, I noticed that  the log: 
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
 
is from xen/common/grant_table.c:266, which is in function _set_status_v1()
so it looks like kernel 2.6.32 use grant table version 1.
 
While in 2.6.31. driver/xen/grant-table.c, I noticed function gnttab_request_version()
which looks like 2.6.31 require grant version 2. But this function cannot be found 
in 2.6.32.
 
Is this correct?
 
Thanks.
 
 
>--------------------------------------------------------------------------------
>From: tinnycloud@hotmail.com
>To: xen-devel@lists.xensource.com
>CC: dave@ivt.com.au; ian.campbell@citrix.com; giamteckchoon@gmail.com; konrad.wilk@oracle.com; jeremy@goop.org
>Subject: Re: kernel BUG at arch/x86/xen/mmu.c:1860!
>Date: Fri, 8 Apr 2011 19:24:35 +0800
>
>Hi: 
>     Unfortunately I met the exactly same bug today. With pvops kernel 2.6.32.36, and xen 4.0.1.
>     Kernel Panic and serial log attached. 
> 
>     Our test cases is quite simple, on a single physical host, we start 12 HVMS(windows 2003),
>each of the HVM reboot every 10minutes. 
> 
>     The bug is easy to hit on our 48G machine(in hours). But We haven't hit the bug in our 24G 
>machine(we have three 24G machine, all works fine.)  -----Is is possible related to Memory capacity?
> 
>Taking a look at the serial output,  the Dom0 code is attempting to pin what it thins 
>is a "PGT_l3_page_table", however the hypervisor returns -EINVAL because it actually  is a "PGT_writable_page". 
> 
>(XEN) mm.c:2364:d0 Bad type (saw 7400000000000001 != exp 4000 0000 0000 0000) for mfn 898a41 (pfn 9ca41)
>(XEN) mm.c:2733:d0 Error while pinning mfn 898a41
> 
>And  before that quite a lot abnormal grant table log like :
> 
>(XEN) grant_table.c:1717:d0 Bad grant reference 4294965983
>(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
>(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
>(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
>(XEN) grant_table.c:1717:d0 Bad grant reference 4294965888
>(XEN) grant_table.c:1717:d0 Bad grant reference 4294965983
> 
>It looks like something wrong with grant table.
> 
>Many thanks.
> 
>> From: Jeremy Fitzhardinge <jeremy@goop.org>
>> Subject: Re: [Xen-devel] [SPAM] Re: kernel BUG at
>> arch/x86/xen/mmu.c:1860! - ideas.
>> To: Ian Campbell <Ian.Campbell@citrix.com>
>> Cc: Dave Hunter <dave@ivt.com.au>, Teck Choon Giam
>> <giamteckchoon@gmail.com>, "xen-devel@lists.xensource.com"
>> xen-devel@lists.xensource.com
> 
>> On 04/06/2011 12:53 AM, Ian Campbell wrote:
>> > Please don't top post.
>> >
>> > On Wed, 2011-04-06 at 00:20 +0100, Dave Hunter wrote:
>> >> Is it likely that Debian would release an updated kernel in squeeze with
>> >> this configuration? (sorry, this might not be the place to ask).
>> > I doubt they will, enabling DEBUG_PAGEALLOC seems very much like a
>> > workaround not a solution to me.
>> 
>> Yes, it will impose a pretty large performance overhead.
>> 
>> J
>> 
>> 
> 		 	   		  

[-- Attachment #1.2: Type: text/html, Size: 4560 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

kernel BUG at arch/x86/xen/mmu.c:1872

[MaoXiaoyun]

[-- Attachment #1.1: Type: text/plain, Size: 537 bytes --]


Hi Konrad & Jeremy:
 
     I'd like to open this BUG in a new thread, since the old thread is too long for read.
     
     We recently want to upgrade our kernel to 2.6.32, but unfortunately, we confront a kernel crash bug.
Our test case is simple, start 24 win2003 HVMS on our physical machine, and each HVM reboot 
every 15minutes. The kernel will crash in half an hour.(That is crash on VM second starts).
 
Our test go much further.
We test different kernel version.
2.6.32.10
2.6.32.10
2.6.32.10
 
      		 	   		  

[-- Attachment #1.2: Type: text/html, Size: 900 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

kernel BUG at arch/x86/xen/mmu.c:1872

[MaoXiaoyun]

[-- Attachment #1.1: Type: text/plain, Size: 2514 bytes --]


(Please ignore my last mail, sent by type..)
 
Hi Konrad & Jeremy:
 
     I'd like to open this BUG in a new thread, since the old thread is too long for easy read.
     
     We recently want to upgrade our kernel to 2.6.32, but unfortunately, we confront a kernel crash bug.
Our test case is simple, start 24 win2003 HVMS on our physical machine, and each HVM reboot 
every 15minutes. The kernel will crash in half an hour.(That is crash on VM second starts).
 
Our test go much further.
We test different kernel version.
2.6.32.10  http://git.kernel.org/?p=linux/kernel/git/jeremy/xen.git;a=commit;h=d945b014ac5df9592c478bf9486d97e8914aab59
2.6.32.11  http://git.kernel.org/?p=linux/kernel/git/jeremy/xen.git;a=commit;h=27f948a3bf365a5bc3d56119637a177d41147815
2.6.32.12  http://git.kernel.org/?p=linux/kernel/git/jeremy/xen.git;a=commit;h=ba739f9abd3f659b907a824af1161926b420a2ce
2.6.32.13  http://git.kernel.org/?p=linux/kernel/git/jeremy/xen.git;a=commit;h=f6fe6583b77a49b569eef1b66c3d761eec2e561b
2.6.32.15  http://git.kernel.org/?p=linux/kernel/git/jeremy/xen.git;a=commit;h=27ed1b0e0dae5f1d5da5c76451bc84cb529128bd
2.6.32.21  http://git.kernel.org/?p=linux/kernel/git/jeremy/xen.git;a=commit;h=69e50db231723596ed8ef9275d0068d6697f466a
 
There are basic three different result we met.
 
i1) grant table issue
The host still function, but use xm  dmesg, we have abnormal log.
please refer to the attched log of grant table
 
i2) kernel crash on a different place.
Host die during the test, after reboot, we can see nothing abnormal in /var/log/messages
 
i3) kernel BUG at arch/x86/xen/mmu.c:1872; 
Host die during the test, after reboot, we see the crash log in messages, refer to the attached log of 2.6.32.36
Summary of the test result, can be classified in two:
 
1) 2.6.32.10
30 machines involved the test, and three has issue (i1), and two has issue (i2), *no* issue (i3)
Other machines run tests successfully till now, more than 8 hours
 
2)2.6.32.11 or later version.
Each version containers 10 machine for tests, and all machine crashed in less than half an hour.
 
Conclusion:
1) grant table issue exists in all kernel version
2) kernerl crash at different place may exist in all kernel versions, but not happen so frequently, 2 out of 30
3) We observe the major difference of issue i3), from the test, it looks like it is introduced between the version
2.6.32.10 and 2.6.32.11.
 
Hope this help to locate the bug.
Many thanks.
 
      		 	   		  

[-- Attachment #1.2: Type: text/html, Size: 4312 bytes --]

[-- Attachment #2: kernel_crash_at_different_place.txt --]
[-- Type: text/plain, Size: 20116 bytes --]


=========================crash log for machine one in 2.6.32.10 ================================================

INIT: Id "s0" respawning too fast: disabled for 5 minutes
blktap_sysfs_destroy
blktap_sysfs_create: adding attributes for dev ffff8800b7a62600
blktap_sysfs_destroy
blktap_sysfs_create: adding attributes for dev ffff8800abb3d200
blktap_sysfs_destroy
blktap_sysfs_create: adding attributes for dev ffff8800ab3a3000
blktap_sysfs_destroy
blktap_sysfs_create: adding attributes for dev ffff8800a7ef8e00
blktap_sysfs_destroy
blktap_sysfs_create: adding attributes for dev ffff8800bd224e00
blktap_sysfs_destroy
blktap_sysfs_create: adding attributes for dev ffff8800bf09f400
blktap_sysfs_destroy
blktap_sysfs_destroy
blktap_sysfs_create: adding attributes for dev ffff8800a96b3c00
blktap_sysfs_create: adding attributes for dev ffff8800bf09ec00
INIT: Id "s0" respawning too fast: disabled for 5 minutes
blktap_sysfs_destroy
blktap_sysfs_create: adding attributes for dev ffff8800b8daae00
blktap_sysfs_destroy
blktap_sysfs_create: adding attributes for dev ffff8800b0ea5400
blktap_sysfs_destroy
blktap_sysfs_create: adding attributes for dev ffff8800b8dab200
blktap_sysfs_destroy
blktap_sysfs_create: adding attributes for dev ffff8800b8daa600
INIT: Id "s0" respawning too fast: disabled for 5 minutes
blktap_sysfs_destroy
blktap_sysfs_create: adding attributes for dev ffff8800ab933200
blktap_sysfs_destroy
blktap_sysfs_create: adding attributes for dev ffff8800b6514000
BUG: scheduling while atomic: swapper/0/0x10000100
Modules linked in: 8021q garp xen_netback xen_blkback blktap blkback_pagemap nbd bridge stp llc autofs4 ipmi_devintf ipmi_si ipmi_msghandler lockd sunrpc bonding ipv6 xenfs dm_multipath video output sbs sbshc parport_pc lp parport ses enclosure serio_raw bnx2 snd_seq_dummy snd_seq_oss snd_seq_midi_event snd_seq snd_seq_device snd_pcm_oss snd_mixer_oss snd_pcm snd_timer snd pata_acpi soundcore ata_generic snd_page_alloc pcspkr iTCO_wdt iTCO_vendor_support i2c_i801 i2c_core ata_piix shpchp mptsas mptscsih mptbase [last unloaded: freq_table]
CPU 0:
Modules linked in: 8021q garp xen_netback xen_blkback blktap blkback_pagemap nbd bridge stp llc autofs4 ipmi_devintf ipmi_si ipmi_msghandler lockd sunrpc bonding ipv6 xenfs dm_multipath video output sbs sbshc parport_pc lp parport ses enclosure serio_raw bnx2 snd_seq_dummy snd_seq_oss snd_seq_midi_event snd_seq snd_seq_device snd_pcm_oss snd_mixer_oss snd_pcm snd_timer snd pata_acpi soundcore ata_generic snd_page_alloc pcspkr iTCO_wdt iTCO_vendor_support i2c_i801 i2c_core ata_piix shpchp mptsas mptscsih mptbase [last unloaded: freq_table]
Pid: 0, comm: swapper Not tainted 2.6.32.10xen #1 Tecal RH2285          
RIP: e030:[<ffffffff810093aa>]  [<ffffffff810093aa>] hypercall_page+0x3aa/0x1000
RSP: e02b:ffffffff81663ed8  EFLAGS: 00000246
RAX: 0000000000000000 RBX: ffffffff81662000 RCX: ffffffff810093aa
RDX: ffffffff8100f23f RSI: 0000000000000000 RDI: 0000000000000001
RBP: ffffffff81663ef0 R08: 0000000000000000 R09: ffff880028092e08
R10: ffff880159558000 R11: 0000000000000246 R12: 0000000000000000
R13: 6db6db6db6db6db7 R14: ffffffff817d73a0 R15: 0000000000000000
FS:  00007f50215db6e0(0000) GS:ffff88002802c000(0000) knlGS:0000000000000000
CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007f449672d000 CR3: 00000000a7b0a000 CR4: 0000000000002660
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Call Trace:
 [<ffffffff8100ebbf>] ? xen_safe_halt+0x10/0x1a
 [<ffffffff8100c102>] xen_idle+0x3c/0x46
 [<ffffffff81010cbd>] cpu_idle+0x5d/0x8c
 [<ffffffff81426742>] rest_init+0x66/0x68
 [<ffffffff8179fd8d>] start_kernel+0x3ef/0x3fb
 [<ffffffff8179f2c3>] x86_64_start_reservations+0xae/0xb2
 [<ffffffff817a2cb3>] xen_start_kernel+0x4c0/0x4c7
divide error: 0000 [#1] SMP 
last sysfs file: /sys/class/net/d342dd/address
CPU 0 
Modules linked in: 8021q garp xen_netback xen_blkback blktap blkback_pagemap nbd bridge stp llc autofs4 ipmi_devintf ipmi_si ipmi_msghandler lockd sunrpc bonding ipv6 xenfs dm_multipath video output sbs sbshc parport_pc lp parport ses enclosure serio_raw bnx2 snd_seq_dummy snd_seq_oss snd_seq_midi_event snd_seq snd_seq_device snd_pcm_oss snd_mixer_oss snd_pcm snd_timer snd pata_acpi soundcore ata_generic snd_page_alloc pcspkr iTCO_wdt iTCO_vendor_support i2c_i801 i2c_core ata_piix shpchp mptsas mptscsih mptbase [last unloaded: freq_table]
Pid: 0, comm: swapper Not tainted 2.6.32.10xen #1 Tecal RH2285          
RIP: e030:[<ffffffff8104ee57>]  [<ffffffff8104ee57>] find_busiest_group+0x37d/0x721
RSP: e02b:ffff88002802fc90  EFLAGS: 00010246
RAX: 0000000000003c00 RBX: 0000000000000000 RCX: ffff880028041501
RDX: 0000000000000000 RSI: 0000000000000040 RDI: 0000000000000040
RBP: ffff88002802fdf0 R08: 0000000000000000 R09: ffff88002803be08
R10: 0000000000000000 R11: 0000000000000001 R12: 0000000000000040
R13: ffff88002803bdf0 R14: ffff88002803bce0 R15: 0000000000000000
FS:  00007f50215db6e0(0000) GS:ffff88002802c000(0000) knlGS:0000000000000000
CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007f449672d000 CR3: 00000000a7b0a000 CR4: 0000000000002660
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process swapper (pid: 0, threadinfo ffffffff81662000, task ffffffff816e8980)
Stack:
 00001a0e95f9ec56 ffff88002803ba48 ffff88002802fe4c 000000008100eb79
<0> ffff88002802fe40 000000008100f252 0000000000003c00 0000000000000001
<0> ffff88002803be00 0000000100000010 ffff88002803bdf0 ffffffff8102edf9
Call Trace:
 <IRQ> 
 [<ffffffff8102edf9>] ? pvclock_clocksource_read+0x47/0x80
 [<ffffffff8100f23f>] ? xen_restore_fl_direct_end+0x0/0x1
 [<ffffffff8104fc43>] rebalance_domains+0x17b/0x45b
 [<ffffffff810480f2>] ? wake_up_process+0x15/0x17
 [<ffffffff8104ff63>] run_rebalance_domains+0x40/0xc5
 [<ffffffff81059b9b>] __do_softirq+0xd2/0x194
 [<ffffffff81012eac>] call_softirq+0x1c/0x30
 [<ffffffff81014627>] do_softirq+0x46/0x87
 [<ffffffff81059c98>] irq_exit+0x3b/0x7a
 [<ffffffff812868ab>] xen_evtchn_do_upcall+0x156/0x172
 [<ffffffff81012efe>] xen_do_hypervisor_callback+0x1e/0x30
 <EOI> 
 [<ffffffff810093aa>] ? hypercall_page+0x3aa/0x1000
 [<ffffffff8100f23f>] ? xen_restore_fl_direct_end+0x0/0x1
 [<ffffffff810093aa>] ? hypercall_page+0x3aa/0x1000
 [<ffffffff8100ebbf>] ? xen_safe_halt+0x10/0x1a
 [<ffffffff8100c102>] ? xen_idle+0x3c/0x46
 [<ffffffff81010cbd>] ? cpu_idle+0x5d/0x8c
 [<ffffffff81426742>] ? rest_init+0x66/0x68
 [<ffffffff8179fd8d>] ? start_kernel+0x3ef/0x3fb
 [<ffffffff8179f2c3>] ? x86_64_start_reservations+0xae/0xb2
 [<ffffffff817a2cb3>] ? xen_start_kernel+0x4c0/0x4c7
Code: 83 7d 10 00 74 0c 48 8b 5d 10 c7 03 00 00 00 00 eb 70 41 8b 55 08 48 8b 45 a8 48 89 d3 48 c1 a5 d0 fe ff ff 0a 48 c1 e0 0a 31 d2 <48> f7 f3 48 89 45 a0 48 8b 85 08 ff ff ff 48 29 85 00 ff ff ff 
RIP  [<ffffffff8104ee57>] find_busiest_group+0x37d/0x721
 RSP <ffff88002802fc90>
---[ end trace 7c3e3b64ca341f0a ]---
divide error: 0000 [#2] SMP 
last sysfs file: /sys/class/net/d342dd/address
CPU 2 
Modules linked in: 8021q garp xen_netback xen_blkback blktap blkback_pagemap nbd bridge stp llc autofs4 ipmi_devintf ipmi_si ipmi_msghandler lockd sunrpc bonding ipv6 xenfs dm_multipath video output sbs sbshc parport_pc lp parport ses enclosure serio_raw bnx2 snd_seq_dummy snd_seq_oss snd_seq_midi_event snd_seq snd_seq_device snd_pcm_oss snd_mixer_oss snd_pcm snd_timer snd pata_acpi soundcore ata_generic snd_page_alloc pcspkr iTCO_wdt iTCO_vendor_support i2c_i801 i2c_core ata_piix shpchp mptsas mptscsih mptbase [last unloaded: freq_table]
Pid: 0, comm: swapper Tainted: G      D    2.6.32.10xen #1 Tecal RH2285          
RIP: e030:[<ffffffff8104ee57>]  [<ffffffff8104ee57>] find_busiest_group+0x37d/0x721
RSP: e02b:ffff880028069c90  EFLAGS: 00010246
RAX: 0000000000000000 RBX: 0000000000000000 RCX: ffff880028041500
RDX: 0000000000000000 RSI: 0000000000000040 RDI: 0000000000000040
RBP: ffff880028069df0 R08: 0000000000000000 R09: ffff88002803be08
R10: ffff880028069d28 R11: ffff88015f8f7e48 R12: 0000000000000040
R13: ffff88002803bdf0 R14: ffff880028058ce0 R15: 0000000000000001
FS:  00007faad3ba8730(0000) GS:ffff880028066000(0000) knlGS:0000000000000000
CS:  e033 DS: 002b ES: 002b CR0: 000000008005003b
CR2: 00007fd1e7ed9000 CR3: 000000015510a000 CR4: 0000000000002660
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process swapper (pid: 0, threadinfo ffff88015f8f6000, task ffff88015f8e4410)
Stack:
 00001a0e95fdb97b ffff880028075a48 ffff880028069e4c 000000008100eb79
<0> ffff880028069e40 000000008100f252 0000000000003c00 ffffffff00000000
<0> ffff88002803be00 0000000100000010 ffff880028058df0 0000000000000002
Call Trace:
 <IRQ> 
 [<ffffffff8107e36d>] ? tick_dev_program_event+0x2f/0xa1
 [<ffffffff8104fc43>] rebalance_domains+0x17b/0x45b
 [<ffffffff8104ff98>] run_rebalance_domains+0x75/0xc5
 [<ffffffff81059b9b>] __do_softirq+0xd2/0x194
 [<ffffffff81012eac>] call_softirq+0x1c/0x30
 [<ffffffff81014627>] do_softirq+0x46/0x87
 [<ffffffff81059c98>] irq_exit+0x3b/0x7a
 [<ffffffff812868ab>] xen_evtchn_do_upcall+0x156/0x172
 [<ffffffff81012efe>] xen_do_hypervisor_callback+0x1e/0x30
 <EOI> 
 [<ffffffff810093aa>] ? hypercall_page+0x3aa/0x1000
 [<ffffffff810093aa>] ? hypercall_page+0x3aa/0x1000
 [<ffffffff8100ebbf>] ? xen_safe_halt+0x10/0x1a
 [<ffffffff8100c102>] ? xen_idle+0x3c/0x46
 [<ffffffff81010cbd>] ? cpu_idle+0x5d/0x8c
 [<ffffffff81432d68>] ? cpu_bringup_and_idle+0x13/0x15
Code: 83 7d 10 00 74 0c 48 8b 5d 10 c7 03 00 00 00 00 eb 70 41 8b 55 08 48 8b 45 a8 48 89 d3 48 c1 a5 d0 fe ff ff 0a 48 c1 e0 0a 31 d2 <48> f7 f3 48 89 45 a0 48 8b 85 08 ff ff ff 48 29 85 00 ff ff ff 
RIP  [<ffffffff8104ee57>] find_busiest_group+0x37d/0x721
 RSP <ffff880028069c90>
---[ end trace 7c3e3b64ca341f0b ]---
Kernel panic - not syncing: Fatal exception in interrupt
Pid: 0, comm: swapper Tainted: G      D    2.6.32.10xen #1
Call Trace:
 <IRQ>  [<ffffffff810402a5>] ? ftrace_profile_enable_sched_process_exit+0x10/0x17
 [<ffffffff81052eea>] panic+0xe0/0x198
 [<ffffffff8100eb79>] ? xen_force_evtchn_callback+0xd/0xf
 [<ffffffff8100f252>] ? check_events+0x12/0x20
 [<ffffffff8100f23f>] ? xen_restore_fl_direct_end+0x0/0x1
 [<ffffffff810402a5>] ? ftrace_profile_enable_sched_process_exit+0x10/0x17
 [<ffffffff81052b43>] ? print_oops_end_marker+0x23/0x25
 [<ffffffff810402a5>] ? ftrace_profile_enable_sched_process_exit+0x10/0x17
 [<ffffffff8143d2b5>] oops_end+0xb6/0xc6
 [<ffffffff810156c1>] die+0x5a/0x63
 [<ffffffff8143cb8c>] do_trap+0x115/0x124
 [<ffffffff81013610>] do_divide_error+0x96/0x9f
 [<ffffffff8104ee57>] ? find_busiest_group+0x37d/0x721
 [<ffffffff8100f23f>] ? xen_restore_fl_direct_end+0x0/0x1
 [<ffffffff8143c3da>] ? _spin_unlock_irqrestore+0x15/0x17
 [<ffffffff813937a0>] ? skb_release_data+0xab/0xb0
 [<ffffffff8100eb79>] ? xen_force_evtchn_callback+0xd/0xf
 [<ffffffff8100f252>] ? check_events+0x12/0x20
 [<ffffffff81012adb>] divide_error+0x1b/0x20
 [<ffffffff8104ee57>] ? find_busiest_group+0x37d/0x721
 [<ffffffff8107e36d>] ? tick_dev_program_event+0x2f/0xa1
 [<ffffffff8104fc43>] rebalance_domains+0x17b/0x45b
 [<ffffffff8104ff98>] run_rebalance_domains+0x75/0xc5
 [<ffffffff81059b9b>] __do_softirq+0xd2/0x194
 [<ffffffff81012eac>] call_softirq+0x1c/0x30
 [<ffffffff81014627>] do_softirq+0x46/0x87
 [<ffffffff81059c98>] irq_exit+0x3b/0x7a
 [<ffffffff812868ab>] xen_evtchn_do_upcall+0x156/0x172
 [<ffffffff81012efe>] xen_do_hypervisor_callback+0x1e/0x30
 <EOI>  [<ffffffff810093aa>] ? hypercall_page+0x3aa/0x1000
 [<ffffffff810093aa>] ? hypercall_page+0x3aa/0x1000
 [<ffffffff8100ebbf>] ? xen_safe_halt+0x10/0x1a
 [<ffffffff8100c102>] ? xen_idle+0x3c/0x46
 [<ffffffff81010cbd>] ? cpu_idle+0x5d/0x8c
 [<ffffffff81432d68>] ? cpu_bringup_and_idle+0x13/0x15
Kernel panic - not syncing: Fatal exception in interrupt
Pid: 0, comm: swapper Tainted: G      D    2.6.32.10xen #1
Call Trace:
 <IRQ>  [<ffffffff81052eea>] panic+0xe0/0x198
 [<ffffffff814300b2>] ? megaraid_probe_one+0xf25/0x116d
 [<ffffffff8100eb79>] ? xen_force_evtchn_callback+0xd/0xf
 [<ffffffff8100f252>] ? check_events+0x12/0x20
 [<ffffffff8100f23f>] ? xen_restore_fl_direct_end+0x0/0x1
 [<ffffffff81052b43>] ? print_oops_end_marker+0x23/0x25
 [<ffffffff8143d2b5>] oops_end+0xb6/0xc6
 [<ffffffff810156c1>] die+0x5a/0x63
 [<ffffffff8143cb8c>] do_trap+0x115/0x124
 [<ffffffff81013610>] do_divide_error+0x96/0x9f
 [<ffffffff8104ee57>] ? find_busiest_group+0x37d/0x721
 [<ffffffff813934b9>] ? __kfree_skb+0x79/0x7d
 [<ffffffff8100f23f>] ? xen_restore_fl_direct_end+0x0/0x1
 [<ffffffff810fd88b>] ? kmem_cache_free+0x88/0xbb
 [<ffffffff813934b9>] ? __kfree_skb+0x79/0x7d
 [<ffffffff81012adb>] divide_error+0x1b/0x20
 [<ffffffff8104ee57>] ? find_busiest_group+0x37d/0x721
 [<ffffffff8102edf9>] ? pvclock_clocksource_read+0x47/0x80
 [<ffffffff8100f23f>] ? xen_restore_fl_direct_end+0x0/0x1
 [<ffffffff8104fc43>] rebalance_domains+0x17b/0x45b
 [<ffffffff810480f2>] ? wake_up_process+0x15/0x17
 [<ffffffff8104ff63>] run_rebalance_domains+0x40/0xc5
 [<ffffffff81059b9b>] __do_softirq+0xd2/0x194
 [<ffffffff81012eac>] call_softirq+0x1c/0x30
 [<ffffffff81014627>] do_softirq+0x46/0x87
 [<ffffffff81059c98>] irq_exit+0x3b/0x7a
 [<ffffffff812868ab>] xen_evtchn_do_upcall+0x156/0x172
 [<ffffffff81012efe>] xen_do_hypervisor_callback+0x1e/0x30
 <EOI>  [<ffffffff810093aa>] ? hypercall_page+0x3aa/0x1000
 [<ffffffff8100f23f>] ? xen_restore_fl_direct_end+0x0/0x1
 [<ffffffff810093aa>] ? hypercall_page+0x3aa/0x1000
 [<ffffffff8100ebbf>] ? xen_safe_halt+0x10/0x1a
 [<ffffffff8100c102>] ? xen_idle+0x3c/0x46
 [<ffffffff81010cbd>] ? cpu_idle+0x5d/0x8c
 [<ffffffff81426742>] ? rest_init+0x66/0x68
 [<ffffffff8179fd8d>] ? start_kernel+0x3ef/0x3fb
 [<ffffffff8179f2c3>] ? x86_64_start_reservations+0xae/0xb2
 [<ffffffff817a2cb3>] ? xen_start_kernel+0x4c0/0x4c7










=========================crash log for machine two in 2.6.32.10 ================================================






blktap_sysfs_create: adding attributes for dev ffff88010b668c00
__ratelimit: 4 callbacks suppressed
blktap_sysfs_create: adding attributes for dev ffff8800bd384200
__ratelimit: 6 callbacks suppressed
INIT: Id "s0" respawning too fast: disabled for 5 minutes
blktap_sysfs_destroy
blktap_sysfs_create: adding attributes for dev ffff8800bf07d800
INIT: Id "s0" respawning too fast: disabled for 5 minutes
divide error: 0000 [#1] SMP 
last sysfs file: /sys/hypervisor/type
CPU 2 
Modules linked in: 8021q garp xen_netback xen_blkback blktap blkback_pagemap nbd bridge stp llc autofs4 ipmi_devintf ipmi_si ipmi_msghandler lockd sunrpc bonding ipv6 xenfs dm_multipath video output sbs sbshc parport_pc lp parport ses enclosure snd_seq_dummy snd_seq_oss snd_seq_midi_event snd_seq snd_seq_device snd_pcm_oss snd_mixer_oss bnx2 snd_pcm serio_raw snd_timer snd soundcore snd_page_alloc i2c_i801 i2c_core pcspkr pata_acpi ata_generic iTCO_wdt iTCO_vendor_support ata_piix shpchp mptsas mptscsih mptbase [last unloaded: freq_table]
Pid: 19632, comm: xenstore-list Not tainted 2.6.32.10xen #1 Tecal RH2285          
RIP: e030:[<ffffffff8104ee57>]  [<ffffffff8104ee57>] find_busiest_group+0x37d/0x721
RSP: e02b:ffff8800b52a1c08  EFLAGS: 00010046
RAX: 0000000000000000 RBX: 0000000000000000 RCX: ffff88002807b501
RDX: 0000000000000000 RSI: 0000000000000040 RDI: 0000000000000040
RBP: ffff8800b52a1d68 R08: 0000000000000000 R09: ffff880028075e08
R10: ffff8800bf0cf6c0 R11: ffff8800b52a1dd8 R12: 0000000000000040
R13: ffff880028075df0 R14: ffff880028075ce0 R15: 0000000000000002
FS:  00007fcb228146e0(0000) GS:ffff880028066000(0000) knlGS:0000000000000000
CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000003581d180e0 CR3: 00000000b83b5000 CR4: 0000000000002660
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process xenstore-list (pid: 19632, threadinfo ffff8800b52a0000, task ffff88010c080000)
Stack:
 ffff8800b52a1c20 ffff880028075a48 ffff8800b52a1dbc 00000002b52a1c30
<0> ffff8800b52a1db0 0000000000000004 0000000000000000 0000000200000000
<0> ffff880028075e00 0000000000000001 ffff880028075df0 ffffffff2807b5c0
Call Trace:
 [<ffffffff8143a805>] schedule+0x27a/0x736
 [<ffffffff8143c3da>] ? _spin_unlock_irqrestore+0x15/0x17
 [<ffffffff81288528>] read_reply+0x86/0x104
 [<ffffffff81071e82>] ? autoremove_wake_function+0x0/0x3d
 [<ffffffff8143ae0e>] ? _cond_resched+0xe/0x22
 [<ffffffff81288639>] xenbus_dev_request_and_reply+0x58/0x89
 [<ffffffffa011554e>] xenbus_file_write+0x16a/0x469 [xenfs]
 [<ffffffff81109671>] vfs_write+0xb0/0x10a
 [<ffffffff8110a3ab>] sys_write+0x4c/0x72
 [<ffffffff81011d72>] system_call_fastpath+0x16/0x1b
Code: 83 7d 10 00 74 0c 48 8b 5d 10 c7 03 00 00 00 00 eb 70 41 8b 55 08 48 8b 45 a8 48 89 d3 48 c1 a5 d0 fe ff ff 0a 48 c1 e0 0a 31 d2 <48> f7 f3 48 89 45 a0 48 8b 85 08 ff ff ff 48 29 85 00 ff ff ff 
RIP  [<ffffffff8104ee57>] find_busiest_group+0x37d/0x721
 RSP <ffff8800b52a1c08>
---[ end trace 13509d88f5b8918c ]---
divide error: 0000 [#2] SMP 
last sysfs file: /sys/hypervisor/type
CPU 1 
Modules linked in: 8021q garp xen_netback xen_blkback blktap blkback_pagemap nbd bridge stp llc autofs4 ipmi_devintf ipmi_si ipmi_msghandler lockd sunrpc bonding ipv6 xenfs dm_multipath video output sbs sbshc parport_pc lp parport ses enclosure snd_seq_dummy snd_seq_oss snd_seq_midi_event snd_seq snd_seq_device snd_pcm_oss snd_mixer_oss bnx2 snd_pcm serio_raw snd_timer snd soundcore snd_page_alloc i2c_i801 i2c_core pcspkr pata_acpi ata_generic iTCO_wdt iTCO_vendor_support ata_piix shpchp mptsas mptscsih mptbase [last unloaded: freq_table]
Pid: 3429, comm: kipmi0 Tainted: G      D    2.6.32.10xen #1 Tecal RH2285          
RIP: e030:[<ffffffff8104ee57>]  [<ffffffff8104ee57>] find_busiest_group+0x37d/0x721
RSP: e02b:ffff880155e2dc70  EFLAGS: 00010046
RAX: 0000000000000000 RBX: 0000000000000000 RCX: ffff88002807b500
RDX: 0000000000000000 RSI: 0000000000000040 RDI: 0000000000000040
RBP: ffff880155e2ddd0 R08: 0000000000000000 R09: ffff880028075e08
R10: ffff880158339400 R11: ffff880028040000 R12: 0000000000000040
R13: ffff880028075df0 R14: ffff880028058ce0 R15: 0000000000000001
FS:  00007fde594896e0(0000) GS:ffff880028049000(0000) knlGS:0000000000000000
CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000469000 CR3: 0000000001001000 CR4: 0000000000002660
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process kipmi0 (pid: 3429, threadinfo ffff880155e2c000, task ffff880155cf2d60)
Stack:
 ffff880155e2dc88 ffff880028058a48 ffff880155e2de24 0000000255e2dc98
<0> ffff880155e2de18 0000000000000004 0000000000000000 ffffffff00000000
<0> ffff880028075e00 0000000028040000 ffff880028058df0 000000002803be08
Call Trace:
 [<ffffffff8143a805>] schedule+0x27a/0x736
 [<ffffffff8143c3da>] ? _spin_unlock_irqrestore+0x15/0x17
 [<ffffffff8143b0c0>] schedule_timeout+0x9d/0xc4
 [<ffffffff810608c4>] ? process_timeout+0x0/0x10
 [<ffffffff8143b125>] schedule_timeout_interruptible+0x1e/0x20
 [<ffffffffa01c9d09>] ipmi_thread+0x6a/0x7e [ipmi_si]
 [<ffffffffa01c9c9f>] ? ipmi_thread+0x0/0x7e [ipmi_si]
 [<ffffffff81071aa3>] kthread+0x6e/0x76
 [<ffffffff81012daa>] child_rip+0xa/0x20
 [<ffffffff81011f91>] ? int_ret_from_sys_call+0x7/0x1b
 [<ffffffff8101271d>] ? retint_restore_args+0x5/0x6
 [<ffffffff81012da0>] ? child_rip+0x0/0x20
Code: 83 7d 10 00 74 0c 48 8b 5d 10 c7 03 00 00 00 00 eb 70 41 8b 55 08 48 8b 45 a8 48 89 d3 48 c1 a5 d0 fe ff ff 0a 48 c1 e0 0a 31 d2 <48> f7 f3 48 89 45 a0 48 8b 85 08 ff ff ff 48 29 85 00 ff ff ff 
RIP  [<ffffffff8104ee57>] find_busiest_group+0x37d/0x721
 RSP <ffff880155e2dc70>
---[ end trace 13509d88f5b8918d ]---

[-- Attachment #3: kernel_bug_at_mmu.c.txt --]
[-- Type: text/plain, Size: 4390 bytes --]

Apr  8 12:19:47 r14a11017 kernel: ------------[ cut here ]------------
Apr  8 12:19:47 r14a11017 kernel: kernel BUG at arch/x86/xen/mmu.c:1872!
Apr  8 12:19:47 r14a11017 kernel: invalid opcode: 0000 [#1] SMP
Apr  8 12:19:47 r14a11017 kernel: last sysfs file: /sys/hypervisor/properties/capabilities
Apr  8 12:19:47 r14a11017 kernel: CPU 0
Apr  8 12:19:47 r14a11017 kernel: Modules linked in: 8021q garp blktap xen_netback xen_blkback blkback_pagemap nbd bridge stp llc autofs4 ipmi_devintf ipmi_si ipmi_msghandler lockd sunrpc bonding ipv6 xenfs dm_multipath video output sbs sbshc parport_pc lp parport ses enclosure snd_seq_dummy snd_seq_oss bnx2 snd_seq_midi_event serio_raw snd_seq snd_seq_device snd_pcm_oss snd_mixer_oss snd_pcm snd_timer i2c_i801 iTCO_wdt i2c_core snd soundcore snd_page_alloc iTCO_vendor_support pata_acpi ata_generic pcspkr ata_piix shpchp mptsas mptscsih mptbase [last unloaded: freq_table]
Apr  8 12:19:47 r14a11017 kernel: Pid: 15769, comm: sh Not tainted 2.6.32.36xen #1 Tecal RH2285
Apr  8 12:19:47 r14a11017 kernel: RIP: e030:[<ffffffff8100cebc>]  [<ffffffff8100cebc>] pin_pagetable_pfn+0x36/0x3c
Apr  8 12:19:47 r14a11017 kernel: RSP: e02b:ffff88001eb7baa8  EFLAGS: 00010282
Apr  8 12:19:47 r14a11017 kernel: RAX: 00000000ffffffea RBX: 000000000007b307 RCX: 0000000000000001
Apr  8 12:19:47 r14a11017 kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88001eb7baa8
Apr  8 12:19:47 r14a11017 kernel: RBP: ffff88001eb7bac8 R08: 0000000000000420 R09: ffff880000000000
Apr  8 12:19:47 r14a11017 kernel: R10: 0000000000007ff0 R11: ffff88008fc97248 R12: ffff88002840b000
Apr  8 12:19:47 r14a11017 kernel: R13: 000000000007b484 R14: 0000000000000003 R15: ffff88009b090000
Apr  8 12:19:47 r14a11017 kernel: FS:  00007fe8bbc656e0(0000) GS:ffff88002803b000(0000) knlGS:0000000000000000
Apr  8 12:19:47 r14a11017 kernel: CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
Apr  8 12:19:47 r14a11017 kernel: CR2: 00000000006bb338 CR3: 000000007b307000 CR4: 0000000000002660
Apr  8 12:19:47 r14a11017 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Apr  8 12:19:47 r14a11017 kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Apr  8 12:19:47 r14a11017 kernel: Process sh (pid: 15769, threadinfo ffff88001eb7a000, task ffff88009b090000)
Apr  8 12:19:47 r14a11017 kernel: Stack:
Apr  8 12:19:47 r14a11017 kernel:  0000000000000000 00000000004b7484 000000011eb7bac8 000000000007b307
Apr  8 12:19:47 r14a11017 kernel: <0> ffff88001eb7baf8 ffffffff8100e8ef ffff88012e4fb100 ffff88000fb5e018
Apr  8 12:19:47 r14a11017 kernel: <0> 000000000007b484 00000000006bb338 ffff88001eb7bb08 ffffffff8100e935
Apr  8 12:19:47 r14a11017 kernel: Call Trace:
Apr  8 12:19:47 r14a11017 kernel:  [<ffffffff8100e8ef>] xen_alloc_ptpage+0x8d/0x96
Apr  8 12:19:47 r14a11017 kernel:  [<ffffffff8100e935>] xen_alloc_pte+0x13/0x15
Apr  8 12:19:47 r14a11017 kernel:  [<ffffffff810eb702>] __pte_alloc+0x7f/0xdc
Apr  8 12:19:47 r14a11017 kernel:  [<ffffffff810e90bd>] ? pmd_offset+0x13/0x3c
Apr  8 12:19:47 r14a11017 kernel:  [<ffffffff810eb818>] handle_mm_fault+0xb9/0x771
Apr  8 12:19:47 r14a11017 kernel:  [<ffffffff810f08fd>] ? vma_link+0x7c/0xa4
Apr  8 12:19:47 r14a11017 kernel:  [<ffffffff810f13b0>] ? mmap_region+0x322/0x42b
Apr  8 12:19:47 r14a11017 kernel:  [<ffffffff8100f169>] ? xen_force_evtchn_callback+0xd/0xf
Apr  8 12:19:47 r14a11017 kernel:  [<ffffffff81449701>] do_page_fault+0x21c/0x288
Apr  8 12:19:47 r14a11017 kernel:  [<ffffffff81447695>] page_fault+0x25/0x30
Apr  8 12:19:47 r14a11017 kernel:  [<ffffffff81222a39>] ? __clear_user+0x33/0x55
Apr  8 12:19:47 r14a11017 kernel:  [<ffffffff81222a1d>] ? __clear_user+0x17/0x55
Apr  8 12:19:47 r14a11017 kernel:  [<ffffffff81222a8b>] clear_user+0x30/0x38
Apr  8 12:19:47 r14a11017 kernel:  [<ffffffff8115139a>] load_elf_binary+0x5d5/0x17ef
Apr  8 12:19:47 r14a11017 kernel:  [<ffffffff811f4648>] ? process_measurement+0xc0/0xd7
Apr  8 12:19:47 r14a11017 kernel:  [<ffffffff81150dc5>] ? load_elf_binary+0x0/0x17ef
Apr  8 12:19:47 r14a11017 kernel:  [<ffffffff81113094>] search_binary_handler+0xc8/0x255
Apr  8 12:19:47 r14a11017 kernel:  [<ffffffff81114362>] do_execve+0x1c3/0x29e
Apr  8 12:19:47 r14a11017 kernel:  [<ffffffff8101155d>] sys_execve+0x43/0x5d
Apr  8 12:19:47 r14a11017 kernel:  [<ffffffff810131ca>] stub_execve+0x6a/0xc0

[-- Attachment #4: granttabl.txt --]
[-- Type: text/plain, Size: 10965 bytes --]

(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) printk: 1 messages suppressed.
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) printk: 5 messages suppressed.
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) printk: 5 messages suppressed.
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) printk: 3 messages suppressed.
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) printk: 1 messages suppressed.
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) printk: 1 messages suppressed.
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) printk: 1 messages suppressed.
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) printk: 1 messages suppressed.
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) printk: 3 messages suppressed.
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) printk: 1 messages suppressed.
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) printk: 5 messages suppressed.
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) printk: 5 messages suppressed.
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) printk: 3 messages suppressed.
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) printk: 1 messages suppressed.
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) printk: 1 messages suppressed.
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) printk: 6 messages suppressed.
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) printk: 1 messages suppressed.
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) printk: 3 messages suppressed.
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) printk: 9 messages suppressed.
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) printk: 9 messages suppressed.
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) printk: 9 messages suppressed.
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) printk: 7 messages suppressed.
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) printk: 5 messages suppressed.
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) printk: 3 messages suppressed.
(XEN) grant_table.c:1717:d0 Bad grant reference 4294965983
(XEN) printk: 1 messages suppressed.
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) printk: 1 messages suppressed.
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) printk: 3 messages suppressed.
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) printk: 3 messages suppressed.
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) printk: 1 messages suppressed.
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) printk: 3 messages suppressed.
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) printk: 2 messages suppressed.
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) grant_table.c:1717:d0 Bad grant reference 4294965983
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) printk: 4 messages suppressed.
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) printk: 2 messages suppressed.
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) printk: 3 messages suppressed.
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) printk: 1 messages suppressed.
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) printk: 1 messages suppressed.
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) printk: 1 messages suppressed.
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) printk: 1 messages suppressed.
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) printk: 1 messages suppressed.
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) printk: 5 messages suppressed.
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) printk: 3 messages suppressed.
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) printk: 1 messages suppressed.
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) printk: 3 messages suppressed.
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) grant_table.c:1717:d0 Bad grant reference 4294965983
(XEN) grant_table.c:1717:d0 Bad grant reference 4294965983
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) printk: 7 messages suppressed.
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) printk: 15 messages suppressed.
(XEN) grant_table.c:1717:d0 Bad grant reference 4294901760
(XEN) printk: 13 messages suppressed.
(XEN) grant_table.c:1717:d0 Bad grant reference 4294901765
(XEN) printk: 9 messages suppressed.
(XEN) grant_table.c:1717:d0 Bad grant reference 4294901765
(XEN) printk: 9 messages suppressed.
(XEN) grant_table.c:1717:d0 Bad grant reference 4294901765
(XEN) printk: 9 messages suppressed.
(XEN) grant_table.c:1717:d0 Bad grant reference 4294901765
(XEN) printk: 13 messages suppressed.
(XEN) grant_table.c:1717:d0 Bad grant reference 4294901760
(XEN) printk: 9 messages suppressed.
(XEN) grant_table.c:1717:d0 Bad grant reference 4294901760
(XEN) grant_table.c:1717:d0 Bad grant reference 4294901760
(XEN) grant_table.c:1717:d0 Bad grant reference 4294901765
(XEN) printk: 5 messages suppressed.
(XEN) grant_table.c:1717:d0 Bad grant reference 4294901765
(XEN) printk: 6 messages suppressed.
(XEN) grant_table.c:1717:d0 Bad grant reference 4294901760
(XEN) printk: 8 messages suppressed.
(XEN) grant_table.c:1717:d0 Bad grant reference 4294901765

[-- Attachment #5: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

RE: kernel BUG at arch/x86/xen/mmu.c:1872

[MaoXiaoyun]

[-- Attachment #1.1: Type: text/plain, Size: 3342 bytes --]


Hi Konrad & Jeremy:

            I think we finally located the missing patch for this commit.
            We test commit http://git.kernel.org/?p=linux/kernel/git/jeremy/xen.git;a=commit;h=c97f681f138039425c87f35ea46a92385d81e70e
            which is works.
 
            We test commit http://git.kernel.org/?p=linux/kernel/git/jeremy/xen.git;a=commit;h=221c64dbf860d37f841f40893bddf8d804aa55bd
            which server crashed.
 
             Later I found the comments for this commit: 
             http://git.kernel.org/?p=linux/kernel/git/jeremy/xen.git;a=commit;h=64141da587241301ce8638cc945f8b67853156ec
         
            So It looks like this fix is not applied on 2.6.32.36, Could you take a look at this? 
 
            Many thanks.
              
=====================================================
>Hi Konrad & Jeremy:
> 
>     I'd like to open this BUG in a new thread, since the old thread is too long for easy read.
>     
>     We recently want to upgrade our kernel to 2.6.32, but unfortunately, we confront a kernel crash bug.
>Our test case is simple, start 24 win2003 HVMS on our physical machine, and each HVM reboot 
>every 15minutes. The kernel will crash in half an hour.(That is crash on VM second starts).
> 
>Our test go much further.
>We test different kernel version.
>2.6.32.10  http://git.kernel.org/?p=linux/kernel/git/jeremy/xen.git;a=commit;h=d945b014ac5df9592c478bf9486d97e8914aab59
>2.6.32.11  http://git.kernel.org/?p=linux/kernel/git/jeremy/xen.git;a=commit;h=27f948a3bf365a5bc3d56119637a177d41147815
>2.6.32.12  http://git.kernel.org/?p=linux/kernel/git/jeremy/xen.git;a=commit;h=ba739f9abd3f659b907a824af1161926b420a2ce
>2.6.32.13  http://git.kernel.org/?p=linux/kernel/git/jeremy/xen.git;a=commit;h=f6fe6583b77a49b569eef1b66c3d761eec2e561b
>2.6.32.15  http://git.kernel.org/?p=linux/kernel/git/jeremy/xen.git;a=commit;h=27ed1b0e0dae5f1d5da5c76451bc84cb529128bd
>2.6.32.21  http://git.kernel.org/?p=linux/kernel/git/jeremy/xen.git;a=commit;h=69e50db231723596ed8ef9275d0068d6697f466a
> 
>There are basic three different result we met.
> 
>i1) grant table issue
>The host still function, but use xm  dmesg, we have abnormal log.
>please refer to the attched log of grant table
> 
>i2) kernel crash on a different place.
>Host die during the test, after reboot, we can see nothing abnormal in /var/log/messages
> 
>i3) kernel BUG at arch/x86/xen/mmu.c:1872; 
>Host die during the test, after reboot, we see the crash log in messages, refer to the attached log of 2.6.32.36
>Summary of the test result, can be classified in two:
> 
>1) 2.6.32.10
>30 machines involved the test, and three has issue (i1), and two has issue (i2), *no* issue (i3)
>Other machines run tests successfully till now, more than 8 hours
> 
>2)2.6.32.11 or later version.
>Each version containers 10 machine for tests, and all machine crashed in less than half an hour.
> 
>Conclusion:
>1) grant table issue exists in all kernel version
>2) kernerl crash at different place may exist in all kernel versions, but not happen so frequently, 2 out of 30
>3) We observe the major difference of issue i3), from the test, it looks like it is introduced between the version
>2.6.32.10 and 2.6.32.11.
> 
>Hope this help to locate the bug.
>Many thanks.
> 
> 		 	   		  

[-- Attachment #1.2: Type: text/html, Size: 5901 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: kernel BUG at arch/x86/xen/mmu.c:1872

[Teck Choon Giam]

[-- Attachment #1: Type: text/plain, Size: 4583 bytes --]

2011/4/10 MaoXiaoyun <tinnycloud@hotmail.com>:
> Hi Konrad & Jeremy:
>
>             I think we finally located the missing patch for this commit.
>             We test commit
> http://git.kernel.org/?p=linux/kernel/git/jeremy/xen.git;a=commit;h=c97f681f138039425c87f35ea46a92385d81e70e
>             which is works.
>
>             We test commit
> http://git.kernel.org/?p=linux/kernel/git/jeremy/xen.git;a=commit;h=221c64dbf860d37f841f40893bddf8d804aa55bd
>             which server crashed.
>
>              Later I found the comments for this commit:
>
> http://git.kernel.org/?p=linux/kernel/git/jeremy/xen.git;a=commit;h=64141da587241301ce8638cc945f8b67853156ec
>
>             So It looks like this fix is not applied on 2.6.32.36, Could you
> take a look at this?
>
>             Many thanks.
>
> =====================================================
>>Hi Konrad & Jeremy:
>>
>>     I'd like to open this BUG in a new thread, since the old thread is too
>> long for easy read.
>>
>>     We recently want to upgrade our kernel to 2.6.32, but unfortunately,
>> we confront a kernel crash bug.
>>Our test case is simple, start 24 win2003 HVMS on our physical machine, and
>> each HVM reboot
>>every 15minutes. The kernel will crash in half an hour.(That is crash on VM
>> second starts).
>>
>>Our test go much further.
>>We test different kernel version.
>>2.6.32.10
>> http://git.kernel.org/?p=linux/kernel/git/jeremy/xen.git;a=commit;h=d945b014ac5df9592c478bf9486d97e8914aab59
>>2.6.32.11
>> http://git.kernel.org/?p=linux/kernel/git/jeremy/xen.git;a=commit;h=27f948a3bf365a5bc3d56119637a177d41147815
>>2.6.32.12
>> http://git.kernel.org/?p=linux/kernel/git/jeremy/xen.git;a=commit;h=ba739f9abd3f659b907a824af1161926b420a2ce
>>2.6.32.13
>> http://git.kernel.org/?p=linux/kernel/git/jeremy/xen.git;a=commit;h=f6fe6583b77a49b569eef1b66c3d761eec2e561b
>>2.6.32.15
>> http://git.kernel.org/?p=linux/kernel/git/jeremy/xen.git;a=commit;h=27ed1b0e0dae5f1d5da5c76451bc84cb529128bd
>>2.6.32.21
>> http://git.kernel.org/?p=linux/kernel/git/jeremy/xen.git;a=commit;h=69e50db231723596ed8ef9275d0068d6697f466a
>>
>>There are basic three different result we met.
>>
>>i1) grant table issue
>>The host still function, but use xm  dmesg, we have abnormal log.
>>please refer to the attched log of grant table
>>
>>i2) kernel crash on a different place.
>>Host die during the test, after reboot, we can see nothing abnormal in
>> /var/log/messages
>>
>>i3) kernel BUG at arch/x86/xen/mmu.c:1872;
>>Host die during the test, after reboot, we see the crash log in messages,
>> refer to the attached log of 2.6.32.36
>>Summary of the test result, can be classified in two:
>>
>>1) 2.6.32.10
>>30 machines involved the test, and three has issue (i1), and two has issue
>> (i2), *no* issue (i3)
>>Other machines run tests successfully till now, more than 8 hours
>>
>>2)2.6.32.11 or later version.
>>Each version containers 10 machine for tests, and all machine crashed in
>> less than half an hour.
>>
>>Conclusion:
>>1) grant table issue exists in all kernel version
>>2) kernerl crash at different place may exist in all kernel versions, but
>> not happen so frequently, 2 out of 30
>>3) We observe the major difference of issue i3), from the test, it looks
>> like it is introduced between the version
>>2.6.32.10 and 2.6.32.11.
>>
>>Hope this help to locate the bug.
>>Many thanks.
>>
>>
>

Hi,

Sorry, since this mmu related BUG has been troubled me for very
long... I really want to "kill" this BUG but my knowledge in kernel
hacking and/or xen is very limited.

While waiting for Jeremy or Konrad or others ...

Many thanks for spending time to track down this mmu related BUG.  I
have backported the commit from
http://git.kernel.org/?p=linux/kernel/git/jeremy/xen.git;a=commit;h=64141da587241301ce8638cc945f8b67853156ec
to 2.6.32.36 PVOPS kernel and patch attached.  I won't know whether
did I backport it correctly nor does it affects anything.  I am
currently testing the 2.6.32.36 PVOPS kernel with this patch applied
and also unset CONFIG_DEBUG_PAGEALLOC.  Currently running testcrash.sh
loop 1000 as I am unable to reproduce this mmu BUG 1872 in
testcrash.sh loop 100.  Please note that when CONFIG_DEBUG_PAGEALLOC
is unset, I can reproduce this mmu BUG 1872 easily within <50
testcrash.sh loop cycle with PVOPS version 2.6.32.24 to 2.6.32.36
kernel.  Now test with this backport patch to see whether I can
reproduce this mmu BUG... ...

Kindest regards,
Giam Teck Choon

[-- Attachment #2: vmalloc__eagerly_clear_ptes_on_vunmap.patch --]
[-- Type: text/x-patch, Size: 3393 bytes --]

Back port from commit http://git.kernel.org/?p=linux/kernel/git/jeremy/xen.git;a=commit;h=64141da587241301ce8638cc945f8b67853156ec

diff -urN a/arch/x86/xen/mmu.c b/arch/x86/xen/mmu.c
--- a/arch/x86/xen/mmu.c	2011-03-30 06:17:46.000000000 +0800
+++ b/arch/x86/xen/mmu.c	2011-04-11 02:17:54.000000000 +0800
@@ -2430,8 +2430,6 @@
 	x86_init.paging.pagetable_setup_start = xen_pagetable_setup_start;
 	x86_init.paging.pagetable_setup_done = xen_pagetable_setup_done;
 	pv_mmu_ops = xen_mmu_ops;
-
-	vmap_lazy_unmap = false;
 }
 
 /* Protected by xen_reservation_lock. */
diff -urN a/include/linux/vmalloc.h b/include/linux/vmalloc.h
--- a/include/linux/vmalloc.h	2011-03-30 06:17:46.000000000 +0800
+++ b/include/linux/vmalloc.h	2011-04-11 02:18:43.000000000 +0800
@@ -7,8 +7,6 @@
 
 struct vm_area_struct;		/* vma defining user mapping in mm_types.h */
 
-extern bool vmap_lazy_unmap;
-
 /* bits in flags of vmalloc's vm_struct below */
 #define VM_IOREMAP	0x00000001	/* ioremap() and friends */
 #define VM_ALLOC	0x00000002	/* vmalloc() */
diff -urN a/mm/vmalloc.c b/mm/vmalloc.c
--- a/mm/vmalloc.c	2011-03-30 06:17:46.000000000 +0800
+++ b/mm/vmalloc.c	2011-04-11 02:25:38.000000000 +0800
@@ -31,8 +31,6 @@
 #include <asm/tlbflush.h>
 #include <asm/shmparam.h>
 
-bool vmap_lazy_unmap __read_mostly = true;
-
 /*** Page table manipulation functions ***/
 
 static void vunmap_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end)
@@ -503,9 +501,6 @@
 {
 	unsigned int log;
 
-	if (!vmap_lazy_unmap)
-		return 0;
-
 	log = fls(num_online_cpus());
 
 	return log * (32UL * 1024 * 1024 / PAGE_SIZE);
@@ -566,7 +561,6 @@
 			if (va->va_end > *end)
 				*end = va->va_end;
 			nr += (va->va_end - va->va_start) >> PAGE_SHIFT;
-			unmap_vmap_area(va);
 			list_add_tail(&va->purge_list, &valist);
 			va->flags |= VM_LAZY_FREEING;
 			va->flags &= ~VM_LAZY_FREE;
@@ -612,10 +606,11 @@
 }
 
 /*
- * Free and unmap a vmap area, caller ensuring flush_cache_vunmap had been
- * called for the correct range previously.
+ * Free a vmap area, caller ensuring that the area has been unmapped
+ * and flush_cache_vunmap had been called for the correct range
+ * previously.
  */
-static void free_unmap_vmap_area_noflush(struct vmap_area *va)
+static void free_vmap_area_noflush(struct vmap_area *va)
 {
 	va->flags |= VM_LAZY_FREE;
 	atomic_add((va->va_end - va->va_start) >> PAGE_SHIFT, &vmap_lazy_nr);
@@ -624,6 +619,16 @@
 }
 
 /*
+ * Free and unmap a vmap area, caller ensuring flush_cache_vunmap had been
+ * called for the correct range previously.
+ */
+static void free_unmap_vmap_area_noflush(struct vmap_area *va)
+{
+	unmap_vmap_area(va);
+	free_vmap_area_noflush(va);
+}
+
+/*
  * Free and unmap a vmap area
  */
 static void free_unmap_vmap_area(struct vmap_area *va)
@@ -799,7 +804,7 @@
 	spin_unlock(&vmap_block_tree_lock);
 	BUG_ON(tmp != vb);
 
-	free_unmap_vmap_area_noflush(vb->va);
+	free_vmap_area_noflush(vb->va);
 	call_rcu(&vb->rcu_head, rcu_free_vb);
 }
 
@@ -936,6 +941,8 @@
 	rcu_read_unlock();
 	BUG_ON(!vb);
 
+	vunmap_page_range((unsigned long)addr, (unsigned long)addr + size);
+
 	spin_lock(&vb->lock);
 	BUG_ON(bitmap_allocate_region(vb->dirty_map, offset >> PAGE_SHIFT, order));
 
@@ -988,7 +995,6 @@
 
 				s = vb->va->va_start + (i << PAGE_SHIFT);
 				e = vb->va->va_start + (j << PAGE_SHIFT);
-				vunmap_page_range(s, e);
 				flush = 1;
 
 				if (s < start)

[-- Attachment #3: testcrash.sh --]
[-- Type: application/x-sh, Size: 5573 bytes --]

[-- Attachment #4: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: kernel BUG at arch/x86/xen/mmu.c:1872

[Teck Choon Giam]

>
> Hi,
>
> Sorry, since this mmu related BUG has been troubled me for very
> long... I really want to "kill" this BUG but my knowledge in kernel
> hacking and/or xen is very limited.
>
> While waiting for Jeremy or Konrad or others ...
>
> Many thanks for spending time to track down this mmu related BUG.  I
> have backported the commit from
> http://git.kernel.org/?p=linux/kernel/git/jeremy/xen.git;a=commit;h=64141da587241301ce8638cc945f8b67853156ec
> to 2.6.32.36 PVOPS kernel and patch attached.  I won't know whether
> did I backport it correctly nor does it affects anything.  I am
> currently testing the 2.6.32.36 PVOPS kernel with this patch applied
> and also unset CONFIG_DEBUG_PAGEALLOC.  Currently running testcrash.sh
> loop 1000 as I am unable to reproduce this mmu BUG 1872 in
> testcrash.sh loop 100.  Please note that when CONFIG_DEBUG_PAGEALLOC
> is unset, I can reproduce this mmu BUG 1872 easily within <50
> testcrash.sh loop cycle with PVOPS version 2.6.32.24 to 2.6.32.36
> kernel.  Now test with this backport patch to see whether I can
> reproduce this mmu BUG... ...
>
> Kindest regards,
> Giam Teck Choon
>

I have tested with my backport patch and it is working fine as I am
unable to reproduce the mmu.c 1872 or 1860 bug with
CONFIG_DEBUG_PAGEALLOC not set.  I tested with testcrash.sh loop 100
and 1000.  Now doing testcrash.sh loop 10000.

Xiaoyun, is it possible for you to test my patch and see whether can
you reproduce the mmu.c 1872/1860 bug?

Can anyone of you review my patch?

I will post a format patch according to
Documentation/SubmittingPatches in my next reply and hopefully can be
reviewed.

Thanks.

Kindest regards,
Giam Teck Choon

Re: kernel BUG at arch/x86/xen/mmu.c:1872

[Teck Choon Giam]

From: Giam Teck Choon <giamteckchoon@gmail.com>

vmalloc: eagerly clear ptes on vunmap

Backport from commit 64141da587241301ce8638cc945f8b67853156ec to 2.6.32.36

URL: http://git.kernel.org/?p=linux/kernel/git/jeremy/xen.git;a=commit;h=64141da587241301ce8638cc945f8b67853156ec

Without this patch, kernel BUG at arch/x86/xen/mmu.c:1860 or kernel BUG at
arch/x86/xen/mmu.c:1872 is easily triggered when CONFIG_DEBUG_PAGEALLOC is
unset especially doing LVM snapshots.

Signed-off-by: Giam Teck Choon <giamteckchoon@gmail.com>
---
 arch/x86/xen/mmu.c      |    2 --
 include/linux/vmalloc.h |    2 --
 mm/vmalloc.c            |   28 +++++++++++++++++-----------
 3 files changed, 17 insertions(+), 15 deletions(-)

diff --git a/arch/x86/xen/mmu.c b/arch/x86/xen/mmu.c
index fa36ab8..204e3ba 100644
--- a/arch/x86/xen/mmu.c
+++ b/arch/x86/xen/mmu.c
@@ -2430,8 +2430,6 @@ void __init xen_init_mmu_ops(void)
 	x86_init.paging.pagetable_setup_start = xen_pagetable_setup_start;
 	x86_init.paging.pagetable_setup_done = xen_pagetable_setup_done;
 	pv_mmu_ops = xen_mmu_ops;
-
-	vmap_lazy_unmap = false;
 }

 /* Protected by xen_reservation_lock. */
diff --git a/include/linux/vmalloc.h b/include/linux/vmalloc.h
index 1a2ba21..3c123c3 100644
--- a/include/linux/vmalloc.h
+++ b/include/linux/vmalloc.h
@@ -7,8 +7,6 @@

 struct vm_area_struct;		/* vma defining user mapping in mm_types.h */

-extern bool vmap_lazy_unmap;
-
 /* bits in flags of vmalloc's vm_struct below */
 #define VM_IOREMAP	0x00000001	/* ioremap() and friends */
 #define VM_ALLOC	0x00000002	/* vmalloc() */
diff --git a/mm/vmalloc.c b/mm/vmalloc.c
index 4f701c2..80cbd7b 100644
--- a/mm/vmalloc.c
+++ b/mm/vmalloc.c
@@ -31,8 +31,6 @@
 #include <asm/tlbflush.h>
 #include <asm/shmparam.h>

-bool vmap_lazy_unmap __read_mostly = true;
-
 /*** Page table manipulation functions ***/

 static void vunmap_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end)
@@ -503,9 +501,6 @@ static unsigned long lazy_max_pages(void)
 {
 	unsigned int log;

-	if (!vmap_lazy_unmap)
-		return 0;
-
 	log = fls(num_online_cpus());

 	return log * (32UL * 1024 * 1024 / PAGE_SIZE);
@@ -566,7 +561,6 @@ static void __purge_vmap_area_lazy(unsigned long
*start, unsigned long *end,
 			if (va->va_end > *end)
 				*end = va->va_end;
 			nr += (va->va_end - va->va_start) >> PAGE_SHIFT;
-			unmap_vmap_area(va);
 			list_add_tail(&va->purge_list, &valist);
 			va->flags |= VM_LAZY_FREEING;
 			va->flags &= ~VM_LAZY_FREE;
@@ -612,10 +606,11 @@ static void purge_vmap_area_lazy(void)
 }

 /*
- * Free and unmap a vmap area, caller ensuring flush_cache_vunmap had been
- * called for the correct range previously.
+ * Free a vmap area, caller ensuring that the area has been unmapped
+ * and flush_cache_vunmap had been called for the correct range
+ * previously.
  */
-static void free_unmap_vmap_area_noflush(struct vmap_area *va)
+static void free_vmap_area_noflush(struct vmap_area *va)
 {
 	va->flags |= VM_LAZY_FREE;
 	atomic_add((va->va_end - va->va_start) >> PAGE_SHIFT, &vmap_lazy_nr);
@@ -624,6 +619,16 @@ static void free_unmap_vmap_area_noflush(struct
vmap_area *va)
 }

 /*
+ * Free and unmap a vmap area, caller ensuring flush_cache_vunmap had been
+ * called for the correct range previously.
+ */
+static void free_unmap_vmap_area_noflush(struct vmap_area *va)
+{
+	unmap_vmap_area(va);
+	free_vmap_area_noflush(va);
+}
+
+/*
  * Free and unmap a vmap area
  */
 static void free_unmap_vmap_area(struct vmap_area *va)
@@ -799,7 +804,7 @@ static void free_vmap_block(struct vmap_block *vb)
 	spin_unlock(&vmap_block_tree_lock);
 	BUG_ON(tmp != vb);

-	free_unmap_vmap_area_noflush(vb->va);
+	free_vmap_area_noflush(vb->va);
 	call_rcu(&vb->rcu_head, rcu_free_vb);
 }

@@ -936,6 +941,8 @@ static void vb_free(const void *addr, unsigned long size)
 	rcu_read_unlock();
 	BUG_ON(!vb);

+	vunmap_page_range((unsigned long)addr, (unsigned long)addr + size);
+
 	spin_lock(&vb->lock);
 	BUG_ON(bitmap_allocate_region(vb->dirty_map, offset >> PAGE_SHIFT, order));

@@ -988,7 +995,6 @@ void vm_unmap_aliases(void)

 				s = vb->va->va_start + (i << PAGE_SHIFT);
 				e = vb->va->va_start + (j << PAGE_SHIFT);
-				vunmap_page_range(s, e);
 				flush = 1;

 				if (s < start)

RE: kernel BUG at arch/x86/xen/mmu.c:1872

[MaoXiaoyun]

[-- Attachment #1.1: Type: text/plain, Size: 2759 bytes --]


Hi:
 
     I believe this is the fix at much extent. 
     Since I have my own test cases which with this patch, my test case will success in 30 rounds run. 
     Every round takes 8hours.  While without this patch, tests fail evey round in 15minutes.
 
      So this really means fix most of the things. 
 
      But during running, I met another crash, from the log it it looks like has relation with
this BUG, since the crash log shows it is tlb related and this BUG also tlb related.
 
      Well, I'm also have poor knowledge of kernel.
      Hope someone from Xen Devel offer some help. 
 
      Many thanks.
 
> Date: Mon, 11 Apr 2011 20:16:53 +0800
> Subject: Re: kernel BUG at arch/x86/xen/mmu.c:1872
> From: giamteckchoon@gmail.com
> To: tinnycloud@hotmail.com
> CC: xen-devel@lists.xensource.com; dave@ivt.com.au; ian.campbell@citrix.com; konrad.wilk@oracle.com; jeremy@goop.org; keir@xen.org
> 
> >
> > Hi,
> >
> > Sorry, since this mmu related BUG has been troubled me for very
> > long... I really want to "kill" this BUG but my knowledge in kernel
> > hacking and/or xen is very limited.
> >
> > While waiting for Jeremy or Konrad or others ...
> >
> > Many thanks for spending time to track down this mmu related BUG.  I
> > have backported the commit from
> > http://git.kernel.org/?p=linux/kernel/git/jeremy/xen.git;a=commit;h=64141da587241301ce8638cc945f8b67853156ec
> > to 2.6.32.36 PVOPS kernel and patch attached.  I won't know whether
> > did I backport it correctly nor does it affects anything.  I am
> > currently testing the 2.6.32.36 PVOPS kernel with this patch applied
> > and also unset CONFIG_DEBUG_PAGEALLOC.  Currently running testcrash.sh
> > loop 1000 as I am unable to reproduce this mmu BUG 1872 in
> > testcrash.sh loop 100.  Please note that when CONFIG_DEBUG_PAGEALLOC
> > is unset, I can reproduce this mmu BUG 1872 easily within <50
> > testcrash.sh loop cycle with PVOPS version 2.6.32.24 to 2.6.32.36
> > kernel.  Now test with this backport patch to see whether I can
> > reproduce this mmu BUG... ...
> >
> > Kindest regards,
> > Giam Teck Choon
> >
> 
> I have tested with my backport patch and it is working fine as I am
> unable to reproduce the mmu.c 1872 or 1860 bug with
> CONFIG_DEBUG_PAGEALLOC not set. I tested with testcrash.sh loop 100
> and 1000. Now doing testcrash.sh loop 10000.
> 
> Xiaoyun, is it possible for you to test my patch and see whether can
> you reproduce the mmu.c 1872/1860 bug?
> 
> Can anyone of you review my patch?
> 
> I will post a format patch according to
> Documentation/SubmittingPatches in my next reply and hopefully can be
> reviewed.
> 
> Thanks.
> 
> Kindest regards,
> Giam Teck Choon
 		 	   		  

[-- Attachment #1.2: Type: text/html, Size: 3639 bytes --]

[-- Attachment #2: 196.22.txt --]
[-- Type: text/plain, Size: 6940 bytes --]

_ratelimit: 62 callbacks suppressed
blktap_sysfs_create: adding attributes for dev ffff88009bdb6000
blktap_sysfs_create: adding attributes for dev ffff88009bdb2200
INIT: Id "s0" respawning too fast: disabled for 5 minutes
__ratelimit: 14 callbacks suppressed
blktap_sysfs_destroy
blktap_sysfs_destroy
------------[ cut here ]------------
kernel BUG at arch/x86/mm/tlb.c:61!
invalid opcode: 0000 [#1] SMP 
last sysfs file: /sys/devices/system/xen_memory/xen_memory0/info/current_kb
CPU 1 
Modules linked in: 8021q garp xen_netback xen_blkback blktap blkback_pagemap nbd bridge stp llc autofs4 ipmi_devintf ipmi_si ipmi_msghandler lockd sunrpc bonding ipv6 xenfs dm_multipath video output sbs sbshc parport_pc lp parport ses enclosure snd_seq_dummy snd_seq_oss snd_seq_midi_event snd_seq snd_seq_device serio_raw bnx2 snd_pcm_oss snd_mixer_oss snd_pcm snd_timer iTCO_wdt snd soundcore snd_page_alloc i2c_i801 iTCO_vendor_support i2c_core pcspkr pata_acpi ata_generic ata_piix shpchp mptsas mptscsih mptbase [last unloaded: freq_table]
Pid: 25581, comm: khelper Not tainted 2.6.32.36fixxen #1 Tecal RH2285          
RIP: e030:[<ffffffff8103a3cb>]  [<ffffffff8103a3cb>] leave_mm+0x15/0x46
RSP: e02b:ffff88002805be48  EFLAGS: 00010046
RAX: 0000000000000000 RBX: 0000000000000001 RCX: ffff88015f8e2da0
RDX: ffff88002805be78 RSI: 0000000000000000 RDI: 0000000000000001
RBP: ffff88002805be48 R08: ffff88009d662000 R09: dead000000200200
R10: dead000000100100 R11: ffffffff814472b2 R12: ffff88009bfc1880
R13: ffff880028063020 R14: 00000000000004f6 R15: 0000000000000000
FS:  00007f62362d66e0(0000) GS:ffff880028058000(0000) knlGS:0000000000000000
CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000003aabc11909 CR3: 000000009b8ca000 CR4: 0000000000002660
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process khelper (pid: 25581, threadinfo ffff88007691e000, task ffff88009b92db40)
Stack:
 ffff88002805be68 ffffffff8100e4ae 0000000000000001 ffff88009d733b88
<0> ffff88002805be98 ffffffff81087224 ffff88002805be78 ffff88002805be78
<0> ffff88015f808360 00000000000004f6 ffff88002805bea8 ffffffff81010108
Call Trace:
 <IRQ> 
 [<ffffffff8100e4ae>] drop_other_mm_ref+0x2a/0x53
 [<ffffffff81087224>] generic_smp_call_function_single_interrupt+0xd8/0xfc
 [<ffffffff81010108>] xen_call_function_single_interrupt+0x13/0x28
 [<ffffffff810a936a>] handle_IRQ_event+0x66/0x120
 [<ffffffff810aac5b>] handle_percpu_irq+0x41/0x6e
 [<ffffffff8128c1c0>] __xen_evtchn_do_upcall+0x1ab/0x27d
 [<ffffffff8128dd11>] xen_evtchn_do_upcall+0x33/0x46
 [<ffffffff81013efe>] xen_do_hypervisor_callback+0x1e/0x30
 <EOI> 
 [<ffffffff814472b2>] ? _spin_unlock_irqrestore+0x15/0x17
 [<ffffffff8100f8cf>] ? xen_restore_fl_direct_end+0x0/0x1
 [<ffffffff81113f71>] ? flush_old_exec+0x3ac/0x500
 [<ffffffff81150dc5>] ? load_elf_binary+0x0/0x17ef
 [<ffffffff81150dc5>] ? load_elf_binary+0x0/0x17ef
 [<ffffffff8115115d>] ? load_elf_binary+0x398/0x17ef
 [<ffffffff81042fcf>] ? need_resched+0x23/0x2d
 [<ffffffff811f4648>] ? process_measurement+0xc0/0xd7
 [<ffffffff81150dc5>] ? load_elf_binary+0x0/0x17ef
 [<ffffffff81113094>] ? search_binary_handler+0xc8/0x255
 [<ffffffff81114362>] ? do_execve+0x1c3/0x29e
 [<ffffffff8101155d>] ? sys_execve+0x43/0x5d
 [<ffffffff8106fc45>] ? __call_usermodehelper+0x0/0x6f
 [<ffffffff81013e28>] ? kernel_execve+0x68/0xd0
 [<ffffffff8106fc45>] ? __call_usermodehelper+0x0/0x6f
 [<ffffffff8100f8cf>] ? xen_restore_fl_direct_end+0x0/0x1
 [<ffffffff8106fb64>] ? ____call_usermodehelper+0x113/0x11e
 [<ffffffff81013daa>] ? child_rip+0xa/0x20
 [<ffffffff8106fc45>] ? __call_usermodehelper+0x0/0x6f
 [<ffffffff81012f91>] ? int_ret_from_sys_call+0x7/0x1b
 [<ffffffff8101371d>] ? retint_restore_args+0x5/0x6
 [<ffffffff81013da0>] ? child_rip+0x0/0x20
Code: 41 5e 41 5f c9 c3 55 48 89 e5 0f 1f 44 00 00 e8 17 ff ff ff c9 c3 55 48 89 e5 0f 1f 44 00 00 65 8b 04 25 c8 55 01 00 ff c8 75 04 <0f> 0b eb fe 65 48 8b 34 25 c0 55 01 00 48 81 c6 b8 02 00 00 e8 
RIP  [<ffffffff8103a3cb>] leave_mm+0x15/0x46
 RSP <ffff88002805be48>
---[ end trace ce9cee6832a9c503 ]---
Kernel panic - not syncing: Fatal exception in interrupt
Pid: 25581, comm: khelper Tainted: G      D    2.6.32.36fixxen #1
Call Trace:
 <IRQ>  [<ffffffff8105682e>] panic+0xe0/0x19a
 [<ffffffff8144008a>] ? init_amd+0x296/0x37a
 [<ffffffff8100f17d>] ? xen_force_evtchn_callback+0xd/0xf
 [<ffffffff8100f8e2>] ? check_events+0x12/0x20
 [<ffffffff8100f8cf>] ? xen_restore_fl_direct_end+0x0/0x1
 [<ffffffff81056487>] ? print_oops_end_marker+0x23/0x25
 [<ffffffff81448185>] oops_end+0xb6/0xc6
 [<ffffffff810166e5>] die+0x5a/0x63
 [<ffffffff81447a5c>] do_trap+0x115/0x124
 [<ffffffff810148e6>] do_invalid_op+0x9c/0xa5
 [<ffffffff8103a3cb>] ? leave_mm+0x15/0x46
 [<ffffffff8100f6fa>] ? xen_clocksource_read+0x21/0x23
 [<ffffffff8100f26c>] ? HYPERVISOR_vcpu_op+0xf/0x11
 [<ffffffff8100f767>] ? xen_vcpuop_set_next_event+0x52/0x67
 [<ffffffff81080bfa>] ? clockevents_program_event+0x78/0x81
 [<ffffffff81013b3b>] invalid_op+0x1b/0x20
 [<ffffffff814472b2>] ? _spin_unlock_irqrestore+0x15/0x17
 [<ffffffff8103a3cb>] ? leave_mm+0x15/0x46
 [<ffffffff8100e4ae>] drop_other_mm_ref+0x2a/0x53
 [<ffffffff81087224>] generic_smp_call_function_single_interrupt+0xd8/0xfc
 [<ffffffff81010108>] xen_call_function_single_interrupt+0x13/0x28
 [<ffffffff810a936a>] handle_IRQ_event+0x66/0x120
 [<ffffffff810aac5b>] handle_percpu_irq+0x41/0x6e
 [<ffffffff8128c1c0>] __xen_evtchn_do_upcall+0x1ab/0x27d
 [<ffffffff8128dd11>] xen_evtchn_do_upcall+0x33/0x46
 [<ffffffff81013efe>] xen_do_hypervisor_callback+0x1e/0x30
 <EOI>  [<ffffffff814472b2>] ? _spin_unlock_irqrestore+0x15/0x17
 [<ffffffff8100f8cf>] ? xen_restore_fl_direct_end+0x0/0x1
 [<ffffffff81113f71>] ? flush_old_exec+0x3ac/0x500
 [<ffffffff81150dc5>] ? load_elf_binary+0x0/0x17ef
 [<ffffffff81150dc5>] ? load_elf_binary+0x0/0x17ef
 [<ffffffff8115115d>] ? load_elf_binary+0x398/0x17ef
 [<ffffffff81042fcf>] ? need_resched+0x23/0x2d
 [<ffffffff811f4648>] ? process_measurement+0xc0/0xd7
 [<ffffffff81150dc5>] ? load_elf_binary+0x0/0x17ef
 [<ffffffff81113094>] ? search_binary_handler+0xc8/0x255
 [<ffffffff81114362>] ? do_execve+0x1c3/0x29e
 [<ffffffff8101155d>] ? sys_execve+0x43/0x5d
 [<ffffffff8106fc45>] ? __call_usermodehelper+0x0/0x6f
 [<ffffffff81013e28>] ? kernel_execve+0x68/0xd0
 [<ffffffff8106fc45>] ? __call_usermodehelper+0x0/0x6f
 [<ffffffff8100f8cf>] ? xen_restore_fl_direct_end+0x0/0x1
 [<ffffffff8106fb64>] ? ____call_usermodehelper+0x113/0x11e
 [<ffffffff81013daa>] ? child_rip+0xa/0x20
 [<ffffffff8106fc45>] ? __call_usermodehelper+0x0/0x6f
 [<ffffffff81012f91>] ? int_ret_from_sys_call+0x7/0x1b
 [<ffffffff8101371d>] ? retint_restore_args+0x5/0x6
 [<ffffffff81013da0>] ? child_rip+0x0/0x20
(XEN) Domain 0 crashed: 'noreboot' set - not rebooting.

[-- Attachment #3: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: kernel BUG at arch/x86/xen/mmu.c:1872

[Teck Choon Giam]

2011/4/11 MaoXiaoyun <tinnycloud@hotmail.com>:
> Hi:
>
>      I believe this is the fix at much extent.
>      Since I have my own test cases which with this patch, my test case will
> success in 30 rounds run.
>      Every round takes 8hours.  While without this patch, tests fail evey
> round in 15minutes.
>
>       So this really means fix most of the things.
>
>       But during running, I met another crash, from the log it it looks like
> has relation with
> this BUG, since the crash log shows it is tlb related and this BUG also tlb
> related.

Are you able to run another test with cpuidle=0 cpufreq=none in kernel
boot option?  Just curious whether can you reproduce the tlb bug when
you boot with cpuidle=0 cpufreq=none... ...

>
>       Well, I'm also have poor knowledge of kernel.
>       Hope someone from Xen Devel offer some help.
>
>       Many thanks.
>
>> Date: Mon, 11 Apr 2011 20:16:53 +0800
>> Subject: Re: kernel BUG at arch/x86/xen/mmu.c:1872
>> From: giamteckchoon@gmail.com
>> To: tinnycloud@hotmail.com
>> CC: xen-devel@lists.xensource.com; dave@ivt.com.au;
>> ian.campbell@citrix.com; konrad.wilk@oracle.com; jeremy@goop.org;
>> keir@xen.org
>>
>> >
>> > Hi,
>> >
>> > Sorry, since this mmu related BUG has been troubled me for very
>> > long... I really want to "kill" this BUG but my knowledge in kernel
>> > hacking and/or xen is very limited.
>> >
>> > While waiting for Jeremy or Konrad or others ...
>> >
>> > Many thanks for spending time to track down this mmu related BUG.  I
>> > have backported the commit from
>> >
>> > http://git.kernel.org/?p=linux/kernel/git/jeremy/xen.git;a=commit;h=64141da587241301ce8638cc945f8b67853156ec
>> > to 2.6.32.36 PVOPS kernel and patch attached.  I won't know whether
>> > did I backport it correctly nor does it affects anything.  I am
>> > currently testing the 2.6.32.36 PVOPS kernel with this patch applied
>> > and also unset CONFIG_DEBUG_PAGEALLOC.  Currently running testcrash.sh
>> > loop 1000 as I am unable to reproduce this mmu BUG 1872 in
>> > testcrash.sh loop 100.  Please note that when CONFIG_DEBUG_PAGEALLOC
>> > is unset, I can reproduce this mmu BUG 1872 easily within <50
>> > testcrash.sh loop cycle with PVOPS version 2.6.32.24 to 2.6.32.36
>> > kernel.  Now test with this backport patch to see whether I can
>> > reproduce this mmu BUG... ...
>> >
>> > Kindest regards,
>> > Giam Teck Choon
>> >
>>
>> I have tested with my backport patch and it is working fine as I am
>> unable to reproduce the mmu.c 1872 or 1860 bug with
>> CONFIG_DEBUG_PAGEALLOC not set. I tested with testcrash.sh loop 100
>> and 1000. Now doing testcrash.sh loop 10000.
>>
>> Xiaoyun, is it possible for you to test my patch and see whether can
>> you reproduce the mmu.c 1872/1860 bug?
>>
>> Can anyone of you review my patch?
>>
>> I will post a format patch according to
>> Documentation/SubmittingPatches in my next reply and hopefully can be
>> reviewed.
>>
>> Thanks.
>>
>> Kindest regards,
>> Giam Teck Choon
>

RE: kernel BUG at arch/x86/xen/mmu.c:1872

[MaoXiaoyun]

[-- Attachment #1.1: Type: text/plain, Size: 4966 bytes --]


Hi:
 
       I have just kicked off cpuidle=0 "cpufreq=none" tests.
 
       What is your Xen version?  Do you use the backend driver of 2.6.32.36?
 
       Beside the "TLB BUG ", I've met at least two other issues
       1)Xen4.0.1 + 2.6.32.36 kernel + backend driver from 2.6.31  ==> will cause "Bad grant reference " log in serial output
       2)Xen4.0.1 + 2.6.32.36 kernel with its owen backend driver   ==> will cause disk error like belows.
 
sd 0:0:0:0: rejecting I/O to offline device
sd 0:0:0:0: rejecting I/O to offline device
sd 0:0:0:0: rejecting I/O to offline device
sd 0:0:0:0: rejecting I/O to offline device
sd 0:0:0:0: rejecting I/O to offline device
sd 0:0:0:0: rejecting I/O to offline device
sd 0:0:0:0: rejecting I/O to offline device
sd 0:0:0:0: rejecting I/O to offline device
sd 0:0:0:0: rejecting I/O to offline device
sd 0:0:0:0: rejecting I/O to offline device
sd 0:0:0:0: rejecting I/O to offline device
sd 0:0:0:0: rejecting I/O to offline device
sd 0:0:0:0: rejecting I/O to offline device
end_request: I/O error, dev tdb, sector 28699593
end_request: I/O error, dev tdb, sector 28699673
end_request: I/O error, dev tdb, sector 28699753
end_request: I/O error, dev tdb, sector 28699833
end_request: I/O error, dev tdb, sector 28699913
end_request: I/O error, dev tdb, sector 28699993
end_request: I/O error, dev tdb, sector 28700073
     
    thanks.
 
 
> Date: Mon, 11 Apr 2011 23:25:19 +0800
> Subject: Re: kernel BUG at arch/x86/xen/mmu.c:1872
> From: giamteckchoon@gmail.com
> To: tinnycloud@hotmail.com
> CC: xen-devel@lists.xensource.com; dave@ivt.com.au; ian.campbell@citrix.com; konrad.wilk@oracle.com; jeremy@goop.org; keir@xen.org
> 
> 2011/4/11 MaoXiaoyun <tinnycloud@hotmail.com>:
> > Hi:
> >
> >      I believe this is the fix at much extent.
> >      Since I have my own test cases which with this patch, my test case will
> > success in 30 rounds run.
> >      Every round takes 8hours.  While without this patch, tests fail evey
> > round in 15minutes.
> >
> >       So this really means fix most of the things.
> >
> >       But during running, I met another crash, from the log it it looks like
> > has relation with
> > this BUG, since the crash log shows it is tlb related and this BUG also tlb
> > related.
> 
> Are you able to run another test with cpuidle=0 cpufreq=none in kernel
> boot option? Just curious whether can you reproduce the tlb bug when
> you boot with cpuidle=0 cpufreq=none... ...
> 
> >
> >       Well, I'm also have poor knowledge of kernel.
> >       Hope someone from Xen Devel offer some help.
> >
> >       Many thanks.
> >
> >> Date: Mon, 11 Apr 2011 20:16:53 +0800
> >> Subject: Re: kernel BUG at arch/x86/xen/mmu.c:1872
> >> From: giamteckchoon@gmail.com
> >> To: tinnycloud@hotmail.com
> >> CC: xen-devel@lists.xensource.com; dave@ivt.com.au;
> >> ian.campbell@citrix.com; konrad.wilk@oracle.com; jeremy@goop.org;
> >> keir@xen.org
> >>
> >> >
> >> > Hi,
> >> >
> >> > Sorry, since this mmu related BUG has been troubled me for very
> >> > long... I really want to "kill" this BUG but my knowledge in kernel
> >> > hacking and/or xen is very limited.
> >> >
> >> > While waiting for Jeremy or Konrad or others ...
> >> >
> >> > Many thanks for spending time to track down this mmu related BUG.  I
> >> > have backported the commit from
> >> >
> >> > http://git.kernel.org/?p=linux/kernel/git/jeremy/xen.git;a=commit;h=64141da587241301ce8638cc945f8b67853156ec
> >> > to 2.6.32.36 PVOPS kernel and patch attached.  I won't know whether
> >> > did I backport it correctly nor does it affects anything.  I am
> >> > currently testing the 2.6.32.36 PVOPS kernel with this patch applied
> >> > and also unset CONFIG_DEBUG_PAGEALLOC.  Currently running testcrash.sh
> >> > loop 1000 as I am unable to reproduce this mmu BUG 1872 in
> >> > testcrash.sh loop 100.  Please note that when CONFIG_DEBUG_PAGEALLOC
> >> > is unset, I can reproduce this mmu BUG 1872 easily within <50
> >> > testcrash.sh loop cycle with PVOPS version 2.6.32.24 to 2.6.32.36
> >> > kernel.  Now test with this backport patch to see whether I can
> >> > reproduce this mmu BUG... ...
> >> >
> >> > Kindest regards,
> >> > Giam Teck Choon
> >> >
> >>
> >> I have tested with my backport patch and it is working fine as I am
> >> unable to reproduce the mmu.c 1872 or 1860 bug with
> >> CONFIG_DEBUG_PAGEALLOC not set. I tested with testcrash.sh loop 100
> >> and 1000. Now doing testcrash.sh loop 10000.
> >>
> >> Xiaoyun, is it possible for you to test my patch and see whether can
> >> you reproduce the mmu.c 1872/1860 bug?
> >>
> >> Can anyone of you review my patch?
> >>
> >> I will post a format patch according to
> >> Documentation/SubmittingPatches in my next reply and hopefully can be
> >> reviewed.
> >>
> >> Thanks.
> >>
> >> Kindest regards,
> >> Giam Teck Choon
> >
 		 	   		  

[-- Attachment #1.2: Type: text/html, Size: 7337 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: kernel BUG at arch/x86/xen/mmu.c:1872

[Teck Choon Giam]

If it is possible, please try not to top-post as this make reading
more confusing for me at least.  Thanks ;)


2011/4/12 MaoXiaoyun <tinnycloud@hotmail.com>:
> Hi:
>
>        I have just kicked off cpuidle=0 "cpufreq=none" tests.

Let see whether are you able to reproduce the tlb BUG with the above.

>
>        What is your Xen version?  Do you use the backend driver of
> 2.6.32.36?

You are asking me?
xen-4.0.2-rc3-pre latest changeset and also xen-4.1.1-rc1-pre.
What do you mean backend driver?  My testing are mostly on PV domU and
HVM on windows with LVM as storage.  I do not use VDH or any PV
drivers for windows.

>
>        Beside the "TLB BUG ", I've met at least two other issues
>        1)Xen4.0.1 + 2.6.32.36 kernel + backend driver from 2.6.31  ==> will
> cause "Bad grant reference " log in serial output
>        2)Xen4.0.1 + 2.6.32.36 kernel with its owen backend driver   ==> will
> cause disk error like belows.
>
> sd 0:0:0:0: rejecting I/O to offline device
> sd 0:0:0:0: rejecting I/O to offline device
> sd 0:0:0:0: rejecting I/O to offline device
> sd 0:0:0:0: rejecting I/O to offline device
> sd 0:0:0:0: rejecting I/O to offline device
> sd 0:0:0:0: rejecting I/O to offline device
> sd 0:0:0:0: rejecting I/O to offline device
> sd 0:0:0:0: rejecting I/O to offline device
> sd 0:0:0:0: rejecting I/O to offline device
> sd 0:0:0:0: rejecting I/O to offline device
> sd 0:0:0:0: rejecting I/O to offline device
> sd 0:0:0:0: rejecting I/O to offline device
> sd 0:0:0:0: rejecting I/O to offline device
> end_request: I/O error, dev tdb, sector 28699593
> end_request: I/O error, dev tdb, sector 28699673
> end_request: I/O error, dev tdb, sector 28699753
> end_request: I/O error, dev tdb, sector 28699833
> end_request: I/O error, dev tdb, sector 28699913
> end_request: I/O error, dev tdb, sector 28699993
> end_request: I/O error, dev tdb, sector 28700073

Is this related to VDH?  What is the specific backend driver?  These
started to surface after you applied my backport patch or regardless
the patch applied it is already there?

Thanks.

Kindest regards,
Giam Teck Choon

Re: kernel BUG at arch/x86/xen/mmu.c:1872

[Jeremy Fitzhardinge]

On 04/11/2011 05:31 AM, MaoXiaoyun wrote:
> Hi:
>
> I believe this is the fix at much extent.
> Since I have my own test cases which with this patch, my test case
> will success in 30 rounds run.
> Every round takes 8hours. While without this patch, tests fail evey
> round in 15minutes.
>
> So this really means fix most of the things.
>
> But during running, I met another crash, from the log it it looks like
> has relation with
> this BUG, since the crash log shows it is tlb related and this BUG
> also tlb related.
>
> Well, I'm also have poor knowledge of kernel.
> Hope someone from Xen Devel offer some help.

Thanks for confirming; it makes sense and explains the symptoms, so I'm
glad it also works ;)


J

RE: kernel BUG at arch/x86/xen/mmu.c:1872

[MaoXiaoyun]

[-- Attachment #1.1: Type: text/plain, Size: 1294 bytes --]


Thanks for your reply and comfirm.
 
Well, what's your opinion of TLB bug?
Is it related to this patch or a new bug?
 
Attached it the new log I've got in 28 machine tests, one crashed.
 
> Date: Mon, 11 Apr 2011 11:08:10 -0700
> From: jeremy@goop.org
> To: tinnycloud@hotmail.com
> CC: giamteckchoon@gmail.com; xen-devel@lists.xensource.com; dave@ivt.com.au; ian.campbell@citrix.com; konrad.wilk@oracle.com; keir@xen.org
> Subject: Re: kernel BUG at arch/x86/xen/mmu.c:1872
> 
> On 04/11/2011 05:31 AM, MaoXiaoyun wrote:
> > Hi:
> >
> > I believe this is the fix at much extent.
> > Since I have my own test cases which with this patch, my test case
> > will success in 30 rounds run.
> > Every round takes 8hours. While without this patch, tests fail evey
> > round in 15minutes.
> >
> > So this really means fix most of the things.
> >
> > But during running, I met another crash, from the log it it looks like
> > has relation with
> > this BUG, since the crash log shows it is tlb related and this BUG
> > also tlb related.
> >
> > Well, I'm also have poor knowledge of kernel.
> > Hope someone from Xen Devel offer some help.
> 
> Thanks for confirming; it makes sense and explains the symptoms, so I'm
> glad it also works ;)
> 
> 
> J
 		 	   		  

[-- Attachment #1.2: Type: text/html, Size: 1745 bytes --]

[-- Attachment #2: 195.31.txt --]
[-- Type: text/plain, Size: 6940 bytes --]

_ratelimit: 62 callbacks suppressed
blktap_sysfs_create: adding attributes for dev ffff88009bdb6000
blktap_sysfs_create: adding attributes for dev ffff88009bdb2200
INIT: Id "s0" respawning too fast: disabled for 5 minutes
__ratelimit: 14 callbacks suppressed
blktap_sysfs_destroy
blktap_sysfs_destroy
------------[ cut here ]------------
kernel BUG at arch/x86/mm/tlb.c:61!
invalid opcode: 0000 [#1] SMP 
last sysfs file: /sys/devices/system/xen_memory/xen_memory0/info/current_kb
CPU 1 
Modules linked in: 8021q garp xen_netback xen_blkback blktap blkback_pagemap nbd bridge stp llc autofs4 ipmi_devintf ipmi_si ipmi_msghandler lockd sunrpc bonding ipv6 xenfs dm_multipath video output sbs sbshc parport_pc lp parport ses enclosure snd_seq_dummy snd_seq_oss snd_seq_midi_event snd_seq snd_seq_device serio_raw bnx2 snd_pcm_oss snd_mixer_oss snd_pcm snd_timer iTCO_wdt snd soundcore snd_page_alloc i2c_i801 iTCO_vendor_support i2c_core pcspkr pata_acpi ata_generic ata_piix shpchp mptsas mptscsih mptbase [last unloaded: freq_table]
Pid: 25581, comm: khelper Not tainted 2.6.32.36fixxen #1 Tecal RH2285          
RIP: e030:[<ffffffff8103a3cb>]  [<ffffffff8103a3cb>] leave_mm+0x15/0x46
RSP: e02b:ffff88002805be48  EFLAGS: 00010046
RAX: 0000000000000000 RBX: 0000000000000001 RCX: ffff88015f8e2da0
RDX: ffff88002805be78 RSI: 0000000000000000 RDI: 0000000000000001
RBP: ffff88002805be48 R08: ffff88009d662000 R09: dead000000200200
R10: dead000000100100 R11: ffffffff814472b2 R12: ffff88009bfc1880
R13: ffff880028063020 R14: 00000000000004f6 R15: 0000000000000000
FS:  00007f62362d66e0(0000) GS:ffff880028058000(0000) knlGS:0000000000000000
CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000003aabc11909 CR3: 000000009b8ca000 CR4: 0000000000002660
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process khelper (pid: 25581, threadinfo ffff88007691e000, task ffff88009b92db40)
Stack:
 ffff88002805be68 ffffffff8100e4ae 0000000000000001 ffff88009d733b88
<0> ffff88002805be98 ffffffff81087224 ffff88002805be78 ffff88002805be78
<0> ffff88015f808360 00000000000004f6 ffff88002805bea8 ffffffff81010108
Call Trace:
 <IRQ> 
 [<ffffffff8100e4ae>] drop_other_mm_ref+0x2a/0x53
 [<ffffffff81087224>] generic_smp_call_function_single_interrupt+0xd8/0xfc
 [<ffffffff81010108>] xen_call_function_single_interrupt+0x13/0x28
 [<ffffffff810a936a>] handle_IRQ_event+0x66/0x120
 [<ffffffff810aac5b>] handle_percpu_irq+0x41/0x6e
 [<ffffffff8128c1c0>] __xen_evtchn_do_upcall+0x1ab/0x27d
 [<ffffffff8128dd11>] xen_evtchn_do_upcall+0x33/0x46
 [<ffffffff81013efe>] xen_do_hypervisor_callback+0x1e/0x30
 <EOI> 
 [<ffffffff814472b2>] ? _spin_unlock_irqrestore+0x15/0x17
 [<ffffffff8100f8cf>] ? xen_restore_fl_direct_end+0x0/0x1
 [<ffffffff81113f71>] ? flush_old_exec+0x3ac/0x500
 [<ffffffff81150dc5>] ? load_elf_binary+0x0/0x17ef
 [<ffffffff81150dc5>] ? load_elf_binary+0x0/0x17ef
 [<ffffffff8115115d>] ? load_elf_binary+0x398/0x17ef
 [<ffffffff81042fcf>] ? need_resched+0x23/0x2d
 [<ffffffff811f4648>] ? process_measurement+0xc0/0xd7
 [<ffffffff81150dc5>] ? load_elf_binary+0x0/0x17ef
 [<ffffffff81113094>] ? search_binary_handler+0xc8/0x255
 [<ffffffff81114362>] ? do_execve+0x1c3/0x29e
 [<ffffffff8101155d>] ? sys_execve+0x43/0x5d
 [<ffffffff8106fc45>] ? __call_usermodehelper+0x0/0x6f
 [<ffffffff81013e28>] ? kernel_execve+0x68/0xd0
 [<ffffffff8106fc45>] ? __call_usermodehelper+0x0/0x6f
 [<ffffffff8100f8cf>] ? xen_restore_fl_direct_end+0x0/0x1
 [<ffffffff8106fb64>] ? ____call_usermodehelper+0x113/0x11e
 [<ffffffff81013daa>] ? child_rip+0xa/0x20
 [<ffffffff8106fc45>] ? __call_usermodehelper+0x0/0x6f
 [<ffffffff81012f91>] ? int_ret_from_sys_call+0x7/0x1b
 [<ffffffff8101371d>] ? retint_restore_args+0x5/0x6
 [<ffffffff81013da0>] ? child_rip+0x0/0x20
Code: 41 5e 41 5f c9 c3 55 48 89 e5 0f 1f 44 00 00 e8 17 ff ff ff c9 c3 55 48 89 e5 0f 1f 44 00 00 65 8b 04 25 c8 55 01 00 ff c8 75 04 <0f> 0b eb fe 65 48 8b 34 25 c0 55 01 00 48 81 c6 b8 02 00 00 e8 
RIP  [<ffffffff8103a3cb>] leave_mm+0x15/0x46
 RSP <ffff88002805be48>
---[ end trace ce9cee6832a9c503 ]---
Kernel panic - not syncing: Fatal exception in interrupt
Pid: 25581, comm: khelper Tainted: G      D    2.6.32.36fixxen #1
Call Trace:
 <IRQ>  [<ffffffff8105682e>] panic+0xe0/0x19a
 [<ffffffff8144008a>] ? init_amd+0x296/0x37a
 [<ffffffff8100f17d>] ? xen_force_evtchn_callback+0xd/0xf
 [<ffffffff8100f8e2>] ? check_events+0x12/0x20
 [<ffffffff8100f8cf>] ? xen_restore_fl_direct_end+0x0/0x1
 [<ffffffff81056487>] ? print_oops_end_marker+0x23/0x25
 [<ffffffff81448185>] oops_end+0xb6/0xc6
 [<ffffffff810166e5>] die+0x5a/0x63
 [<ffffffff81447a5c>] do_trap+0x115/0x124
 [<ffffffff810148e6>] do_invalid_op+0x9c/0xa5
 [<ffffffff8103a3cb>] ? leave_mm+0x15/0x46
 [<ffffffff8100f6fa>] ? xen_clocksource_read+0x21/0x23
 [<ffffffff8100f26c>] ? HYPERVISOR_vcpu_op+0xf/0x11
 [<ffffffff8100f767>] ? xen_vcpuop_set_next_event+0x52/0x67
 [<ffffffff81080bfa>] ? clockevents_program_event+0x78/0x81
 [<ffffffff81013b3b>] invalid_op+0x1b/0x20
 [<ffffffff814472b2>] ? _spin_unlock_irqrestore+0x15/0x17
 [<ffffffff8103a3cb>] ? leave_mm+0x15/0x46
 [<ffffffff8100e4ae>] drop_other_mm_ref+0x2a/0x53
 [<ffffffff81087224>] generic_smp_call_function_single_interrupt+0xd8/0xfc
 [<ffffffff81010108>] xen_call_function_single_interrupt+0x13/0x28
 [<ffffffff810a936a>] handle_IRQ_event+0x66/0x120
 [<ffffffff810aac5b>] handle_percpu_irq+0x41/0x6e
 [<ffffffff8128c1c0>] __xen_evtchn_do_upcall+0x1ab/0x27d
 [<ffffffff8128dd11>] xen_evtchn_do_upcall+0x33/0x46
 [<ffffffff81013efe>] xen_do_hypervisor_callback+0x1e/0x30
 <EOI>  [<ffffffff814472b2>] ? _spin_unlock_irqrestore+0x15/0x17
 [<ffffffff8100f8cf>] ? xen_restore_fl_direct_end+0x0/0x1
 [<ffffffff81113f71>] ? flush_old_exec+0x3ac/0x500
 [<ffffffff81150dc5>] ? load_elf_binary+0x0/0x17ef
 [<ffffffff81150dc5>] ? load_elf_binary+0x0/0x17ef
 [<ffffffff8115115d>] ? load_elf_binary+0x398/0x17ef
 [<ffffffff81042fcf>] ? need_resched+0x23/0x2d
 [<ffffffff811f4648>] ? process_measurement+0xc0/0xd7
 [<ffffffff81150dc5>] ? load_elf_binary+0x0/0x17ef
 [<ffffffff81113094>] ? search_binary_handler+0xc8/0x255
 [<ffffffff81114362>] ? do_execve+0x1c3/0x29e
 [<ffffffff8101155d>] ? sys_execve+0x43/0x5d
 [<ffffffff8106fc45>] ? __call_usermodehelper+0x0/0x6f
 [<ffffffff81013e28>] ? kernel_execve+0x68/0xd0
 [<ffffffff8106fc45>] ? __call_usermodehelper+0x0/0x6f
 [<ffffffff8100f8cf>] ? xen_restore_fl_direct_end+0x0/0x1
 [<ffffffff8106fb64>] ? ____call_usermodehelper+0x113/0x11e
 [<ffffffff81013daa>] ? child_rip+0xa/0x20
 [<ffffffff8106fc45>] ? __call_usermodehelper+0x0/0x6f
 [<ffffffff81012f91>] ? int_ret_from_sys_call+0x7/0x1b
 [<ffffffff8101371d>] ? retint_restore_args+0x5/0x6
 [<ffffffff81013da0>] ? child_rip+0x0/0x20
(XEN) Domain 0 crashed: 'noreboot' set - not rebooting.

[-- Attachment #3: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Grant Table Error on 2.6.32.36 + Xen 4.0.1

[MaoXiaoyun]

[-- Attachment #1.1: Type: text/plain, Size: 16628 bytes --]


Hi:
 
      We are just about to try the new Kernel, but confront Error on grant table.
        
     2.6.32.36 Kernel: http://git.kernel.org/?p=linux/kernel/git/jeremy/xen.git;a=commit;h=bb1a15e55ec665a64c8a9c6bd699b1f16ac01ff4
     Xen 4.0.1 http://xenbits.xen.org/hg/xen-4.0-testing.hg/rev/b536ebfba183
       
    Our test is simple, 24 HVMS(Win2003 )  on a single host, each HVM loopes in restart every 15minutes.
    Please refer to error log from serial output 
              
    I've traced the log a bit, and the log is from xen/common/grant_table.c
 
1) log " grant_table.c:1717:d0 Bad grant reference 4294965983 " if from 

1715     if ( unlikely(gref >= nr_grant_entries(rd->grant_table)) ){
1716         PIN_FAIL(unlock_out, GNTST_bad_gntref,
1717                  "Bad grant reference %ld\n", gref);
1718         BUG();
1719     }
 
2) log "grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0) " is from 
 
  grant_table.c:1967 =>  __acquire_grant_for_copy  => _set_status
 
 ( not from __gnttab_map_grant_ref, since I add some log to identify this )

The log shows that all are from gnttab_copy, which I later found only netback
has grant copy hypercall. 
 
I also tried netback code from 2.6.31(which works well with kernel 2.6.31), but
still met these errors. So it looks like it is kernel related.
 
What happened for this, will this harmful for the usage of HVM?
 
Many thanks.
 
 =-=====================================
(XEN) Xen trace buffers: disabled
(XEN) Std. Loglevel: Errors and warnings
(XEN) Guest Loglevel: Nothing (Rate-limited: Errors and warnings)
(XEN) Xen is relinquishing VGA console.
(XEN) *** Serial input -> DOM0 (type 'CTRL-a' three times to switch input to Xen)
(XEN) Freed 168kB init memory.
(XEN) grant_table.c:1717:d0 Bad grant reference 4294965983
(XEN) grant_table.c:1717:d0 Bad grant reference 4294965983
(XEN) grant_table.c:1717:d0 Bad grant reference 4294965983
(XEN) grant_table.c:1717:d0 Bad grant reference 4294965983
(XEN) grant_table.c:1717:d0 Bad grant reference 4294965983
(XEN) grant_table.c:1717:d0 Bad grant reference 4294965983
(XEN) grant_table.c:1717:d0 Bad grant reference 4294965983
(XEN) grant_table.c:1717:d0 Bad grant reference 4294965983
(XEN) grant_table.c:1717:d0 Bad grant reference 4294965983
(XEN) grant_table.c:1717:d0 Bad grant reference 4294965983
(XEN) printk: 3 messages suppressed.
(XEN) grant_table.c:1717:d0 Bad grant reference 4294965983
(XEN) printk: 3 messages suppressed.
(XEN) grant_table.c:1717:d0 Bad grant reference 4294965983
(XEN) printk: 3 messages suppressed.
(XEN) grant_table.c:1717:d0 Bad grant reference 4294965983
(XEN) printk: 1 messages suppressed.
(XEN) grant_table.c:1717:d0 Bad grant reference 4294965983
(XEN) printk: 1 messages suppressed.
(XEN) grant_table.c:1717:d0 Bad grant reference 4294965983
(XEN) grant_table.c:1717:d0 Bad grant reference 4294965983
(XEN) printk: 1 messages suppressed.
(XEN) grant_table.c:1717:d0 Bad grant reference 4294965983
(XEN) printk: 1 messages suppressed.
(XEN) grant_table.c:1717:d0 Bad grant reference 4294965983
(XEN) printk: 1 messages suppressed.
(XEN) grant_table.c:1717:d0 Bad grant reference 4294965983
(XEN) printk: 1 messages suppressed.
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) printk: 3 messages suppressed.
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) printk: 17 messages suppressed.
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) printk: 13 messages suppressed.
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) printk: 11 messages suppressed.
(XEN) grant_table.c:1717:d0 Bad grant reference 4294965983
(XEN) printk: 11 messages suppressed.
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) printk: 10 messages suppressed.
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) printk: 3 messages suppressed.
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) printk: 2 messages suppressed.
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) grant_table.c:1717:d0 Bad grant reference 4294965983
(XEN) grant_table.c:1717:d0 Bad grant reference 4294965983
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) grant_table.c:1717:d0 Bad grant reference 4294965983
(XEN) grant_table.c:1717:d0 Bad grant reference 4294965983
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) printk: 6 messages suppressed.
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) printk: 7 messages suppressed.
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) printk: 2 messages suppressed.
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) printk: 3 messages suppressed.
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) printk: 2 messages suppressed.
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) grant_table.c:1717:d0 Bad grant reference 4294965983
(XEN) grant_table.c:1717:d0 Bad grant reference 4294965983
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) grant_table.c:1717:d0 Bad grant reference 4294965983
(XEN) grant_table.c:1717:d0 Bad grant reference 4294965983
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) grant_table.c:1717:d0 Bad grant reference 4294965983
(XEN) printk: 10 messages suppressed.
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) grant_table.c:1717:d0 Bad grant reference 4294965983
(XEN) grant_table.c:1717:d0 Bad grant reference 4294965983
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) grant_table.c:1717:d0 Bad grant reference 4294965983
(XEN) grant_table.c:1717:d0 Bad grant reference 4294965983
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) grant_table.c:1717:d0 Bad grant reference 4294965983
(XEN) grant_table.c:1717:d0 Bad grant reference 4294965983
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) grant_table.c:1717:d0 Bad grant reference 4294965983
(XEN) printk: 1 messages suppressed.
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) grant_table.c:1717:d0 Bad grant reference 4294965983
(XEN) grant_table.c:1717:d0 Bad grant reference 4294965983
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) grant_table.c:1717:d0 Bad grant reference 4294965983
(XEN) grant_table.c:1717:d0 Bad grant reference 4294965983
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) printk: 3 messages suppressed.
(XEN) grant_table.c:1717:d0 Bad grant reference 4294965983
(XEN) printk: 15 messages suppressed.
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) printk: 7 messages suppressed.
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) printk: 7 messages suppressed.
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) printk: 3 messages suppressed.
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) printk: 7 messages suppressed.
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) printk: 3 messages suppressed.
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) printk: 3 messages suppressed.
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) printk: 3 messages suppressed.
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) printk: 3 messages suppressed.
(XEN) grant_table.c:1717:d0 Bad grant reference 4294965983
(XEN) grant_table.c:1717:d0 Bad grant reference 4294965983
(XEN) printk: 8 messages suppressed.
(XEN) grant_table.c:1717:d0 Bad grant reference 4294965983
(XEN) printk: 15 messages suppressed.
(XEN) grant_table.c:1717:d0 Bad grant reference 4294965983
(XEN) printk: 7 messages suppressed.
(XEN) grant_table.c:1717:d0 Bad grant reference 4294965983
(XEN) printk: 29 messages suppressed.
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) printk: 25 messages suppressed.
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) printk: 25 messages suppressed.
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) printk: 19 messages suppressed.
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) printk: 27 messages suppressed.
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) printk: 27 messages suppressed.
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) printk: 7 messages suppressed.
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) printk: 3 messages suppressed.
(XEN) grant_table.c:1717:d0 Bad grant reference 4294965983
(XEN) grant_table.c:1717:d0 Bad grant reference 4294965983
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) printk: 5 messages suppressed.
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) grant_table.c:1717:d0 Bad grant reference 4294965983
(XEN) printk: 10 messages suppressed.
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) printk: 7 messages suppressed.
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) printk: 2 messages suppressed.
(XEN) grant_table.c:1717:d0 Bad grant reference 4294965983
(XEN) printk: 3 messages suppressed.
(XEN) grant_table.c:1717:d0 Bad grant reference 4294965983
(XEN) printk: 15 messages suppressed.
(XEN) grant_table.c:1717:d0 Bad grant reference 4294965983
(XEN) printk: 7 messages suppressed.
(XEN) grant_table.c:1717:d0 Bad grant reference 4294965983
(XEN) grant_table.c:1717:d0 Bad grant reference 4294965983
(XEN) printk: 2 messages suppressed.
(XEN) grant_table.c:1717:d0 Bad grant reference 4294965983
(XEN) grant_table.c:1717:d0 Bad grant reference 4294965983
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) printk: 1 messages suppressed.
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) printk: 8 messages suppressed.
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) grant_table.c:1717:d0 Bad grant reference 4294965983
(XEN) grant_table.c:1717:d0 Bad grant reference 4294965983
(XEN) grant_table.c:1717:d0 Bad grant reference 4294965983
(XEN) grant_table.c:1717:d0 Bad grant reference 4294965983
(XEN) grant_table.c:1717:d0 Bad grant reference 4294965983
(XEN) printk: 1 messages suppressed.
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) printk: 3 messages suppressed.
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) printk: 3 messages suppressed.
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) grant_table.c:1717:d0 Bad grant reference 4294965983
(XEN) printk: 1 messages suppressed.
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) grant_table.c:1717:d0 Bad grant reference 4294965983
(XEN) printk: 1 messages suppressed.
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) printk: 3 messages suppressed.
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) printk: 3 messages suppressed.
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) grant_table.c:1717:d0 Bad grant reference 4294965983
(XEN) printk: 8 messages suppressed.
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) printk: 9 messages suppressed.
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) printk: 1 messages suppressed.
(XEN) grant_table.c:1717:d0 Bad grant reference 4294965983
(XEN) grant_table.c:1717:d0 Bad grant reference 4294965983
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) grant_table.c:1717:d0 Bad grant reference 4294965983
(XEN) grant_table.c:1717:d0 Bad grant reference 4294965983
(XEN) grant_table.c:1717:d0 Bad grant reference 4294965983
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) printk: 2 messages suppressed.
(XEN) grant_table.c:1717:d0 Bad grant reference 4294965983
(XEN) printk: 7 messages suppressed.
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) printk: 5 messages suppressed.
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) printk: 2 messages suppressed.
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) printk: 2 messages suppressed.
(XEN) grant_table.c:1717:d0 Bad grant reference 4294965983
(XEN) printk: 1 messages suppressed.
(XEN) grant_table.c:1717:d0 Bad grant reference 4294965983
(XEN) printk: 3 messages suppressed.
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) printk: 3 messages suppressed.
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) printk: 1 messages suppressed.
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) printk: 1 messages suppressed.
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) printk: 1 messages suppressed.
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0)
(XEN) grant_table.c:578:d0 Iomem mapping not permitted ffffffffffffffff (domain 137)
(XEN) grant_table.c:578:d0 Iomem mapping not permitted ffffffffffffffff (domain 137)
(XEN) grant_table.c:578:d0 Iomem mapping not permitted ffffffffffffffff (domain 137)
(XEN) grant_table.c:578:d0 Iomem mapping not permitted ffffffffffffffff (domain 137)
(XEN) grant_table.c:578:d0 Iomem mapping not permitted ffffffffffffffff (domain 137)
 		 	   		  

[-- Attachment #1.2: Type: text/html, Size: 18335 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Grant Table Error on 2.6.32.36 + Xen 4.0.1

[Konrad Rzeszutek Wilk]

On Tue, Apr 12, 2011 at 02:48:36PM +0800, MaoXiaoyun wrote:
> 
> Hi:
>  
>       We are just about to try the new Kernel, but confront Error on grant table.

Please open a new thread on this one. This is getting confusing.
>         
>      2.6.32.36 Kernel: http://git.kernel.org/?p=linux/kernel/git/jeremy/xen.git;a=commit;h=bb1a15e55ec665a64c8a9c6bd699b1f16ac01ff4
>      Xen 4.0.1 http://xenbits.xen.org/hg/xen-4.0-testing.hg/rev/b536ebfba183
>        
>     Our test is simple, 24 HVMS(Win2003 )  on a single host, each HVM loopes in restart every 15minutes.
>     Please refer to error log from serial output 
>               
>     I've traced the log a bit, and the log is from xen/common/grant_table.c
>  
> 1) log " grant_table.c:1717:d0 Bad grant reference 4294965983 " if from 
> 
> 1715     if ( unlikely(gref >= nr_grant_entries(rd->grant_table)) ){
> 1716         PIN_FAIL(unlock_out, GNTST_bad_gntref,
> 1717                  "Bad grant reference %ld\n", gref);
> 1718         BUG();
> 1719     }
>  
> 2) log "grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0) " is from 
>  
>   grant_table.c:1967 =>  __acquire_grant_for_copy  => _set_status
>  
>  ( not from __gnttab_map_grant_ref, since I add some log to identify this )
> 
> The log shows that all are from gnttab_copy, which I later found only netback
> has grant copy hypercall. 
>  
> I also tried netback code from 2.6.31(which works well with kernel 2.6.31), but
> still met these errors. So it looks like it is kernel related.
>  
> What happened for this, will this harmful for the usage of HVM?

What is the storage for your HVM guests? iSCSI?

RE: Grant Table Error on 2.6.32.36 + Xen 4.0.1

[MaoXiaoyun]

[-- Attachment #1.1: Type: text/plain, Size: 8727 bytes --]


Thanks Konrad.
 
I will new a thread on TLB bug.
For grant table error.  I add some debug log on netback.c , line 388. 
 
 358 static u16 netbk_gop_frag(struct xen_netif *netif, struct netbk_rx_meta *meta,
 359                           int i, struct netrx_pending_operations *npo,
 360                           struct page *page, unsigned long size,
 361                           unsigned long offset)
 362 {
 363         struct gnttab_copy *copy_gop;
 364         struct xen_netif_rx_request *req;
 365         unsigned long old_mfn;
 366         int idx = netif_page_index(page);
 367 
 368         old_mfn = virt_to_mfn(page_address(page));
 369 
 370         req = RING_GET_REQUEST(&netif->rx, netif->rx.req_cons + i);
 371 
 372         copy_gop = npo->copy + npo->copy_prod++;
 373         copy_gop->flags = GNTCOPY_dest_gref;
 374         if (idx > -1) {
 375                 struct pending_tx_info *src_pend = &pending_tx_info[idx];
 376                 copy_gop->source.domid = src_pend->netif->domid;
 377                 copy_gop->source.u.ref = src_pend->req.gref;
 378                 copy_gop->flags |= GNTCOPY_source_gref;
 379         } else {
 380                 copy_gop->source.domid = DOMID_SELF;
 381                 copy_gop->source.u.gmfn = old_mfn;
 382         }
 383         copy_gop->source.offset = offset;
 384         copy_gop->dest.domid = netif->domid;
 385         copy_gop->dest.offset = 0;
 386         copy_gop->dest.u.ref = req->gref;
 387         copy_gop->len = size;
 388         if(req->gref > 16384)
 389            IPRINTK("dom %d, req gref %d size = %lu\n", netif->domid, req->gref, size);
 390 
 391         return req->id;
 392 }
 
And the output below, indicates something might wrong on grant table.
 
Apr 12 16:38:31 xmao kernel: xen_net: dom 23, req gref -1313 size = 270
Apr 12 16:38:31 xmao kernel: xen_net: dom 23, req gref -1313 size = 72
Apr 12 16:38:31 xmao kernel: xen_net: dom 14, req gref -1313 size = 270
Apr 12 16:38:31 xmao kernel: xen_net: dom 14, req gref -1313 size = 72
Apr 12 16:38:33 xmao kernel: xen_net: dom 23, req gref -1313 size = 270
Apr 12 16:38:33 xmao kernel: xen_net: dom 23, req gref -1313 size = 72
Apr 12 16:38:34 xmao kernel: xen_net: dom 23, req gref -1313 size = 270
Apr 12 16:38:34 xmao kernel: xen_net: dom 23, req gref -1313 size = 72
Apr 12 16:38:34 xmao kernel: xen_net: dom 14, req gref -1313 size = 270
Apr 12 16:38:35 xmao kernel: xen_net: dom 23, req gref -1313 size = 270
Apr 12 16:38:35 xmao kernel: xen_net: dom 23, req gref -1313 size = 72
Apr 12 16:38:40 xmao kernel: xen_net: dom 23, req gref -1313 size = 270
Apr 12 16:38:40 xmao kernel: xen_net: dom 23, req gref -1313 size = 72
Apr 12 16:38:42 xmao kernel: xen_net: dom 23, req gref -1313 size = 270
Apr 12 16:38:42 xmao kernel: xen_net: dom 23, req gref -1313 size = 72
Apr 12 16:38:44 xmao kernel: xen_net: dom 23, req gref -1313 size = 270
Apr 12 16:38:44 xmao kernel: xen_net: dom 23, req gref -1313 size = 72
Apr 12 16:38:57 xmao kernel: xen_net: dom 23, req gref -1313 size = 270
Apr 12 16:38:57 xmao kernel: xen_net: dom 23, req gref -1313 size = 72
Apr 12 16:38:59 xmao kernel: xen_net: dom 23, req gref -1313 size = 270
Apr 12 16:38:59 xmao kernel: xen_net: dom 23, req gref -1313 size = 72
Apr 12 16:38:59 xmao kernel: xen_net: dom 23, req gref -1313 size = 270
Apr 12 16:38:59 xmao kernel: xen_net: dom 23, req gref -1313 size = 72
Apr 12 16:39:22 xmao kernel: xen_net: dom 23, req gref -1313 size = 270
Apr 12 16:39:22 xmao kernel: xen_net: dom 23, req gref -1313 size = 72
Apr 12 16:39:26 xmao kernel: xen_net: dom 23, req gref -1313 size = 270
Apr 12 16:39:26 xmao kernel: xen_net: dom 23, req gref -1313 size = 72
Apr 12 16:39:29 xmao kernel: xen_net: dom 23, req gref -1313 size = 42
Apr 12 16:39:29 xmao kernel: xen_net: dom 14, req gref -1313 size = 42
Apr 12 16:39:29 xmao kernel: xen_net: dom 23, req gref -1313 size = 42
Apr 12 16:39:29 xmao kernel: xen_net: dom 14, req gref 5242956 size = 42
Apr 12 16:39:30 xmao kernel: xen_net: dom 23, req gref -1313 size = 42
Apr 12 16:39:30 xmao kernel: xen_net: dom 14, req gref 1817341261 size = 42
Apr 12 16:39:31 xmao kernel: xen_net: dom 23, req gref -1313 size = 42
Apr 12 16:39:31 xmao kernel: xen_net: dom 23, req gref -1313 size = 38
Apr 12 16:39:31 xmao kernel: xen_net: dom 23, req gref -1313 size = 72
Apr 12 16:39:31 xmao kernel: xen_net: dom 14, req gref -1313 size = 38
Apr 12 16:39:31 xmao kernel: xen_net: dom 14, req gref -1313 size = 72
Apr 12 16:39:32 xmao kernel: xen_net: dom 23, req gref -1313 size = 270
Apr 12 16:39:32 xmao kernel: xen_net: dom 23, req gref -1313 size = 72
Apr 12 16:39:32 xmao kernel: xen_net: dom 23, req gref -1313 size = 270
Apr 12 16:39:32 xmao kernel: xen_net: dom 23, req gref -1313 size = 72
Apr 12 16:39:32 xmao kernel: xen_net: dom 23, req gref -1313 size = 42
Apr 12 16:39:32 xmao kernel: xen_net: dom 14, req gref -1408 size = 42
Apr 12 16:39:32 xmao kernel: xen_net: dom 23, req gref -1313 size = 38
Apr 12 16:39:32 xmao kernel: xen_net: dom 23, req gref -1313 size = 72
Apr 12 16:39:32 xmao kernel: xen_net: dom 14, req gref -1408 size = 38
Apr 12 16:39:32 xmao kernel: xen_net: dom 14, req gref -1408 size = 72
Apr 12 16:39:33 xmao kernel: xen_net: dom 23, req gref -1313 size = 42
Apr 12 16:39:33 xmao kernel: xen_net: dom 14, req gref -1408 size = 42
Apr 12 16:39:33 xmao kernel: xen_net: dom 23, req gref -1313 size = 42
Apr 12 16:39:33 xmao kernel: xen_net: dom 14, req gref -1408 size = 42
Apr 12 16:39:33 xmao kernel: xen_net: dom 23, req gref -1313 size = 38
Apr 12 16:39:33 xmao kernel: xen_net: dom 23, req gref -1313 size = 72
Apr 12 16:39:33 xmao kernel: xen_net: dom 14, req gref 1850305869 size = 38
Apr 12 16:39:33 xmao kernel: xen_net: dom 23, req gref -1313 size = 42
Apr 12 16:39:34 xmao kernel: xen_net: dom 23, req gref -1313 size = 38
Apr 12 16:39:34 xmao kernel: xen_net: dom 23, req gref -1313 size = 72
Apr 12 16:39:34 xmao kernel: xen_net: dom 14, req gref -1313 size = 38
Apr 12 16:39:34 xmao kernel: xen_net: dom 14, req gref -1313 size = 72
Apr 12 16:39:34 xmao kernel: xen_net: dom 23, req gref -1313 size = 42
Apr 12 16:39:34 xmao kernel: xen_net: dom 14, req gref -1313 size = 42
Apr 12 16:39:35 xmao kernel: xen_net: dom 23, req gref -1313 size = 270
Apr 12 16:39:35 xmao kernel: xen_net: dom 23, req gref -1313 size = 72
Apr 12 16:39:35 xmao kernel: xen_net: dom 23, req gref -1313 size = 38
Apr 12 16:39:35 xmao kernel: xen_net: dom 23, req gref -1313 size = 72
Apr 12 16:39:35 xmao kernel: xen_net: dom 23, req gref -1313 size = 38
Apr 12 16:39:35 xmao kernel: xen_net: dom 23, req gref -1313 size = 72
 
> Date: Tue, 12 Apr 2011 04:46:29 -0400
> From: konrad.wilk@oracle.com
> To: tinnycloud@hotmail.com
> CC: xen-devel@lists.xensource.com; tim.deegan@citrix.com; george.dunlap@eu.citrix.com; giamteckchoon@gmail.com; ian.campbell@citrix.com; keir.fraser@eu.citrix.com
> Subject: Re: [Xen-devel] Grant Table Error on 2.6.32.36 + Xen 4.0.1
> 
> On Tue, Apr 12, 2011 at 02:48:36PM +0800, MaoXiaoyun wrote:
> > 
> > Hi:
> > 
> > We are just about to try the new Kernel, but confront Error on grant table.
> 
> Please open a new thread on this one. This is getting confusing.
> > 
> > 2.6.32.36 Kernel: http://git.kernel.org/?p=linux/kernel/git/jeremy/xen.git;a=commit;h=bb1a15e55ec665a64c8a9c6bd699b1f16ac01ff4
> > Xen 4.0.1 http://xenbits.xen.org/hg/xen-4.0-testing.hg/rev/b536ebfba183
> > 
> > Our test is simple, 24 HVMS(Win2003 ) on a single host, each HVM loopes in restart every 15minutes.
> > Please refer to error log from serial output 
> > 
> > I've traced the log a bit, and the log is from xen/common/grant_table.c
> > 
> > 1) log " grant_table.c:1717:d0 Bad grant reference 4294965983 " if from 
> > 
> > 1715 if ( unlikely(gref >= nr_grant_entries(rd->grant_table)) ){
> > 1716 PIN_FAIL(unlock_out, GNTST_bad_gntref,
> > 1717 "Bad grant reference %ld\n", gref);
> > 1718 BUG();
> > 1719 }
> > 
> > 2) log "grant_table.c:266:d0 Bad flags (0) or dom (0). (expected dom 0) " is from 
> > 
> > grant_table.c:1967 => __acquire_grant_for_copy => _set_status
> > 
> > ( not from __gnttab_map_grant_ref, since I add some log to identify this )
> > 
> > The log shows that all are from gnttab_copy, which I later found only netback
> > has grant copy hypercall. 
> > 
> > I also tried netback code from 2.6.31(which works well with kernel 2.6.31), but
> > still met these errors. So it looks like it is kernel related.
> > 
> > What happened for this, will this harmful for the usage of HVM?
> 
> What is the storage for your HVM guests? iSCSI?
 		 	   		  

[-- Attachment #1.2: Type: text/html, Size: 11480 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Kernel BUG at arch/x86/mm/tlb.c:61

[MaoXiaoyun]

[-- Attachment #1.1: Type: text/plain, Size: 7435 bytes --]


Hi :
 
  We are using pvops kernel 2.6.32.36 + xen 4.0.1, but confront a kernel panic bug.
 
  2.6.32.36 Kernel: http://git.kernel.org/?p=linux/kernel/git/jeremy/xen.git;a=commit;h=bb1a15e55ec665a64c8a9c6bd699b1f16ac01ff4
  Xen 4.0.1 http://xenbits.xen.org/hg/xen-4.0-testing.hg/rev/b536ebfba183  
 
  Our test is simple, 24 HVMS(Win2003 )  on a single host, each HVM loopes in restart every 15minutes.
  About 17 machines are invovled in the test,  after 10 hours run, one confrontted a crash at arch/x86/mm/tlb.c:61
 
  Currently I am trying "cpuidle=0 cpufreq=none" tests based on Teck's suggestion.
 
  Any comments, thanks. 
 
===============crash log==========================
INIT: Id "s0" respawning too fast: disabled for 5 minutes
__ratelimit: 14 callbacks suppressed
blktap_sysfs_destroy
blktap_sysfs_destroy
------------[ cut here ]------------
kernel BUG at arch/x86/mm/tlb.c:61!
invalid opcode: 0000 [#1] SMP 
last sysfs file: /sys/devices/system/xen_memory/xen_memory0/info/current_kb
CPU 1 
Modules linked in: 8021q garp xen_netback xen_blkback blktap blkback_pagemap nbd bridge stp llc autofs4 ipmi_devintf ipmi_si ipmi_msghandler lockd sunrpc bonding ipv6 xenfs dm_multipath video output sbs sbshc parport_pc lp parport ses enclosure snd_seq_dummy snd_seq_oss snd_seq_midi_event snd_seq snd_seq_device serio_raw bnx2 snd_pcm_oss snd_mixer_oss snd_pcm snd_timer iTCO_wdt snd soundcore snd_page_alloc i2c_i801 iTCO_vendor_support i2c_core pcspkr pata_acpi ata_generic ata_piix shpchp mptsas mptscsih mptbase [last unloaded: freq_table]
Pid: 25581, comm: khelper Not tainted 2.6.32.36fixxen #1 Tecal RH2285          
RIP: e030:[<ffffffff8103a3cb>]  [<ffffffff8103a3cb>] leave_mm+0x15/0x46
RSP: e02b:ffff88002805be48  EFLAGS: 00010046
RAX: 0000000000000000 RBX: 0000000000000001 RCX: ffff88015f8e2da0
RDX: ffff88002805be78 RSI: 0000000000000000 RDI: 0000000000000001
RBP: ffff88002805be48 R08: ffff88009d662000 R09: dead000000200200
R10: dead000000100100 R11: ffffffff814472b2 R12: ffff88009bfc1880
R13: ffff880028063020 R14: 00000000000004f6 R15: 0000000000000000
FS:  00007f62362d66e0(0000) GS:ffff880028058000(0000) knlGS:0000000000000000
CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000003aabc11909 CR3: 000000009b8ca000 CR4: 0000000000002660
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process khelper (pid: 25581, threadinfo ffff88007691e000, task ffff88009b92db40)
Stack:
 ffff88002805be68 ffffffff8100e4ae 0000000000000001 ffff88009d733b88
<0> ffff88002805be98 ffffffff81087224 ffff88002805be78 ffff88002805be78
<0> ffff88015f808360 00000000000004f6 ffff88002805bea8 ffffffff81010108
Call Trace:
 <IRQ> 
 [<ffffffff8100e4ae>] drop_other_mm_ref+0x2a/0x53
 [<ffffffff81087224>] generic_smp_call_function_single_interrupt+0xd8/0xfc
 [<ffffffff81010108>] xen_call_function_single_interrupt+0x13/0x28
 [<ffffffff810a936a>] handle_IRQ_event+0x66/0x120
 [<ffffffff810aac5b>] handle_percpu_irq+0x41/0x6e
 [<ffffffff8128c1c0>] __xen_evtchn_do_upcall+0x1ab/0x27d
 [<ffffffff8128dd11>] xen_evtchn_do_upcall+0x33/0x46
 [<ffffffff81013efe>] xen_do_hypervisor_callback+0x1e/0x30
 <EOI> 
 [<ffffffff814472b2>] ? _spin_unlock_irqrestore+0x15/0x17
 [<ffffffff8100f8cf>] ? xen_restore_fl_direct_end+0x0/0x1
 [<ffffffff81113f71>] ? flush_old_exec+0x3ac/0x500
 [<ffffffff81150dc5>] ? load_elf_binary+0x0/0x17ef
 [<ffffffff81150dc5>] ? load_elf_binary+0x0/0x17ef
 [<ffffffff8115115d>] ? load_elf_binary+0x398/0x17ef
 [<ffffffff81042fcf>] ? need_resched+0x23/0x2d
 [<ffffffff811f4648>] ? process_measurement+0xc0/0xd7
 [<ffffffff81150dc5>] ? load_elf_binary+0x0/0x17ef
 [<ffffffff81113094>] ? search_binary_handler+0xc8/0x255
 [<ffffffff81114362>] ? do_execve+0x1c3/0x29e
 [<ffffffff8101155d>] ? sys_execve+0x43/0x5d
 [<ffffffff8106fc45>] ? __call_usermodehelper+0x0/0x6f
 [<ffffffff81013e28>] ? kernel_execve+0x68/0xd0
 [<ffffffff8106fc45>] ? __call_usermodehelper+0x0/0x6f
 [<ffffffff8100f8cf>] ? xen_restore_fl_direct_end+0x0/0x1
 [<ffffffff8106fb64>] ? ____call_usermodehelper+0x113/0x11e
 [<ffffffff81013daa>] ? child_rip+0xa/0x20
 [<ffffffff8106fc45>] ? __call_usermodehelper+0x0/0x6f
 [<ffffffff81012f91>] ? int_ret_from_sys_call+0x7/0x1b
 [<ffffffff8101371d>] ? retint_restore_args+0x5/0x6
 [<ffffffff81013da0>] ? child_rip+0x0/0x20
Code: 41 5e 41 5f c9 c3 55 48 89 e5 0f 1f 44 00 00 e8 17 ff ff ff c9 c3 55 48 89 e5 0f 1f 44 00 00 65 8b 04 25 c8 55 01 00 ff c8 75 04 <0f> 0b eb fe 65 48 8b 34 25 c0 55 01 00 48 81 c6 b8 02 00 00 e8 
RIP  [<ffffffff8103a3cb>] leave_mm+0x15/0x46
 RSP <ffff88002805be48>
---[ end trace ce9cee6832a9c503 ]---
Kernel panic - not syncing: Fatal exception in interrupt
Pid: 25581, comm: khelper Tainted: G      D    2.6.32.36fixxen #1
Call Trace:
 <IRQ>  [<ffffffff8105682e>] panic+0xe0/0x19a
 [<ffffffff8144008a>] ? init_amd+0x296/0x37a
 [<ffffffff8100f17d>] ? xen_force_evtchn_callback+0xd/0xf
 [<ffffffff8100f8e2>] ? check_events+0x12/0x20
 [<ffffffff8100f8cf>] ? xen_restore_fl_direct_end+0x0/0x1
 [<ffffffff81056487>] ? print_oops_end_marker+0x23/0x25
 [<ffffffff81448185>] oops_end+0xb6/0xc6
 [<ffffffff810166e5>] die+0x5a/0x63
 [<ffffffff81447a5c>] do_trap+0x115/0x124
 [<ffffffff810148e6>] do_invalid_op+0x9c/0xa5
 [<ffffffff8103a3cb>] ? leave_mm+0x15/0x46
 [<ffffffff8100f6fa>] ? xen_clocksource_read+0x21/0x23
 [<ffffffff8100f26c>] ? HYPERVISOR_vcpu_op+0xf/0x11
 [<ffffffff8100f767>] ? xen_vcpuop_set_next_event+0x52/0x67
 [<ffffffff81080bfa>] ? clockevents_program_event+0x78/0x81
 [<ffffffff81013b3b>] invalid_op+0x1b/0x20
 [<ffffffff814472b2>] ? _spin_unlock_irqrestore+0x15/0x17
 [<ffffffff8103a3cb>] ? leave_mm+0x15/0x46
 [<ffffffff8100e4ae>] drop_other_mm_ref+0x2a/0x53
 [<ffffffff81087224>] generic_smp_call_function_single_interrupt+0xd8/0xfc
 [<ffffffff81010108>] xen_call_function_single_interrupt+0x13/0x28
 [<ffffffff810a936a>] handle_IRQ_event+0x66/0x120
 [<ffffffff810aac5b>] handle_percpu_irq+0x41/0x6e
 [<ffffffff8128c1c0>] __xen_evtchn_do_upcall+0x1ab/0x27d
 [<ffffffff8128dd11>] xen_evtchn_do_upcall+0x33/0x46
 [<ffffffff81013efe>] xen_do_hypervisor_callback+0x1e/0x30
 <EOI>  [<ffffffff814472b2>] ? _spin_unlock_irqrestore+0x15/0x17
 [<ffffffff8100f8cf>] ? xen_restore_fl_direct_end+0x0/0x1
 [<ffffffff81113f71>] ? flush_old_exec+0x3ac/0x500
 [<ffffffff81150dc5>] ? load_elf_binary+0x0/0x17ef
 [<ffffffff81150dc5>] ? load_elf_binary+0x0/0x17ef
 [<ffffffff8115115d>] ? load_elf_binary+0x398/0x17ef
 [<ffffffff81042fcf>] ? need_resched+0x23/0x2d
 [<ffffffff811f4648>] ? process_measurement+0xc0/0xd7
 [<ffffffff81150dc5>] ? load_elf_binary+0x0/0x17ef
 [<ffffffff81113094>] ? search_binary_handler+0xc8/0x255
 [<ffffffff81114362>] ? do_execve+0x1c3/0x29e
 [<ffffffff8101155d>] ? sys_execve+0x43/0x5d
 [<ffffffff8106fc45>] ? __call_usermodehelper+0x0/0x6f
 [<ffffffff81013e28>] ? kernel_execve+0x68/0xd0
 [<ffffffff8106fc45>] ? __call_usermodehelper+0x0/0x6f
 [<ffffffff8100f8cf>] ? xen_restore_fl_direct_end+0x0/0x1
 [<ffffffff8106fb64>] ? ____call_usermodehelper+0x113/0x11e
 [<ffffffff81013daa>] ? child_rip+0xa/0x20
 [<ffffffff8106fc45>] ? __call_usermodehelper+0x0/0x6f
 [<ffffffff81012f91>] ? int_ret_from_sys_call+0x7/0x1b
 [<ffffffff8101371d>] ? retint_restore_args+0x5/0x6
 [<ffffffff81013da0>] ? child_rip+0x0/0x20
  
  		 	   		  

[-- Attachment #1.2: Type: text/html, Size: 12935 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Kernel BUG at arch/x86/mm/tlb.c:61

[Konrad Rzeszutek Wilk]

On Tue, Apr 12, 2011 at 05:11:51PM +0800, MaoXiaoyun wrote:
> 
> Hi :
>  
>   We are using pvops kernel 2.6.32.36 + xen 4.0.1, but confront a kernel panic bug.
>  
>   2.6.32.36 Kernel: http://git.kernel.org/?p=linux/kernel/git/jeremy/xen.git;a=commit;h=bb1a15e55ec665a64c8a9c6bd699b1f16ac01ff4
>   Xen 4.0.1 http://xenbits.xen.org/hg/xen-4.0-testing.hg/rev/b536ebfba183  
>  
>   Our test is simple, 24 HVMS(Win2003 )  on a single host, each HVM loopes in restart every 15minutes.

What is the storage that you are using for your guests? AoE? Local disks?

>   About 17 machines are invovled in the test,  after 10 hours run, one confrontted a crash at arch/x86/mm/tlb.c:61
>  
>   Currently I am trying "cpuidle=0 cpufreq=none" tests based on Teck's suggestion.
>  
>   Any comments, thanks. 
>  
> ===============crash log==========================
> INIT: Id "s0" respawning too fast: disabled for 5 minutes
> __ratelimit: 14 callbacks suppressed
> blktap_sysfs_destroy
> blktap_sysfs_destroy
> ------------[ cut here ]------------
> kernel BUG at arch/x86/mm/tlb.c:61!
> invalid opcode: 0000 [#1] SMP 
> last sysfs file: /sys/devices/system/xen_memory/xen_memory0/info/current_kb
> CPU 1 
> Modules linked in: 8021q garp xen_netback xen_blkback blktap blkback_pagemap nbd bridge stp llc autofs4 ipmi_devintf ipmi_si ipmi_msghandler lockd sunrpc bonding ipv6 xenfs dm_multipath video output sbs sbshc parport_pc lp parport ses enclosure snd_seq_dummy snd_seq_oss snd_seq_midi_event snd_seq snd_seq_device serio_raw bnx2 snd_pcm_oss snd_mixer_oss snd_pcm snd_timer iTCO_wdt snd soundcore snd_page_alloc i2c_i801 iTCO_vendor_support i2c_core pcspkr pata_acpi ata_generic ata_piix shpchp mptsas mptscsih mptbase [last unloaded: freq_table]
> Pid: 25581, comm: khelper Not tainted 2.6.32.36fixxen #1 Tecal RH2285          
> RIP: e030:[<ffffffff8103a3cb>]  [<ffffffff8103a3cb>] leave_mm+0x15/0x46
> RSP: e02b:ffff88002805be48  EFLAGS: 00010046
> RAX: 0000000000000000 RBX: 0000000000000001 RCX: ffff88015f8e2da0
> RDX: ffff88002805be78 RSI: 0000000000000000 RDI: 0000000000000001
> RBP: ffff88002805be48 R08: ffff88009d662000 R09: dead000000200200
> R10: dead000000100100 R11: ffffffff814472b2 R12: ffff88009bfc1880
> R13: ffff880028063020 R14: 00000000000004f6 R15: 0000000000000000
> FS:  00007f62362d66e0(0000) GS:ffff880028058000(0000) knlGS:0000000000000000
> CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
> CR2: 0000003aabc11909 CR3: 000000009b8ca000 CR4: 0000000000002660
> DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> Process khelper (pid: 25581, threadinfo ffff88007691e000, task ffff88009b92db40)
> Stack:
>  ffff88002805be68 ffffffff8100e4ae 0000000000000001 ffff88009d733b88
> <0> ffff88002805be98 ffffffff81087224 ffff88002805be78 ffff88002805be78
> <0> ffff88015f808360 00000000000004f6 ffff88002805bea8 ffffffff81010108
> Call Trace:
>  <IRQ> 
>  [<ffffffff8100e4ae>] drop_other_mm_ref+0x2a/0x53
>  [<ffffffff81087224>] generic_smp_call_function_single_interrupt+0xd8/0xfc
>  [<ffffffff81010108>] xen_call_function_single_interrupt+0x13/0x28
>  [<ffffffff810a936a>] handle_IRQ_event+0x66/0x120
>  [<ffffffff810aac5b>] handle_percpu_irq+0x41/0x6e
>  [<ffffffff8128c1c0>] __xen_evtchn_do_upcall+0x1ab/0x27d
>  [<ffffffff8128dd11>] xen_evtchn_do_upcall+0x33/0x46
>  [<ffffffff81013efe>] xen_do_hypervisor_callback+0x1e/0x30
>  <EOI> 
>  [<ffffffff814472b2>] ? _spin_unlock_irqrestore+0x15/0x17
>  [<ffffffff8100f8cf>] ? xen_restore_fl_direct_end+0x0/0x1
>  [<ffffffff81113f71>] ? flush_old_exec+0x3ac/0x500
>  [<ffffffff81150dc5>] ? load_elf_binary+0x0/0x17ef
>  [<ffffffff81150dc5>] ? load_elf_binary+0x0/0x17ef
>  [<ffffffff8115115d>] ? load_elf_binary+0x398/0x17ef
>  [<ffffffff81042fcf>] ? need_resched+0x23/0x2d
>  [<ffffffff811f4648>] ? process_measurement+0xc0/0xd7
>  [<ffffffff81150dc5>] ? load_elf_binary+0x0/0x17ef
>  [<ffffffff81113094>] ? search_binary_handler+0xc8/0x255
>  [<ffffffff81114362>] ? do_execve+0x1c3/0x29e
>  [<ffffffff8101155d>] ? sys_execve+0x43/0x5d
>  [<ffffffff8106fc45>] ? __call_usermodehelper+0x0/0x6f
>  [<ffffffff81013e28>] ? kernel_execve+0x68/0xd0
>  [<ffffffff8106fc45>] ? __call_usermodehelper+0x0/0x6f
>  [<ffffffff8100f8cf>] ? xen_restore_fl_direct_end+0x0/0x1
>  [<ffffffff8106fb64>] ? ____call_usermodehelper+0x113/0x11e
>  [<ffffffff81013daa>] ? child_rip+0xa/0x20
>  [<ffffffff8106fc45>] ? __call_usermodehelper+0x0/0x6f
>  [<ffffffff81012f91>] ? int_ret_from_sys_call+0x7/0x1b
>  [<ffffffff8101371d>] ? retint_restore_args+0x5/0x6
>  [<ffffffff81013da0>] ? child_rip+0x0/0x20
> Code: 41 5e 41 5f c9 c3 55 48 89 e5 0f 1f 44 00 00 e8 17 ff ff ff c9 c3 55 48 89 e5 0f 1f 44 00 00 65 8b 04 25 c8 55 01 00 ff c8 75 04 <0f> 0b eb fe 65 48 8b 34 25 c0 55 01 00 48 81 c6 b8 02 00 00 e8 
> RIP  [<ffffffff8103a3cb>] leave_mm+0x15/0x46
>  RSP <ffff88002805be48>
> ---[ end trace ce9cee6832a9c503 ]---
> Kernel panic - not syncing: Fatal exception in interrupt
> Pid: 25581, comm: khelper Tainted: G      D    2.6.32.36fixxen #1
> Call Trace:
>  <IRQ>  [<ffffffff8105682e>] panic+0xe0/0x19a
>  [<ffffffff8144008a>] ? init_amd+0x296/0x37a
>  [<ffffffff8100f17d>] ? xen_force_evtchn_callback+0xd/0xf
>  [<ffffffff8100f8e2>] ? check_events+0x12/0x20
>  [<ffffffff8100f8cf>] ? xen_restore_fl_direct_end+0x0/0x1
>  [<ffffffff81056487>] ? print_oops_end_marker+0x23/0x25
>  [<ffffffff81448185>] oops_end+0xb6/0xc6
>  [<ffffffff810166e5>] die+0x5a/0x63
>  [<ffffffff81447a5c>] do_trap+0x115/0x124
>  [<ffffffff810148e6>] do_invalid_op+0x9c/0xa5
>  [<ffffffff8103a3cb>] ? leave_mm+0x15/0x46
>  [<ffffffff8100f6fa>] ? xen_clocksource_read+0x21/0x23
>  [<ffffffff8100f26c>] ? HYPERVISOR_vcpu_op+0xf/0x11
>  [<ffffffff8100f767>] ? xen_vcpuop_set_next_event+0x52/0x67
>  [<ffffffff81080bfa>] ? clockevents_program_event+0x78/0x81
>  [<ffffffff81013b3b>] invalid_op+0x1b/0x20
>  [<ffffffff814472b2>] ? _spin_unlock_irqrestore+0x15/0x17
>  [<ffffffff8103a3cb>] ? leave_mm+0x15/0x46
>  [<ffffffff8100e4ae>] drop_other_mm_ref+0x2a/0x53
>  [<ffffffff81087224>] generic_smp_call_function_single_interrupt+0xd8/0xfc
>  [<ffffffff81010108>] xen_call_function_single_interrupt+0x13/0x28
>  [<ffffffff810a936a>] handle_IRQ_event+0x66/0x120
>  [<ffffffff810aac5b>] handle_percpu_irq+0x41/0x6e
>  [<ffffffff8128c1c0>] __xen_evtchn_do_upcall+0x1ab/0x27d
>  [<ffffffff8128dd11>] xen_evtchn_do_upcall+0x33/0x46
>  [<ffffffff81013efe>] xen_do_hypervisor_callback+0x1e/0x30
>  <EOI>  [<ffffffff814472b2>] ? _spin_unlock_irqrestore+0x15/0x17
>  [<ffffffff8100f8cf>] ? xen_restore_fl_direct_end+0x0/0x1
>  [<ffffffff81113f71>] ? flush_old_exec+0x3ac/0x500
>  [<ffffffff81150dc5>] ? load_elf_binary+0x0/0x17ef
>  [<ffffffff81150dc5>] ? load_elf_binary+0x0/0x17ef
>  [<ffffffff8115115d>] ? load_elf_binary+0x398/0x17ef
>  [<ffffffff81042fcf>] ? need_resched+0x23/0x2d
>  [<ffffffff811f4648>] ? process_measurement+0xc0/0xd7
>  [<ffffffff81150dc5>] ? load_elf_binary+0x0/0x17ef
>  [<ffffffff81113094>] ? search_binary_handler+0xc8/0x255
>  [<ffffffff81114362>] ? do_execve+0x1c3/0x29e
>  [<ffffffff8101155d>] ? sys_execve+0x43/0x5d
>  [<ffffffff8106fc45>] ? __call_usermodehelper+0x0/0x6f
>  [<ffffffff81013e28>] ? kernel_execve+0x68/0xd0
>  [<ffffffff8106fc45>] ? __call_usermodehelper+0x0/0x6f
>  [<ffffffff8100f8cf>] ? xen_restore_fl_direct_end+0x0/0x1
>  [<ffffffff8106fb64>] ? ____call_usermodehelper+0x113/0x11e
>  [<ffffffff81013daa>] ? child_rip+0xa/0x20
>  [<ffffffff8106fc45>] ? __call_usermodehelper+0x0/0x6f
>  [<ffffffff81012f91>] ? int_ret_from_sys_call+0x7/0x1b
>  [<ffffffff8101371d>] ? retint_restore_args+0x5/0x6
>  [<ffffffff81013da0>] ? child_rip+0x0/0x20
>   
>   		 	   		  

RE: Kernel BUG at arch/x86/mm/tlb.c:61

[MaoXiaoyun]

[-- Attachment #1.1: Type: text/plain, Size: 8334 bytes --]


VHD file in local disk. 
 
disk = [ 'tap:vhd:/mnt/xmao/test/img/win2003.cp1.vhd,hda,w']
 
thanks.
 
> Date: Tue, 12 Apr 2011 06:00:00 -0400
> From: konrad.wilk@oracle.com
> To: tinnycloud@hotmail.com
> CC: xen-devel@lists.xensource.com; giamteckchoon@gmail.com; jeremy@goop.org
> Subject: Re: Kernel BUG at arch/x86/mm/tlb.c:61
> 
> On Tue, Apr 12, 2011 at 05:11:51PM +0800, MaoXiaoyun wrote:
> > 
> > Hi :
> > 
> > We are using pvops kernel 2.6.32.36 + xen 4.0.1, but confront a kernel panic bug.
> > 
> > 2.6.32.36 Kernel: http://git.kernel.org/?p=linux/kernel/git/jeremy/xen.git;a=commit;h=bb1a15e55ec665a64c8a9c6bd699b1f16ac01ff4
> > Xen 4.0.1 http://xenbits.xen.org/hg/xen-4.0-testing.hg/rev/b536ebfba183 
> > 
> > Our test is simple, 24 HVMS(Win2003 ) on a single host, each HVM loopes in restart every 15minutes.
> 
> What is the storage that you are using for your guests? AoE? Local disks?
> 
> > About 17 machines are invovled in the test, after 10 hours run, one confrontted a crash at arch/x86/mm/tlb.c:61
> > 
> > Currently I am trying "cpuidle=0 cpufreq=none" tests based on Teck's suggestion.
> > 
> > Any comments, thanks. 
> > 
> > ===============crash log==========================
> > INIT: Id "s0" respawning too fast: disabled for 5 minutes
> > __ratelimit: 14 callbacks suppressed
> > blktap_sysfs_destroy
> > blktap_sysfs_destroy
> > ------------[ cut here ]------------
> > kernel BUG at arch/x86/mm/tlb.c:61!
> > invalid opcode: 0000 [#1] SMP 
> > last sysfs file: /sys/devices/system/xen_memory/xen_memory0/info/current_kb
> > CPU 1 
> > Modules linked in: 8021q garp xen_netback xen_blkback blktap blkback_pagemap nbd bridge stp llc autofs4 ipmi_devintf ipmi_si ipmi_msghandler lockd sunrpc bonding ipv6 xenfs dm_multipath video output sbs sbshc parport_pc lp parport ses enclosure snd_seq_dummy snd_seq_oss snd_seq_midi_event snd_seq snd_seq_device serio_raw bnx2 snd_pcm_oss snd_mixer_oss snd_pcm snd_timer iTCO_wdt snd soundcore snd_page_alloc i2c_i801 iTCO_vendor_support i2c_core pcspkr pata_acpi ata_generic ata_piix shpchp mptsas mptscsih mptbase [last unloaded: freq_table]
> > Pid: 25581, comm: khelper Not tainted 2.6.32.36fixxen #1 Tecal RH2285 
> > RIP: e030:[<ffffffff8103a3cb>] [<ffffffff8103a3cb>] leave_mm+0x15/0x46
> > RSP: e02b:ffff88002805be48 EFLAGS: 00010046
> > RAX: 0000000000000000 RBX: 0000000000000001 RCX: ffff88015f8e2da0
> > RDX: ffff88002805be78 RSI: 0000000000000000 RDI: 0000000000000001
> > RBP: ffff88002805be48 R08: ffff88009d662000 R09: dead000000200200
> > R10: dead000000100100 R11: ffffffff814472b2 R12: ffff88009bfc1880
> > R13: ffff880028063020 R14: 00000000000004f6 R15: 0000000000000000
> > FS: 00007f62362d66e0(0000) GS:ffff880028058000(0000) knlGS:0000000000000000
> > CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b
> > CR2: 0000003aabc11909 CR3: 000000009b8ca000 CR4: 0000000000002660
> > DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> > DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> > Process khelper (pid: 25581, threadinfo ffff88007691e000, task ffff88009b92db40)
> > Stack:
> > ffff88002805be68 ffffffff8100e4ae 0000000000000001 ffff88009d733b88
> > <0> ffff88002805be98 ffffffff81087224 ffff88002805be78 ffff88002805be78
> > <0> ffff88015f808360 00000000000004f6 ffff88002805bea8 ffffffff81010108
> > Call Trace:
> > <IRQ> 
> > [<ffffffff8100e4ae>] drop_other_mm_ref+0x2a/0x53
> > [<ffffffff81087224>] generic_smp_call_function_single_interrupt+0xd8/0xfc
> > [<ffffffff81010108>] xen_call_function_single_interrupt+0x13/0x28
> > [<ffffffff810a936a>] handle_IRQ_event+0x66/0x120
> > [<ffffffff810aac5b>] handle_percpu_irq+0x41/0x6e
> > [<ffffffff8128c1c0>] __xen_evtchn_do_upcall+0x1ab/0x27d
> > [<ffffffff8128dd11>] xen_evtchn_do_upcall+0x33/0x46
> > [<ffffffff81013efe>] xen_do_hypervisor_callback+0x1e/0x30
> > <EOI> 
> > [<ffffffff814472b2>] ? _spin_unlock_irqrestore+0x15/0x17
> > [<ffffffff8100f8cf>] ? xen_restore_fl_direct_end+0x0/0x1
> > [<ffffffff81113f71>] ? flush_old_exec+0x3ac/0x500
> > [<ffffffff81150dc5>] ? load_elf_binary+0x0/0x17ef
> > [<ffffffff81150dc5>] ? load_elf_binary+0x0/0x17ef
> > [<ffffffff8115115d>] ? load_elf_binary+0x398/0x17ef
> > [<ffffffff81042fcf>] ? need_resched+0x23/0x2d
> > [<ffffffff811f4648>] ? process_measurement+0xc0/0xd7
> > [<ffffffff81150dc5>] ? load_elf_binary+0x0/0x17ef
> > [<ffffffff81113094>] ? search_binary_handler+0xc8/0x255
> > [<ffffffff81114362>] ? do_execve+0x1c3/0x29e
> > [<ffffffff8101155d>] ? sys_execve+0x43/0x5d
> > [<ffffffff8106fc45>] ? __call_usermodehelper+0x0/0x6f
> > [<ffffffff81013e28>] ? kernel_execve+0x68/0xd0
> > [<ffffffff8106fc45>] ? __call_usermodehelper+0x0/0x6f
> > [<ffffffff8100f8cf>] ? xen_restore_fl_direct_end+0x0/0x1
> > [<ffffffff8106fb64>] ? ____call_usermodehelper+0x113/0x11e
> > [<ffffffff81013daa>] ? child_rip+0xa/0x20
> > [<ffffffff8106fc45>] ? __call_usermodehelper+0x0/0x6f
> > [<ffffffff81012f91>] ? int_ret_from_sys_call+0x7/0x1b
> > [<ffffffff8101371d>] ? retint_restore_args+0x5/0x6
> > [<ffffffff81013da0>] ? child_rip+0x0/0x20
> > Code: 41 5e 41 5f c9 c3 55 48 89 e5 0f 1f 44 00 00 e8 17 ff ff ff c9 c3 55 48 89 e5 0f 1f 44 00 00 65 8b 04 25 c8 55 01 00 ff c8 75 04 <0f> 0b eb fe 65 48 8b 34 25 c0 55 01 00 48 81 c6 b8 02 00 00 e8 
> > RIP [<ffffffff8103a3cb>] leave_mm+0x15/0x46
> > RSP <ffff88002805be48>
> > ---[ end trace ce9cee6832a9c503 ]---
> > Kernel panic - not syncing: Fatal exception in interrupt
> > Pid: 25581, comm: khelper Tainted: G D 2.6.32.36fixxen #1
> > Call Trace:
> > <IRQ> [<ffffffff8105682e>] panic+0xe0/0x19a
> > [<ffffffff8144008a>] ? init_amd+0x296/0x37a
> > [<ffffffff8100f17d>] ? xen_force_evtchn_callback+0xd/0xf
> > [<ffffffff8100f8e2>] ? check_events+0x12/0x20
> > [<ffffffff8100f8cf>] ? xen_restore_fl_direct_end+0x0/0x1
> > [<ffffffff81056487>] ? print_oops_end_marker+0x23/0x25
> > [<ffffffff81448185>] oops_end+0xb6/0xc6
> > [<ffffffff810166e5>] die+0x5a/0x63
> > [<ffffffff81447a5c>] do_trap+0x115/0x124
> > [<ffffffff810148e6>] do_invalid_op+0x9c/0xa5
> > [<ffffffff8103a3cb>] ? leave_mm+0x15/0x46
> > [<ffffffff8100f6fa>] ? xen_clocksource_read+0x21/0x23
> > [<ffffffff8100f26c>] ? HYPERVISOR_vcpu_op+0xf/0x11
> > [<ffffffff8100f767>] ? xen_vcpuop_set_next_event+0x52/0x67
> > [<ffffffff81080bfa>] ? clockevents_program_event+0x78/0x81
> > [<ffffffff81013b3b>] invalid_op+0x1b/0x20
> > [<ffffffff814472b2>] ? _spin_unlock_irqrestore+0x15/0x17
> > [<ffffffff8103a3cb>] ? leave_mm+0x15/0x46
> > [<ffffffff8100e4ae>] drop_other_mm_ref+0x2a/0x53
> > [<ffffffff81087224>] generic_smp_call_function_single_interrupt+0xd8/0xfc
> > [<ffffffff81010108>] xen_call_function_single_interrupt+0x13/0x28
> > [<ffffffff810a936a>] handle_IRQ_event+0x66/0x120
> > [<ffffffff810aac5b>] handle_percpu_irq+0x41/0x6e
> > [<ffffffff8128c1c0>] __xen_evtchn_do_upcall+0x1ab/0x27d
> > [<ffffffff8128dd11>] xen_evtchn_do_upcall+0x33/0x46
> > [<ffffffff81013efe>] xen_do_hypervisor_callback+0x1e/0x30
> > <EOI> [<ffffffff814472b2>] ? _spin_unlock_irqrestore+0x15/0x17
> > [<ffffffff8100f8cf>] ? xen_restore_fl_direct_end+0x0/0x1
> > [<ffffffff81113f71>] ? flush_old_exec+0x3ac/0x500
> > [<ffffffff81150dc5>] ? load_elf_binary+0x0/0x17ef
> > [<ffffffff81150dc5>] ? load_elf_binary+0x0/0x17ef
> > [<ffffffff8115115d>] ? load_elf_binary+0x398/0x17ef
> > [<ffffffff81042fcf>] ? need_resched+0x23/0x2d
> > [<ffffffff811f4648>] ? process_measurement+0xc0/0xd7
> > [<ffffffff81150dc5>] ? load_elf_binary+0x0/0x17ef
> > [<ffffffff81113094>] ? search_binary_handler+0xc8/0x255
> > [<ffffffff81114362>] ? do_execve+0x1c3/0x29e
> > [<ffffffff8101155d>] ? sys_execve+0x43/0x5d
> > [<ffffffff8106fc45>] ? __call_usermodehelper+0x0/0x6f
> > [<ffffffff81013e28>] ? kernel_execve+0x68/0xd0
> > [<ffffffff8106fc45>] ? __call_usermodehelper+0x0/0x6f
> > [<ffffffff8100f8cf>] ? xen_restore_fl_direct_end+0x0/0x1
> > [<ffffffff8106fb64>] ? ____call_usermodehelper+0x113/0x11e
> > [<ffffffff81013daa>] ? child_rip+0xa/0x20
> > [<ffffffff8106fc45>] ? __call_usermodehelper+0x0/0x6f
> > [<ffffffff81012f91>] ? int_ret_from_sys_call+0x7/0x1b
> > [<ffffffff8101371d>] ? retint_restore_args+0x5/0x6
> > [<ffffffff81013da0>] ? child_rip+0x0/0x20
> > 
> > 
 		 	   		  

[-- Attachment #1.2: Type: text/html, Size: 10255 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

RE: Kernel BUG at arch/x86/mm/tlb.c:61

[MaoXiaoyun]

[-- Attachment #1.1: Type: text/plain, Size: 15036 bytes --]


Hi:
 
      I've done test with "cpuidle=0 cpufreq=none", two machine crashed.
 
blktap_sysfs_destroy
blktap_sysfs_destroy
blktap_sysfs_create: adding attributes for dev ffff8800ad581000
blktap_sysfs_create: adding attributes for dev ffff8800a48e3e00
------------[ cut here ]------------
kernel BUG at arch/x86/mm/tlb.c:61!
invalid opcode: 0000 [#1] SMP 
last sysfs file: /sys/block/tapdeve/dev
CPU 0 
Modules linked in: 8021q garp blktap xen_netback xen_blkback blkback_pagemap nbd bridge stp llc autofs4 ipmi_devintf ipmi_si ipmi_ms
ghandler lockd sunrpc bonding ipv6 xenfs dm_multipath video output sbs sbshc parport_pc lp parport ses enclosure snd_seq_dummy bnx2 
serio_raw snd_seq_oss snd_seq_midi_event snd_seq snd_seq_device snd_pcm_oss snd_mixer_oss snd_pcm i2c_i801 snd_timer i2c_core snd iT
CO_wdt pata_acpi soundcore iTCO_vendor_
support ata_generic snd_page_alloc pcspkr ata_piix shpchp mptsas mptscsih mptbase [last unloa
ded: freq_table]
Pid: 8022, comm: khelper Not tainted 2.6.32.36xen #1 Tecal RH2285          
RIP: e030:[<ffffffff8103a3cb>]  [<ffffffff8103a3cb>] leave_mm+0x15/0x46
RSP: e02b:ffff88002803ee48  EFLAGS: 00010046
RAX: 0000000000000000 RBX: 0000000000000001 RCX: ffffffff81675980
RDX: ffff88002803ee78 RSI: 0000000000000000 RDI: 0000000000000000
RBP: ffff88002803ee48 R08: ffff8800a4929000 R09: dead000000200200
R10: dead000000100100 R11: ffffffff81447292 R12: ffff88012ba07b80
R13: ffff880028046020 R14: 00000000000004fb R15: 0000000000000000
FS:  00007f410af416e0(0000) GS:ffff88002803b000(0000) knlGS:0000000000000000
CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000469000 CR3: 00000000ad639000 CR4: 0000000000002660
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process khelper (pid: 8022, threadinfo ffff8800a4846000, task ffff8800a9ed0000)
Stack:
 ffff88002803ee68 ffffffff8100e4a4 0000000000000001 ffff880097de3b88
<0> ffff88002803ee98 ffffffff81087224 ffff88002803ee78 ffff88002803ee78
<0> ffff88015f808180 00000000000004fb ffff88002803eea8 ffffffff810100e8
Call Trace:
 <IRQ> 
 [<ffffffff8100e4a4>] drop_other_mm_ref+0x2a/0x53
 [<ffffffff81087224>] generic_smp_call_function_single_interrupt+0xd8/0xfc
 [<ffffffff810100e8>] xen_call_function_single_interrupt+0x13/0x28
 [<ffffffff810a936a>] handle_IRQ_event+0x66/0x120
 [<ffffffff810aac5b>] handle_percpu_irq+0x41/0x6e
 [<ffffffff8128c1a8>] __xen_evtchn_do_upcall+0x1ab/0x27d
 [<ffffffff8128dcf9>] xen_evtchn_do_upcall+0x33/0x46
 [<ffffffff81013efe>] xen_do_hypervisor_callback+0x1e/0x30
 <EOI> 
 [<ffffffff81447292>] ? _spin_unlock_irqrestore+0x15/0x17
 [<ffffffff8100f8af>] ? xen_restore_fl_direct_end+0x0/0x1
 [<ffffffff81113f75>] ? flush_old_exec+0x3ac/0x500
 [<ffffffff81150dc9>] ? load_elf_binary+0x0/0x17ef
 [<ffffffff81150dc9>] ? load_elf_binary+0x0/0x17ef
 [<ffffffff81151161>] ? load_elf_binary+0x398/0x17ef
 [<ffffffff81042fcf>] ? need_resched+0x23/0x2d
 
[<ffffffff811f463c>] ? process_measurement+0xc0/0xd7
 [<ffffffff81150dc9>] ? load_elf_binary+0x0/0x17ef
 [<ffffffff81113098>] ? search_binary_handler+0xc8/0x255
 [<ffffffff81114366>] ? do_execve+0x1c3/0x29e
 [<ffffffff8101155d>] ? sys_execve+0x43/0x5d
 [<ffffffff8106fc45>] ? __call_usermodehelper+0x0/0x6f
 [<ffffffff81013e28>] ? kernel_execve+0x68/0xd0
 [<ffffffff8106fc45>] ? __call_usermodehelper+0x0/0x6f
 [<ffffffff8100f8af>] ? xen_restore_fl_direct_end+0x0/0x1
 [<ffffffff8106fb64>] ? ____call_usermodehelper+0x113/0x11e
 [<ffffffff81013daa>] ? child_rip+0xa/0x20
 [<ffffffff8106fc45>] ? __call_usermodehelper+0x0/0x6f
 [<ffffffff81012f91>] ? int_ret_from_sys_call+0x7/0x1b
 [<ffffffff8101371d>] ? retint_restore_args+0x5/0x6
 [<ffffffff81013da0>] ? c
hild_rip+0x0/0x20
Code: 41 5e 41 5f c9 c3 55 48 89 e5 0f 1f 44 00 00 e8 17 ff ff ff c9 c3 55 48 89 e5 0f 1f 44 00 00 65 8b 04 25 c8 55 01 00 ff c8 75 04 <0f> 0b eb fe 65 48 8b 34 25 c0 55 01 00 48 81 c6 b8 02 00 00 e8 
RIP  [<ffffffff8103a3cb>] leave_mm+0x15/0x46
 RSP <ffff88002803ee48>
---[ end trace 1522f17fdfc9162d ]---
Kernel panic - not syncing: Fatal exception in interrupt
Pid: 8022, comm: khelper Tainted: G      D    2.6.32.36xen #1
Call Trace:
 <IRQ>  [<ffffffff8105682e>] panic+0xe0/0x19a
 [<ffffffff8144006a>] ? init_amd+0x296/0x37a
 [<ffffffff8100f169>] ? xen_force_evtchn_callback+0xd/0xf
 [<ffffffff8100f8c2>] ? check_events+0x12/0x20
 [<ffffffff8100f8af>] ? xen_restore_fl_direct_end+0x0/0x1
 [<ffffffff81056487>] ? print_oops_end_marker+0x23/0x25
 [<ffffffff81448165>] oops_end+0xb6/0xc6
 [<ffffffff810166e5>] die+0x5a/0x63
 [<ffffffff81447a3c>] do_trap+0x115/0x124
 [<ffffffff810148e6>] do_invalid_op+0x9c/0xa5
 [<ffffffff8103a3cb>] ? leave_mm+0x15/0x46
 [<ffffffff8100f6e6>] ? xen_clocksource_read+0x21/0x23
 [<ffffffff8100f258>] ? HYPERVISOR_vcpu_op+0xf/0x11
 [<ffffffff8100f753>] ? xen_vcpuop_set_next_event+0x52/0x67
 [<ffffffff81013b3b>] invalid_op+0x1b/0x20
 [<ffffffff81447292>] ? _spin_unlock_irqrestore+0x15/0x17
 [<ffffffff8103a3cb>] ? leave_mm+0x15/0x46
 [<ffffffff8100e4a4>] drop_other_mm_ref+0x2a/0x53
 [<ffffffff81087224>] generic_smp_call_function_single_interrupt+0xd8/0xfc
 [<ffffffff810100e8>] xen_call_function_single_interrupt+0x13/0x28
 [<ffffffff810a936a>] handle_IRQ_event+0x66/0x120
 [<ffffffff810aac5b>] handle_percpu_irq+0x41/0x6e
 [<ffffffff8128c1a8>] __xen_evtchn_do_upcall+0x1ab/0x27d
 [<ffffffff8128dcf9>] xen_evtchn_do_upcall+0x33/0x46
 [<ffffffff81013efe>] xen_do_hypervisor_callback+0x1e/0x30
 <EOI>  [<ffffffff81447292>] ? _spin_unlock_irqrestore+0x15/0x17
 [<ffffffff8100f8af>] ? xen_restore_fl_direct_end+0x0/0x1
 [<ffffffff81113f75>] ? flush_old_exec+0x3ac/0x500
 [<ffffffff81150dc9>] ? load_elf_binary+0x0/0x17ef
 [<ffffffff81150dc9>] ? load_elf_binary+0x0/0x17ef
 [<ffffffff81151161>] ? load_elf_binary+0x398/0x17ef
 [<ffffffff81042fcf>] ? need_resched+0x23/0x
2d
 [<ffffffff811f463c>] ? process_measurement+0xc0/0xd7
 [<ffffffff81150dc9>] ? load_elf_binary+0x0/0x17ef
 [<ffffffff81113098>] ? search_binary_handler+0xc8/0x255
 [<ffffffff81114366>] ? do_execve+0x1c3/0x29e
 [<ffffffff8101155d>] ? sys_execve+0x43/0x5d
 [<ffffffff8106fc45>] ? __call_usermodehelper+0x0/0x6f
 [<ffffffff81013e28>] ? kernel_execve+0x68/0xd0
 [<ffffffff8106fc45>] ? __call_usermodehelper+0x0/0x6f
 [<ffffffff8100f8af>] ? xen_restore_fl_direct_end+0x0/0x1
 [<ffffffff8106fb64>] ? ____call_usermodehelper+0x113/0x11e
 [<ffffffff81013daa>] ? child_rip+0xa/0x20
 [<ffffffff8106fc45>] ? __call_usermodehelper+0x0/0x6f
 [<ffffffff81012f91>] ? int_ret_from_sys_call+0x7/0x1b
 [<ffffffff8101371d>] ? retint_restore_args+0x5/0x6
 [<ffffffff81013da0>] ? child_rip+0x0/0x20
(XEN) Domain 0 crashed: 'noreboot' set - not rebooting.

 
> Date: Tue, 12 Apr 2011 06:00:00 -0400
> From: konrad.wilk@oracle.com
> To: tinnycloud@hotmail.com
> CC: xen-devel@lists.xensource.com; giamteckchoon@gmail.com; jeremy@goop.org
> Subject: Re: Kernel BUG at arch/x86/mm/tlb.c:61
> 
> On Tue, Apr 12, 2011 at 05:11:51PM +0800, MaoXiaoyun wrote:
> > 
> > Hi :
> > 
> > We are using pvops kernel 2.6.32.36 + xen 4.0.1, but confront a kernel panic bug.
> > 
> > 2.6.32.36 Kernel: http://git.kernel.org/?p=linux/kernel/git/jeremy/xen.git;a=commit;h=bb1a15e55ec665a64c8a9c6bd699b1f16ac01ff4
> > Xen 4.0.1 http://xenbits.xen.org/hg/xen-4.0-testing.hg/rev/b536ebfba183 
> > 
> > Our test is simple, 24 HVMS(Win2003 ) on a single host, each HVM loopes in restart every 15minutes.
> 
> What is the storage that you are using for your guests? AoE? Local disks?
> 
> > About 17 machines are invovled in the test, after 10 hours run, one confrontted a crash at arch/x86/mm/tlb.c:61
> > 
> > Currently I am trying "cpuidle=0 cpufreq=none" tests based on Teck's suggestion.
> > 
> > Any comments, thanks. 
> > 
> > ===============crash log==========================
> > INIT: Id "s0" respawning too fast: disabled for 5 minutes
> > __ratelimit: 14 callbacks suppressed
> > blktap_sysfs_destroy
> > blktap_sysfs_destroy
> > ------------[ cut here ]------------
> > kernel BUG at arch/x86/mm/tlb.c:61!
> > invalid opcode: 0000 [#1] SMP 
> > last sysfs file: /sys/devices/system/xen_memory/xen_memory0/info/current_kb
> > CPU 1 
> > Modules linked in: 8021q garp xen_netback xen_blkback blktap blkback_pagemap nbd bridge stp llc autofs4 ipmi_devintf ipmi_si ipmi_msghandler lockd sunrpc bonding ipv6 xenfs dm_multipath video output sbs sbshc parport_pc lp parport ses enclosure snd_seq_dummy snd_seq_oss snd_seq_midi_event snd_seq snd_seq_device serio_raw bnx2 snd_pcm_oss snd_mixer_oss snd_pcm snd_timer iTCO_wdt snd soundcore snd_page_alloc i2c_i801 iTCO_vendor_support i2c_core pcspkr pata_acpi ata_generic ata_piix shpchp mptsas mptscsih mptbase [last unloaded: freq_table]
> > Pid: 25581, comm: khelper Not tainted 2.6.32.36fixxen #1 Tecal RH2285 
> > RIP: e030:[<ffffffff8103a3cb>] [<ffffffff8103a3cb>] leave_mm+0x15/0x46
> > RSP: e02b:ffff88002805be48 EFLAGS: 00010046
> > RAX: 0000000000000000 RBX: 0000000000000001 RCX: ffff88015f8e2da0
> > RDX: ffff88002805be78 RSI: 0000000000000000 RDI: 0000000000000001
> > RBP: ffff88002805be48 R08: ffff88009d662000 R09: dead000000200200
> > R10: dead000000100100 R11: ffffffff814472b2 R12: ffff88009bfc1880
> > R13: ffff880028063020 R14: 00000000000004f6 R15: 0000000000000000
> > FS: 00007f62362d66e0(0000) GS:ffff880028058000(0000) knlGS:0000000000000000
> > CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b
> > CR2: 0000003aabc11909 CR3: 000000009b8ca000 CR4: 0000000000002660
> > DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> > DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> > Process khelper (pid: 25581, threadinfo ffff88007691e000, task ffff88009b92db40)
> > Stack:
> > ffff88002805be68 ffffffff8100e4ae 0000000000000001 ffff88009d733b88
> > <0> ffff88002805be98 ffffffff81087224 ffff88002805be78 ffff88002805be78
> > <0> ffff88015f808360 00000000000004f6 ffff88002805bea8 ffffffff81010108
> > Call Trace:
> > <IRQ> 
> > [<ffffffff8100e4ae>] drop_other_mm_ref+0x2a/0x53
> > [<ffffffff81087224>] generic_smp_call_function_single_interrupt+0xd8/0xfc
> > [<ffffffff81010108>] xen_call_function_single_interrupt+0x13/0x28
> > [<ffffffff810a936a>] handle_IRQ_event+0x66/0x120
> > [<ffffffff810aac5b>] handle_percpu_irq+0x41/0x6e
> > [<ffffffff8128c1c0>] __xen_evtchn_do_upcall+0x1ab/0x27d
> > [<ffffffff8128dd11>] xen_evtchn_do_upcall+0x33/0x46
> > [<ffffffff81013efe>] xen_do_hypervisor_callback+0x1e/0x30
> > <EOI> 
> > [<ffffffff814472b2>] ? _spin_unlock_irqrestore+0x15/0x17
> > [<ffffffff8100f8cf>] ? xen_restore_fl_direct_end+0x0/0x1
> > [<ffffffff81113f71>] ? flush_old_exec+0x3ac/0x500
> > [<ffffffff81150dc5>] ? load_elf_binary+0x0/0x17ef
> > [<ffffffff81150dc5>] ? load_elf_binary+0x0/0x17ef
> > [<ffffffff8115115d>] ? load_elf_binary+0x398/0x17ef
> > [<ffffffff81042fcf>] ? need_resched+0x23/0x2d
> > [<ffffffff811f4648>] ? process_measurement+0xc0/0xd7
> > [<ffffffff81150dc5>] ? load_elf_binary+0x0/0x17ef
> > [<ffffffff81113094>] ? search_binary_handler+0xc8/0x255
> > [<ffffffff81114362>] ? do_execve+0x1c3/0x29e
> > [<ffffffff8101155d>] ? sys_execve+0x43/0x5d
> > [<ffffffff8106fc45>] ? __call_usermodehelper+0x0/0x6f
> > [<ffffffff81013e28>] ? kernel_execve+0x68/0xd0
> > [<ffffffff8106fc45>] ? __call_usermodehelper+0x0/0x6f
> > [<ffffffff8100f8cf>] ? xen_restore_fl_direct_end+0x0/0x1
> > [<ffffffff8106fb64>] ? ____call_usermodehelper+0x113/0x11e
> > [<ffffffff81013daa>] ? child_rip+0xa/0x20
> > [<ffffffff8106fc45>] ? __call_usermodehelper+0x0/0x6f
> > [<ffffffff81012f91>] ? int_ret_from_sys_call+0x7/0x1b
> > [<ffffffff8101371d>] ? retint_restore_args+0x5/0x6
> > [<ffffffff81013da0>] ? child_rip+0x0/0x20
> > Code: 41 5e 41 5f c9 c3 55 48 89 e5 0f 1f 44 00 00 e8 17 ff ff ff c9 c3 55 48 89 e5 0f 1f 44 00 00 65 8b 04 25 c8 55 01 00 ff c8 75 04 <0f> 0b eb fe 65 48 8b 34 25 c0 55 01 00 48 81 c6 b8 02 00 00 e8 
> > RIP [<ffffffff8103a3cb>] leave_mm+0x15/0x46
> > RSP <ffff88002805be48>
> > ---[ end trace ce9cee6832a9c503 ]---
> > Kernel panic - not syncing: Fatal exception in interrupt
> > Pid: 25581, comm: khelper Tainted: G D 2.6.32.36fixxen #1
> > Call Trace:
> > <IRQ> [<ffffffff8105682e>] panic+0xe0/0x19a
> > [<ffffffff8144008a>] ? init_amd+0x296/0x37a
> > [<ffffffff8100f17d>] ? xen_force_evtchn_callback+0xd/0xf
> > [<ffffffff8100f8e2>] ? check_events+0x12/0x20
> > [<ffffffff8100f8cf>] ? xen_restore_fl_direct_end+0x0/0x1
> > [<ffffffff81056487>] ? print_oops_end_marker+0x23/0x25
> > [<ffffffff81448185>] oops_end+0xb6/0xc6
> > [<ffffffff810166e5>] die+0x5a/0x63
> > [<ffffffff81447a5c>] do_trap+0x115/0x124
> > [<ffffffff810148e6>] do_invalid_op+0x9c/0xa5
> > [<ffffffff8103a3cb>] ? leave_mm+0x15/0x46
> > [<ffffffff8100f6fa>] ? xen_clocksource_read+0x21/0x23
> > [<ffffffff8100f26c>] ? HYPERVISOR_vcpu_op+0xf/0x11
> > [<ffffffff8100f767>] ? xen_vcpuop_set_next_event+0x52/0x67
> > [<ffffffff81080bfa>] ? clockevents_program_event+0x78/0x81
> > [<ffffffff81013b3b>] invalid_op+0x1b/0x20
> > [<ffffffff814472b2>] ? _spin_unlock_irqrestore+0x15/0x17
> > [<ffffffff8103a3cb>] ? leave_mm+0x15/0x46
> > [<ffffffff8100e4ae>] drop_other_mm_ref+0x2a/0x53
> > [<ffffffff81087224>] generic_smp_call_function_single_interrupt+0xd8/0xfc
> > [<ffffffff81010108>] xen_call_function_single_interrupt+0x13/0x28
> > [<ffffffff810a936a>] handle_IRQ_event+0x66/0x120
> > [<ffffffff810aac5b>] handle_percpu_irq+0x41/0x6e
> > [<ffffffff8128c1c0>] __xen_evtchn_do_upcall+0x1ab/0x27d
> > [<ffffffff8128dd11>] xen_evtchn_do_upcall+0x33/0x46
> > [<ffffffff81013efe>] xen_do_hypervisor_callback+0x1e/0x30
> > <EOI> [<ffffffff814472b2>] ? _spin_unlock_irqrestore+0x15/0x17
> > [<ffffffff8100f8cf>] ? xen_restore_fl_direct_end+0x0/0x1
> > [<ffffffff81113f71>] ? flush_old_exec+0x3ac/0x500
> > [<ffffffff81150dc5>] ? load_elf_binary+0x0/0x17ef
> > [<ffffffff81150dc5>] ? load_elf_binary+0x0/0x17ef
> > [<ffffffff8115115d>] ? load_elf_binary+0x398/0x17ef
> > [<ffffffff81042fcf>] ? need_resched+0x23/0x2d
> > [<ffffffff811f4648>] ? process_measurement+0xc0/0xd7
> > [<ffffffff81150dc5>] ? load_elf_binary+0x0/0x17ef
> > [<ffffffff81113094>] ? search_binary_handler+0xc8/0x255
> > [<ffffffff81114362>] ? do_execve+0x1c3/0x29e
> > [<ffffffff8101155d>] ? sys_execve+0x43/0x5d
> > [<ffffffff8106fc45>] ? __call_usermodehelper+0x0/0x6f
> > [<ffffffff81013e28>] ? kernel_execve+0x68/0xd0
> > [<ffffffff8106fc45>] ? __call_usermodehelper+0x0/0x6f
> > [<ffffffff8100f8cf>] ? xen_restore_fl_direct_end+0x0/0x1
> > [<ffffffff8106fb64>] ? ____call_usermodehelper+0x113/0x11e
> > [<ffffffff81013daa>] ? child_rip+0xa/0x20
> > [<ffffffff8106fc45>] ? __call_usermodehelper+0x0/0x6f
> > [<ffffffff81012f91>] ? int_ret_from_sys_call+0x7/0x1b
> > [<ffffffff8101371d>] ? retint_restore_args+0x5/0x6
> > [<ffffffff81013da0>] ? child_rip+0x0/0x20
> > 
> > 
 		 	   		  

[-- Attachment #1.2: Type: text/html, Size: 20788 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Kernel BUG at arch/x86/mm/tlb.c:61

[Teck Choon Giam]

2011/4/14 MaoXiaoyun <tinnycloud@hotmail.com>:
> Hi:
>
>       I've done test with "cpuidle=0 cpufreq=none", two machine crashed.
>
> blktap_sysfs_destroy
> blktap_sysfs_destroy
> blktap_sysfs_create: adding attributes for dev ffff8800ad581000
> blktap_sysfs_create: adding attributes for dev ffff8800a48e3e00
> ------------[ cut here ]------------
> kernel BUG at arch/x86/mm/tlb.c:61!
> invalid opcode: 0000 [#1] SMP
> last sysfs file: /sys/block/tapdeve/dev
> CPU 0
> Modules linked in: 8021q garp blktap xen_netback xen_blkback blkback_pagemap nbd bridge stp llc autofs4 ipmi_devintf ipmi_si ipmi_ms
> ghandler lockd sunrpc bonding ipv6 xenfs dm_multipath video output sbs sbshc parport_pc lp parport ses enclosure snd_seq_dummy bnx2
> serio_raw snd_seq_oss snd_seq_midi_event snd_seq snd_seq_device snd_pcm_oss snd_mixer_oss snd_pcm i2c_i801 snd_timer i2c_core snd iT
> CO_wdt pata_acpi soundcore iTCO_vendor_
> support ata_generic snd_page_alloc pcspkr ata_piix shpchp mptsas mptscsih mptbase [last unloa
> ded: freq_table]
> Pid: 8022, comm: khelper Not tainted 2.6.32.36xen #1 Tecal RH2285
> RIP: e030:[<ffffffff8103a3cb>]  [<ffffffff8103a3cb>] leave_mm+0x15/0x46
> RSP: e02b:ffff88002803ee48  EFLAGS: 00010046
> RAX: 0000000000000000 RBX: 0000000000000001 RCX: ffffffff81675980
> RDX: ffff88002803ee78 RSI: 0000000000000000 RDI: 0000000000000000
> RBP: ffff88002803ee48 R08: ffff8800a4929000 R09: dead000000200200
> R10: dead000000100100 R11: ffffffff81447292 R12: ffff88012ba07b80
> R13: ffff880028046020 R14: 00000000000004fb R15: 0000000000000000
> FS:  00007f410af416e0(0000) GS:ffff88002803b000(0000) knlGS:0000000000000000
> CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
> CR2: 0000000000469000 CR3: 00000000ad639000 CR4: 0000000000002660
> DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> Process khelper (pid: 8022, threadinfo ffff8800a4846000, task ffff8800a9ed0000)
> Stack:
>  ffff88002803ee68 ffffffff8100e4a4 0000000000000001 ffff880097de3b88
> <0> ffff88002803ee98 ffffffff81087224 ffff88002803ee78 ffff88002803ee78
> <0> ffff88015f808180 00000000000004fb ffff88002803eea8 ffffffff810100e8
> Call Trace:
>  <IRQ>
>  [<ffffffff8100e4a4>] drop_other_mm_ref+0x2a/0x53
>  [<ffffffff81087224>] generic_smp_call_function_single_interrupt+0xd8/0xfc
>  [<ffffffff810100e8>] xen_call_function_single_interrupt+0x13/0x28
>  [<ffffffff810a936a>] handle_IRQ_event+0x66/0x120
>  [<ffffffff810aac5b>] handle_percpu_irq+0x41/0x6e
>  [<ffffffff8128c1a8>] __xen_evtchn_do_upcall+0x1ab/0x27d
>  [<ffffffff8128dcf9>] xen_evtchn_do_upcall+0x33/0x46
>  [<ffffffff81013efe>] xen_do_hypervisor_callback+0x1e/0x30
>  <EOI>
>  [<ffffffff81447292>] ? _spin_unlock_irqrestore+0x15/0x17
>  [<ffffffff8100f8af>] ? xen_restore_fl_direct_end+0x0/0x1
>  [<ffffffff81113f75>] ? flush_old_exec+0x3ac/0x500
>  [<ffffffff81150dc9>] ? load_elf_binary+0x0/0x17ef
>  [<ffffffff81150dc9>] ? load_elf_binary+0x0/0x17ef
>  [<ffffffff81151161>] ? load_elf_binary+0x398/0x17ef
>  [<ffffffff81042fcf>] ? need_resched+0x23/0x2d
>
> [<ffffffff811f463c>] ? process_measurement+0xc0/0xd7
>  [<ffffffff81150dc9>] ? load_elf_binary+0x0/0x17ef
>  [<ffffffff81113098>] ? search_binary_handler+0xc8/0x255
>  [<ffffffff81114366>] ? do_execve+0x1c3/0x29e
>  [<ffffffff8101155d>] ? sys_execve+0x43/0x5d
>  [<ffffffff8106fc45>] ? __call_usermodehelper+0x0/0x6f
>  [<ffffffff81013e28>] ? kernel_execve+0x68/0xd0
>  [<ffffffff8106fc45>] ? __call_usermodehelper+0x0/0x6f
>  [<ffffffff8100f8af>] ? xen_restore_fl_direct_end+0x0/0x1
>  [<ffffffff8106fb64>] ? ____call_usermodehelper+0x113/0x11e
>  [<ffffffff81013daa>] ? child_rip+0xa/0x20
>  [<ffffffff8106fc45>] ? __call_usermodehelper+0x0/0x6f
>  [<ffffffff81012f91>] ? int_ret_from_sys_call+0x7/0x1b
>  [<ffffffff8101371d>] ? retint_restore_args+0x5/0x6
>  [<ffffffff81013da0>] ? c
> hild_rip+0x0/0x20
> Code: 41 5e 41 5f c9 c3 55 48 89 e5 0f 1f 44 00 00 e8 17 ff ff ff c9 c3 55 48 89 e5 0f 1f 44 00 00 65 8b 04 25 c8 55 01 00 ff c8 75 04 <0f> 0b eb fe 65 48 8b 34 25 c0 55 01 00 48 81 c6 b8 02 00 00 e8
> RIP  [<ffffffff8103a3cb>] leave_mm+0x15/0x46
>  RSP <ffff88002803ee48>
> ---[ end trace 1522f17fdfc9162d ]---
> Kernel panic - not syncing: Fatal exception in interrupt
> Pid: 8022, comm: khelper Tainted: G      D    2.6.32.36xen #1
> Call Trace:
>  <IRQ>  [<ffffffff8105682e>] panic+0xe0/0x19a
>  [<ffffffff8144006a>] ? init_amd+0x296/0x37a

Hmmm... both machines are using AMD CPU?  Did you hit the same bug on Intel CPU?


>  [<ffffffff8100f169>] ? xen_force_evtchn_callback+0xd/0xf
>  [<ffffffff8100f8c2>] ? check_events+0x12/0x20
>  [<ffffffff8100f8af>] ? xen_restore_fl_direct_end+0x0/0x1
>  [<ffffffff81056487>] ? print_oops_end_marker+0x23/0x25
>  [<ffffffff81448165>] oops_end+0xb6/0xc6
>  [<ffffffff810166e5>] die+0x5a/0x63
>  [<ffffffff81447a3c>] do_trap+0x115/0x124
>  [<ffffffff810148e6>] do_invalid_op+0x9c/0xa5
>  [<ffffffff8103a3cb>] ? leave_mm+0x15/0x46
>  [<ffffffff8100f6e6>] ? xen_clocksource_read+0x21/0x23
>  [<ffffffff8100f258>] ? HYPERVISOR_vcpu_op+0xf/0x11
>  [<ffffffff8100f753>] ? xen_vcpuop_set_next_event+0x52/0x67
>  [<ffffffff81013b3b>] invalid_op+0x1b/0x20
>  [<ffffffff81447292>] ? _spin_unlock_irqrestore+0x15/0x17
>  [<ffffffff8103a3cb>] ? leave_mm+0x15/0x46
>  [<ffffffff8100e4a4>] drop_other_mm_ref+0x2a/0x53
>  [<ffffffff81087224>] generic_smp_call_function_single_interrupt+0xd8/0xfc
>  [<ffffffff810100e8>] xen_call_function_single_interrupt+0x13/0x28
>  [<ffffffff810a936a>] handle_IRQ_event+0x66/0x120
>  [<ffffffff810aac5b>] handle_percpu_irq+0x41/0x6e
>  [<ffffffff8128c1a8>] __xen_evtchn_do_upcall+0x1ab/0x27d
>  [<ffffffff8128dcf9>] xen_evtchn_do_upcall+0x33/0x46
>  [<ffffffff81013efe>] xen_do_hypervisor_callback+0x1e/0x30
>  <EOI>  [<ffffffff81447292>] ? _spin_unlock_irqrestore+0x15/0x17
>  [<ffffffff8100f8af>] ? xen_restore_fl_direct_end+0x0/0x1
>  [<ffffffff81113f75>] ? flush_old_exec+0x3ac/0x500
>  [<ffffffff81150dc9>] ? load_elf_binary+0x0/0x17ef
>  [<ffffffff81150dc9>] ? load_elf_binary+0x0/0x17ef
>  [<ffffffff81151161>] ? load_elf_binary+0x398/0x17ef
>  [<ffffffff81042fcf>] ? need_resched+0x23/0x
> 2d
>  [<ffffffff811f463c>] ? process_measurement+0xc0/0xd7
>  [<ffffffff81150dc9>] ? load_elf_binary+0x0/0x17ef
>  [<ffffffff81113098>] ? search_binary_handler+0xc8/0x255
>  [<ffffffff81114366>] ? do_execve+0x1c3/0x29e
>  [<ffffffff8101155d>] ? sys_execve+0x43/0x5d
>  [<ffffffff8106fc45>] ? __call_usermodehelper+0x0/0x6f
>  [<ffffffff81013e28>] ? kernel_execve+0x68/0xd0
>  [<ffffffff8106fc45>] ? __call_usermodehelper+0x0/0x6f
>  [<ffffffff8100f8af>] ? xen_restore_fl_direct_end+0x0/0x1
>  [<ffffffff8106fb64>] ? ____call_usermodehelper+0x113/0x11e
>  [<ffffffff81013daa>] ? child_rip+0xa/0x20
>  [<ffffffff8106fc45>] ? __call_usermodehelper+0x0/0x6f
>  [<ffffffff81012f91>] ? int_ret_from_sys_call+0x7/0x1b
>  [<ffffffff8101371d>] ? retint_restore_args+0x5/0x6
>  [<ffffffff81013da0>] ? child_rip+0x0/0x20
> (XEN) Domain 0 crashed: 'noreboot' set - not rebooting.
>
>> Date: Tue, 12 Apr 2011 06:00:00 -0400
>> From: konrad.wilk@oracle.com
>> To: tinnycloud@hotmail.com
>> CC: xen-devel@lists.xensource.com; giamteckchoon@gmail.com;
>> jeremy@goop.org
>> Subject: Re: Kernel BUG at arch/x86/mm/tlb.c:61
>>
>> On Tue, Apr 12, 2011 at 05:11:51PM +0800, MaoXiaoyun wrote:
>> >
>> > Hi :
>> >
>> > We are using pvops kernel 2.6.32.36 + xen 4.0.1, but confront a kernel
>> > panic bug.
>> >
>> > 2.6.32.36 Kernel:
>> > http://git.kernel.org/?p=linux/kernel/git/jeremy/xen.git;a=commit;h=bb1a15e55ec665a64c8a9c6bd699b1f16ac01ff4
>> > Xen 4.0.1 http://xenbits.xen.org/hg/xen-4.0-testing.hg/rev/b536ebfba183
>> >
>> > Our test is simple, 24 HVMS(Win2003 ) on a single host, each HVM loopes
>> > in restart every 15minutes.
>>
>> What is the storage that you are using for your guests? AoE? Local disks?
>>
>> > About 17 machines are invovled in the test, after 10 hours run, one
>> > confrontted a crash at arch/x86/mm/tlb.c:61
>> >
>> > Currently I am trying "cpuidle=0 cpufreq=none" tests based on Teck's
>> > suggestion.
>> >
>> > Any comments, thanks.
>> >
>> > ===============crash log==========================
>> > INIT: Id "s0" respawning too fast: disabled for 5 minutes
>> > __ratelimit: 14 callbacks suppressed
>> > blktap_sysfs_destroy
>> > blktap_sysfs_destroy
>> > ------------[ cut here ]------------
>> > kernel BUG at arch/x86/mm/tlb.c:61!
>> > invalid opcode: 0000 [#1] SMP
>> > last sysfs file:
>> > /sys/devices/system/xen_memory/xen_memory0/info/current_kb
>> > CPU 1
>> > Modules linked in: 8021q garp xen_netback xen_blkback blktap
>> > blkback_pagemap nbd bridge stp llc autofs4 ipmi_devintf ipmi_si
>> > ipmi_msghandler lockd sunrpc bonding ipv6 xenfs dm_multipath video output
>> > sbs sbshc parport_pc lp parport ses enclosure snd_seq_dummy snd_seq_oss
>> > snd_seq_midi_event snd_seq snd_seq_device serio_raw bnx2 snd_pcm_oss
>> > snd_mixer_oss snd_pcm snd_timer iTCO_wdt snd soundcore snd_page_alloc
>> > i2c_i801 iTCO_vendor_support i2c_core pcspkr pata_acpi ata_generic ata_piix
>> > shpchp mptsas mptscsih mptbase [last unloaded: freq_table]
>> > Pid: 25581, comm: khelper Not tainted 2.6.32.36fixxen #1 Tecal RH2285
>> > RIP: e030:[<ffffffff8103a3cb>] [<ffffffff8103a3cb>] leave_mm+0x15/0x46
>> > RSP: e02b:ffff88002805be48 EFLAGS: 00010046
>> > RAX: 0000000000000000 RBX: 0000000000000001 RCX: ffff88015f8e2da0
>> > RDX: ffff88002805be78 RSI: 0000000000000000 RDI: 0000000000000001
>> > RBP: ffff88002805be48 R08: ffff88009d662000 R09: dead000000200200
>> > R10: dead000000100100 R11: ffffffff814472b2 R12: ffff88009bfc1880
>> > R13: ffff880028063020 R14: 00000000000004f6 R15: 0000000000000000
>> > FS: 00007f62362d66e0(0000) GS:ffff880028058000(0000)
>> > knlGS:0000000000000000
>> > CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b
>> > CR2: 0000003aabc11909 CR3: 000000009b8ca000 CR4: 0000000000002660
>> > DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
>> > DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
>> > Process khelper (pid: 25581, threadinfo ffff88007691e000, task
>> > ffff88009b92db40)
>> > Stack:
>> > ffff88002805be68 ffffffff8100e4ae 0000000000000001 ffff88009d733b88
>> > <0> ffff88002805be98 ffffffff81087224 ffff88002805be78 ffff88002805be78
>> > <0> ffff88015f808360 00000000000004f6 ffff88002805bea8 ffffffff81010108
>> > Call Trace:
>> > <IRQ>
>> > [<ffffffff8100e4ae>] drop_other_mm_ref+0x2a/0x53
>> > [<ffffffff81087224>]
>> > generic_smp_call_function_single_interrupt+0xd8/0xfc
>> > [<ffffffff81010108>] xen_call_function_single_interrupt+0x13/0x28
>> > [<ffffffff810a936a>] handle_IRQ_event+0x66/0x120
>> > [<ffffffff810aac5b>] handle_percpu_irq+0x41/0x6e
>> > [<ffffffff8128c1c0>] __xen_evtchn_do_upcall+0x1ab/0x27d
>> > [<ffffffff8128dd11>] xen_evtchn_do_upcall+0x33/0x46
>> > [<ffffffff81013efe>] xen_do_hypervisor_callback+0x1e/0x30
>> > <EOI>
>> > [<ffffffff814472b2>] ? _spin_unlock_irqrestore+0x15/0x17
>> > [<ffffffff8100f8cf>] ? xen_restore_fl_direct_end+0x0/0x1
>> > [<ffffffff81113f71>] ? flush_old_exec+0x3ac/0x500
>> > [<ffffffff81150dc5>] ? load_elf_binary+0x0/0x17ef
>> > [<ffffffff81150dc5>] ? load_elf_binary+0x0/0x17ef
>> > [<ffffffff8115115d>] ? load_elf_binary+0x398/0x17ef
>> > [<ffffffff81042fcf>] ? need_resched+0x23/0x2d
>> > [<ffffffff811f4648>] ? process_measurement+0xc0/0xd7
>> > [<ffffffff81150dc5>] ? load_elf_binary+0x0/0x17ef
>> > [<ffffffff81113094>] ? search_binary_handler+0xc8/0x255
>> > [<ffffffff81114362>] ? do_execve+0x1c3/0x29e
>> > [<ffffffff8101155d>] ? sys_execve+0x43/0x5d
>> > [<ffffffff8106fc45>] ? __call_usermodehelper+0x0/0x6f
>> > [<ffffffff81013e28>] ? kernel_execve+0x68/0xd0
>> > [<ffffffff8106fc45>] ? __call_usermodehelper+0x0/0x6f
>> > [<ffffffff8100f8cf>] ? xen_restore_fl_direct_end+0x0/0x1
>> > [<ffffffff8106fb64>] ? ____call_usermodehelper+0x113/0x11e
>> > [<ffffffff81013daa>] ? child_rip+0xa/0x20
>> > [<ffffffff8106fc45>] ? __call_usermodehelper+0x0/0x6f
>> > [<ffffffff81012f91>] ? int_ret_from_sys_call+0x7/0x1b
>> > [<ffffffff8101371d>] ? retint_restore_args+0x5/0x6
>> > [<ffffffff81013da0>] ? child_rip+0x0/0x20
>> > Code: 41 5e 41 5f c9 c3 55 48 89 e5 0f 1f 44 00 00 e8 17 ff ff ff c9 c3
>> > 55 48 89 e5 0f 1f 44 00 00 65 8b 04 25 c8 55 01 00 ff c8 75 04 <0f> 0b eb fe
>> > 65 48 8b 34 25 c0 55 01 00 48 81 c6 b8 02 00 00 e8
>> > RIP [<ffffffff8103a3cb>] leave_mm+0x15/0x46
>> > RSP <ffff88002805be48>
>> > ---[ end trace ce9cee6832a9c503 ]---
>> > Kernel panic - not syncing: Fatal exception in interrupt
>> > Pid: 25581, comm: khelper Tainted: G D 2.6.32.36fixxen #1
>> > Call Trace:
>> > <IRQ> [<ffffffff8105682e>] panic+0xe0/0x19a
>> > [<ffffffff8144008a>] ? init_amd+0x296/0x37a
>> > [<ffffffff8100f17d>] ? xen_force_evtchn_callback+0xd/0xf
>> > [<ffffffff8100f8e2>] ? check_events+0x12/0x20
>> > [<ffffffff8100f8cf>] ? xen_restore_fl_direct_end+0x0/0x1
>> > [<ffffffff81056487>] ? print_oops_end_marker+0x23/0x25
>> > [<ffffffff81448185>] oops_end+0xb6/0xc6
>> > [<ffffffff810166e5>] die+0x5a/0x63
>> > [<ffffffff81447a5c>] do_trap+0x115/0x124
>> > [<ffffffff810148e6>] do_invalid_op+0x9c/0xa5
>> > [<ffffffff8103a3cb>] ? leave_mm+0x15/0x46
>> > [<ffffffff8100f6fa>] ? xen_clocksource_read+0x21/0x23
>> > [<ffffffff8100f26c>] ? HYPERVISOR_vcpu_op+0xf/0x11
>> > [<ffffffff8100f767>] ? xen_vcpuop_set_next_event+0x52/0x67
>> > [<ffffffff81080bfa>] ? clockevents_program_event+0x78/0x81
>> > [<ffffffff81013b3b>] invalid_op+0x1b/0x20
>> > [<ffffffff814472b2>] ? _spin_unlock_irqrestore+0x15/0x17
>> > [<ffffffff8103a3cb>] ? leave_mm+0x15/0x46
>> > [<ffffffff8100e4ae>] drop_other_mm_ref+0x2a/0x53
>> > [<ffffffff81087224>]
>> > generic_smp_call_function_single_interrupt+0xd8/0xfc
>> > [<ffffffff81010108>] xen_call_function_single_interrupt+0x13/0x28
>> > [<ffffffff810a936a>] handle_IRQ_event+0x66/0x120
>> > [<ffffffff810aac5b>] handle_percpu_irq+0x41/0x6e
>> > [<ffffffff8128c1c0>] __xen_evtchn_do_upcall+0x1ab/0x27d
>> > [<ffffffff8128dd11>] xen_evtchn_do_upcall+0x33/0x46
>> > [<ffffffff81013efe>] xen_do_hypervisor_callback+0x1e/0x30
>> > <EOI> [<ffffffff814472b2>] ? _spin_unlock_irqrestore+0x15/0x17
>> > [<ffffffff8100f8cf>] ? xen_restore_fl_direct_end+0x0/0x1
>> > [<ffffffff81113f71>] ? flush_old_exec+0x3ac/0x500
>> > [<ffffffff81150dc5>] ? load_elf_binary+0x0/0x17ef
>> > [<ffffffff81150dc5>] ? load_elf_binary+0x0/0x17ef
>> > [<ffffffff8115115d>] ? load_elf_binary+0x398/0x17ef
>> > [<ffffffff81042fcf>] ? need_resched+0x23/0x2d
>> > [<ffffffff811f4648>] ? process_measurement+0xc0/0xd7
>> > [<ffffffff81150dc5>] ? load_elf_binary+0x0/0x17ef
>> > [<ffffffff81113094>] ? search_binary_handler+0xc8/0x255
>> > [<ffffffff81114362>] ? do_execve+0x1c3/0x29e
>> > [<ffffffff8101155d>] ? sys_execve+0x43/0x5d
>> > [<ffffffff8106fc45>] ? __call_usermodehelper+0x0/0x6f
>> > [<ffffffff81013e28>] ? kernel_execve+0x68/0xd0
>> > [<ffffffff8106fc45>] ? __call_usermodehelper+0x0/0x6f
>> > [<ffffffff8100f8cf>] ? xen_restore_fl_direct_end+0x0/0x1
>> > [<ffffffff8106fb64>] ? ____call_usermodehelper+0x113/0x11e
>> > [<ffffffff81013daa>] ? child_rip+0xa/0x20
>> > [<ffffffff8106fc45>] ? __call_usermodehelper+0x0/0x6f
>> > [<ffffffff81012f91>] ? int_ret_from_sys_call+0x7/0x1b
>> > [<ffffffff8101371d>] ? retint_restore_args+0x5/0x6
>> > [<ffffffff81013da0>] ? child_rip+0x0/0x20
>> >
>> >
>

RE: Kernel BUG at arch/x86/mm/tlb.c:61

[MaoXiaoyun]

[-- Attachment #1.1: Type: text/plain, Size: 8988 bytes --]



 

> Date: Thu, 14 Apr 2011 15:26:14 +0800
> Subject: Re: Kernel BUG at arch/x86/mm/tlb.c:61
> From: giamteckchoon@gmail.com
> To: tinnycloud@hotmail.com
> CC: xen-devel@lists.xensource.com; jeremy@goop.org; konrad.wilk@oracle.com
> 
> 2011/4/14 MaoXiaoyun <tinnycloud@hotmail.com>:
> > Hi:
> >
> >       I've done test with "cpuidle=0 cpufreq=none", two machine crashed.
> >
> > blktap_sysfs_destroy
> > blktap_sysfs_destroy
> > blktap_sysfs_create: adding attributes for dev ffff8800ad581000
> > blktap_sysfs_create: adding attributes for dev ffff8800a48e3e00
> > ------------[ cut here ]------------
> > kernel BUG at arch/x86/mm/tlb.c:61!
> > invalid opcode: 0000 [#1] SMP
> > last sysfs file: /sys/block/tapdeve/dev
> > CPU 0
> > Modules linked in: 8021q garp blktap xen_netback xen_blkback blkback_pagemap nbd bridge stp llc autofs4 ipmi_devintf ipmi_si ipmi_ms
> > ghandler lockd sunrpc bonding ipv6 xenfs dm_multipath video output sbs sbshc parport_pc lp parport ses enclosure snd_seq_dummy bnx2
> > serio_raw snd_seq_oss snd_seq_midi_event snd_seq snd_seq_device snd_pcm_oss snd_mixer_oss snd_pcm i2c_i801 snd_timer i2c_core snd iT
> > CO_wdt pata_acpi soundcore iTCO_vendor_
> > support ata_generic snd_page_alloc pcspkr ata_piix shpchp mptsas mptscsih mptbase [last unloa
> > ded: freq_table]
> > Pid: 8022, comm: khelper Not tainted 2.6.32.36xen #1 Tecal RH2285
> > RIP: e030:[<ffffffff8103a3cb>]  [<ffffffff8103a3cb>] leave_mm+0x15/0x46
> > RSP: e02b:ffff88002803ee48  EFLAGS: 00010046
> > RAX: 0000000000000000 RBX: 0000000000000001 RCX: ffffffff81675980
> > RDX: ffff88002803ee78 RSI: 0000000000000000 RDI: 0000000000000000
> > RBP: ffff88002803ee48 R08: ffff8800a4929000 R09: dead000000200200
> > R10: dead000000100100 R11: ffffffff81447292 R12: ffff88012ba07b80
> > R13: ffff880028046020 R14: 00000000000004fb R15: 0000000000000000
> > FS:  00007f410af416e0(0000) GS:ffff88002803b000(0000) knlGS:0000000000000000
> > CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
> > CR2: 0000000000469000 CR3: 00000000ad639000 CR4: 0000000000002660
> > DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> > DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> > Process khelper (pid: 8022, threadinfo ffff8800a4846000, task ffff8800a9ed0000)
> > Stack:
> >  ffff88002803ee68 ffffffff8100e4a4 0000000000000001 ffff880097de3b88
> > <0> ffff88002803ee98 ffffffff81087224 ffff88002803ee78 ffff88002803ee78
> > <0> ffff88015f808180 00000000000004fb ffff88002803eea8 ffffffff810100e8
> > Call Trace:
> >  <IRQ>
> >  [<ffffffff8100e4a4>] drop_other_mm_ref+0x2a/0x53
> >  [<ffffffff81087224>] generic_smp_call_function_single_interrupt+0xd8/0xfc
> >  [<ffffffff810100e8>] xen_call_function_single_interrupt+0x13/0x28
> >  [<ffffffff810a936a>] handle_IRQ_event+0x66/0x120
> >  [<ffffffff810aac5b>] handle_percpu_irq+0x41/0x6e
> >  [<ffffffff8128c1a8>] __xen_evtchn_do_upcall+0x1ab/0x27d
> >  [<ffffffff8128dcf9>] xen_evtchn_do_upcall+0x33/0x46
> >  [<ffffffff81013efe>] xen_do_hypervisor_callback+0x1e/0x30
> >  <EOI>
> >  [<ffffffff81447292>] ? _spin_unlock_irqrestore+0x15/0x17
> >  [<ffffffff8100f8af>] ? xen_restore_fl_direct_end+0x0/0x1
> >  [<ffffffff81113f75>] ? flush_old_exec+0x3ac/0x500
> >  [<ffffffff81150dc9>] ? load_elf_binary+0x0/0x17ef
> >  [<ffffffff81150dc9>] ? load_elf_binary+0x0/0x17ef
> >  [<ffffffff81151161>] ? load_elf_binary+0x398/0x17ef
> >  [<ffffffff81042fcf>] ? need_resched+0x23/0x2d
> >
> > [<ffffffff811f463c>] ? process_measurement+0xc0/0xd7
> >  [<ffffffff81150dc9>] ? load_elf_binary+0x0/0x17ef
> >  [<ffffffff81113098>] ? search_binary_handler+0xc8/0x255
> >  [<ffffffff81114366>] ? do_execve+0x1c3/0x29e
> >  [<ffffffff8101155d>] ? sys_execve+0x43/0x5d
> >  [<ffffffff8106fc45>] ? __call_usermodehelper+0x0/0x6f
> >  [<ffffffff81013e28>] ? kernel_execve+0x68/0xd0
> >  [<ffffffff8106fc45>] ? __call_usermodehelper+0x0/0x6f
> >  [<ffffffff8100f8af>] ? xen_restore_fl_direct_end+0x0/0x1
> >  [<ffffffff8106fb64>] ? ____call_usermodehelper+0x113/0x11e
> >  [<ffffffff81013daa>] ? child_rip+0xa/0x20
> >  [<ffffffff8106fc45>] ? __call_usermodehelper+0x0/0x6f
> >  [<ffffffff81012f91>] ? int_ret_from_sys_call+0x7/0x1b
> >  [<ffffffff8101371d>] ? retint_restore_args+0x5/0x6
> >  [<ffffffff81013da0>] ? c
> > hild_rip+0x0/0x20
> > Code: 41 5e 41 5f c9 c3 55 48 89 e5 0f 1f 44 00 00 e8 17 ff ff ff c9 c3 55 48 89 e5 0f 1f 44 00 00 65 8b 04 25 c8 55 01 00 ff c8 75 04 <0f> 0b eb fe 65 48 8b 34 25 c0 55 01 00 48 81 c6 b8 02 00 00 e8
> > RIP  [<ffffffff8103a3cb>] leave_mm+0x15/0x46
> >  RSP <ffff88002803ee48>
> > ---[ end trace 1522f17fdfc9162d ]---
> > Kernel panic - not syncing: Fatal exception in interrupt
> > Pid: 8022, comm: khelper Tainted: G      D    2.6.32.36xen #1
> > Call Trace:
> >  <IRQ>  [<ffffffff8105682e>] panic+0xe0/0x19a
> >  [<ffffffff8144006a>] ? init_amd+0x296/0x37a
> 
> Hmmm... both machines are using AMD CPU? Did you hit the same bug on Intel CPU?
> 
> 
 
It is Intel CPU, not AMD. 
 
model name      : Intel(R) Xeon(R) CPU           E5620  @ 2.40GHz
 

> >  [<ffffffff8100f169>] ? xen_force_evtchn_callback+0xd/0xf
> >  [<ffffffff8100f8c2>] ? check_events+0x12/0x20
> >  [<ffffffff8100f8af>] ? xen_restore_fl_direct_end+0x0/0x1
> >  [<ffffffff81056487>] ? print_oops_end_marker+0x23/0x25
> >  [<ffffffff81448165>] oops_end+0xb6/0xc6
> >  [<ffffffff810166e5>] die+0x5a/0x63
> >  [<ffffffff81447a3c>] do_trap+0x115/0x124
> >  [<ffffffff810148e6>] do_invalid_op+0x9c/0xa5
> >  [<ffffffff8103a3cb>] ? leave_mm+0x15/0x46
> >  [<ffffffff8100f6e6>] ? xen_clocksource_read+0x21/0x23
> >  [<ffffffff8100f258>] ? HYPERVISOR_vcpu_op+0xf/0x11
> >  [<ffffffff8100f753>] ? xen_vcpuop_set_next_event+0x52/0x67
> >  [<ffffffff81013b3b>] invalid_op+0x1b/0x20
> >  [<ffffffff81447292>] ? _spin_unlock_irqrestore+0x15/0x17
> >  [<ffffffff8103a3cb>] ? leave_mm+0x15/0x46
> >  [<ffffffff8100e4a4>] drop_other_mm_ref+0x2a/0x53
> >  [<ffffffff81087224>] generic_smp_call_function_single_interrupt+0xd8/0xfc
> >  [<ffffffff810100e8>] xen_call_function_single_interrupt+0x13/0x28
> >  [<ffffffff810a936a>] handle_IRQ_event+0x66/0x120
> >  [<ffffffff810aac5b>] handle_percpu_irq+0x41/0x6e
> >  [<ffffffff8128c1a8>] __xen_evtchn_do_upcall+0x1ab/0x27d
> >  [<ffffffff8128dcf9>] xen_evtchn_do_upcall+0x33/0x46
> >  [<ffffffff81013efe>] xen_do_hypervisor_callback+0x1e/0x30
> >  <EOI>  [<ffffffff81447292>] ? _spin_unlock_irqrestore+0x15/0x17
> >  [<ffffffff8100f8af>] ? xen_restore_fl_direct_end+0x0/0x1
> >  [<ffffffff81113f75>] ? flush_old_exec+0x3ac/0x500
> >  [<ffffffff81150dc9>] ? load_elf_binary+0x0/0x17ef
> >  [<ffffffff81150dc9>] ? load_elf_binary+0x0/0x17ef
> >  [<ffffffff81151161>] ? load_elf_binary+0x398/0x17ef
> >  [<ffffffff81042fcf>] ? need_resched+0x23/0x
> > 2d
> >  [<ffffffff811f463c>] ? process_measurement+0xc0/0xd7
> >  [<ffffffff81150dc9>] ? load_elf_binary+0x0/0x17ef
> >  [<ffffffff81113098>] ? search_binary_handler+0xc8/0x255
> >  [<ffffffff81114366>] ? do_execve+0x1c3/0x29e
> >  [<ffffffff8101155d>] ? sys_execve+0x43/0x5d
> >  [<ffffffff8106fc45>] ? __call_usermodehelper+0x0/0x6f
> >  [<ffffffff81013e28>] ? kernel_execve+0x68/0xd0
> >  [<ffffffff8106fc45>] ? __call_usermodehelper+0x0/0x6f
> >  [<ffffffff8100f8af>] ? xen_restore_fl_direct_end+0x0/0x1
> >  [<ffffffff8106fb64>] ? ____call_usermodehelper+0x113/0x11e
> >  [<ffffffff81013daa>] ? child_rip+0xa/0x20
> >  [<ffffffff8106fc45>] ? __call_usermodehelper+0x0/0x6f
> >  [<ffffffff81012f91>] ? int_ret_from_sys_call+0x7/0x1b
> >  [<ffffffff8101371d>] ? retint_restore_args+0x5/0x6
> >  [<ffffffff81013da0>] ? child_rip+0x0/0x20
> > (XEN) Domain 0 crashed: 'noreboot' set - not rebooting.
> >
> >> Date: Tue, 12 Apr 2011 06:00:00 -0400
> >> From: konrad.wilk@oracle.com
> >> To: tinnycloud@hotmail.com
> >> CC: xen-devel@lists.xensource.com; giamteckchoon@gmail.com;
> >> jeremy@goop.org
> >> Subject: Re: Kernel BUG at arch/x86/mm/tlb.c:61
> >>
> >> On Tue, Apr 12, 2011 at 05:11:51PM +0800, MaoXiaoyun wrote:
> >> >
> >> > Hi :
> >> >
> >> > We are using pvops kernel 2.6.32.36 + xen 4.0.1, but confront a kernel
> >> > panic bug.
> >> >
> >> > 2.6.32.36 Kernel:
> >> > http://git.kernel.org/?p=linux/kernel/git/jeremy/xen.git;a=commit;h=bb1a15e55ec665a64c8a9c6bd699b1f16ac01ff4
> >> > Xen 4.0.1 http://xenbits.xen.org/hg/xen-4.0-testing.hg/rev/b536ebfba183
> >> >
> >> > Our test is simple, 24 HVMS(Win2003 ) on a single host, each HVM loopes
> >> > in restart every 15minutes.
> >>
> >> What is the storage that you are using for your guests? AoE? Local disks?
> >>
> >> > About 17 machines are invovled in the test, after 10 hours run, one
> >> > confrontted a crash at arch/x86/mm/tlb.c:61
> >> >
> >> > Currently I am trying "cpuidle=0 cpufreq=none" tests based on Teck's
> >> > suggestion.
> >> >
> >> > Any comments, thanks.
> >> >

 		 	   		  

[-- Attachment #1.2: Type: text/html, Size: 13881 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

RE: Kernel BUG at arch/x86/mm/tlb.c:61

[MaoXiaoyun]

[-- Attachment #1.1: Type: text/plain, Size: 10460 bytes --]


Hi:
 
       As I go through the code. 
       From tlb.c:60, it looks like it  cpu_tlbstate.state  is TLBSTATE_OK, 
which indicates in user space, but the caller, in mmu.c:1512,  
(active_mm == mm) indicates kernel space, that the conflict.
     
   Well, the panic CPU is processing IPI interrupt, could it be something wrong
with CPU mask? 
 
   thanks. 
 
======arch/x86/mm/tlb.c===
 58 void leave_mm(int cpu)
 59 {
 60 <+++if (percpu_read(cpu_tlbstate.state) == TLBSTATE_OK)
 61 <+++<+++BUG();                                                                                                                                           
 62 <+++cpumask_clear_cpu(cpu,
 63 <+++<+++<+++  mm_cpumask(percpu_read(cpu_tlbstate.active_mm)));
 64 <+++load_cr3(swapper_pg_dir);
 65 }
 66 EXPORT_SYMBOL_GPL(leave_mm);
 67 
 
///arch/x86/xen/mmu.c 
 
1502 #ifdef CONFIG_SMP
1503 /* Another cpu may still have their %cr3 pointing at the pagetable, so
1504    we need to repoint it somewhere else before we can unpin it. */
1505 static void drop_other_mm_ref(void *info)
1506 {
1507 <+++struct mm_struct *mm = info;
1508 <+++struct mm_struct *active_mm;
1509 
1510 <+++active_mm = percpu_read(cpu_tlbstate.active_mm);
1511 
1512 <+++if (active_mm == mm)
1513 <+++<+++leave_mm(smp_processor_id());                                                                                                                   
1514 
1515 <+++/* If this cpu still has a stale cr3 reference, then make sure
1516 <+++   it has been flushed. */
1517 <+++if (percpu_read(xen_current_cr3) == __pa(mm->pgd))
1518 <+++<+++load_cr3(swapper_pg_dir);
1519 }





 
> Date: Thu, 14 Apr 2011 15:26:14 +0800
> Subject: Re: Kernel BUG at arch/x86/mm/tlb.c:61
> From: giamteckchoon@gmail.com
> To: tinnycloud@hotmail.com
> CC: xen-devel@lists.xensource.com; jeremy@goop.org; konrad.wilk@oracle.com
> 
> 2011/4/14 MaoXiaoyun <tinnycloud@hotmail.com>:
> > Hi:
> >
> >       I've done test with "cpuidle=0 cpufreq=none", two machine crashed.
> >
> > blktap_sysfs_destroy
> > blktap_sysfs_destroy
> > blktap_sysfs_create: adding attributes for dev ffff8800ad581000
> > blktap_sysfs_create: adding attributes for dev ffff8800a48e3e00
> > ------------[ cut here ]------------
> > kernel BUG at arch/x86/mm/tlb.c:61!
> > invalid opcode: 0000 [#1] SMP
> > last sysfs file: /sys/block/tapdeve/dev
> > CPU 0
> > Modules linked in: 8021q garp blktap xen_netback xen_blkback blkback_pagemap nbd bridge stp llc autofs4 ipmi_devintf ipmi_si ipmi_ms
> > ghandler lockd sunrpc bonding ipv6 xenfs dm_multipath video output sbs sbshc parport_pc lp parport ses enclosure snd_seq_dummy bnx2
> > serio_raw snd_seq_oss snd_seq_midi_event snd_seq snd_seq_device snd_pcm_oss snd_mixer_oss snd_pcm i2c_i801 snd_timer i2c_core snd iT
> > CO_wdt pata_acpi soundcore iTCO_vendor_
> > support ata_generic snd_page_alloc pcspkr ata_piix shpchp mptsas mptscsih mptbase [last unloa
> > ded: freq_table]
> > Pid: 8022, comm: khelper Not tainted 2.6.32.36xen #1 Tecal RH2285
> > RIP: e030:[<ffffffff8103a3cb>]  [<ffffffff8103a3cb>] leave_mm+0x15/0x46
> > RSP: e02b:ffff88002803ee48  EFLAGS: 00010046
> > RAX: 0000000000000000 RBX: 0000000000000001 RCX: ffffffff81675980
> > RDX: ffff88002803ee78 RSI: 0000000000000000 RDI: 0000000000000000
> > RBP: ffff88002803ee48 R08: ffff8800a4929000 R09: dead000000200200
> > R10: dead000000100100 R11: ffffffff81447292 R12: ffff88012ba07b80
> > R13: ffff880028046020 R14: 00000000000004fb R15: 0000000000000000
> > FS:  00007f410af416e0(0000) GS:ffff88002803b000(0000) knlGS:0000000000000000
> > CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
> > CR2: 0000000000469000 CR3: 00000000ad639000 CR4: 0000000000002660
> > DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> > DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> > Process khelper (pid: 8022, threadinfo ffff8800a4846000, task ffff8800a9ed0000)
> > Stack:
> >  ffff88002803ee68 ffffffff8100e4a4 0000000000000001 ffff880097de3b88
> > <0> ffff88002803ee98 ffffffff81087224 ffff88002803ee78 ffff88002803ee78
> > <0> ffff88015f808180 00000000000004fb ffff88002803eea8 ffffffff810100e8
> > Call Trace:
> >  <IRQ>
> >  [<ffffffff8100e4a4>] drop_other_mm_ref+0x2a/0x53
> >  [<ffffffff81087224>] generic_smp_call_function_single_interrupt+0xd8/0xfc
> >  [<ffffffff810100e8>] xen_call_function_single_interrupt+0x13/0x28
> >  [<ffffffff810a936a>] handle_IRQ_event+0x66/0x120
> >  [<ffffffff810aac5b>] handle_percpu_irq+0x41/0x6e
> >  [<ffffffff8128c1a8>] __xen_evtchn_do_upcall+0x1ab/0x27d
> >  [<ffffffff8128dcf9>] xen_evtchn_do_upcall+0x33/0x46
> >  [<ffffffff81013efe>] xen_do_hypervisor_callback+0x1e/0x30
> >  <EOI>
> >  [<ffffffff81447292>] ? _spin_unlock_irqrestore+0x15/0x17
> >  [<ffffffff8100f8af>] ? xen_restore_fl_direct_end+0x0/0x1
> >  [<ffffffff81113f75>] ? flush_old_exec+0x3ac/0x500
> >  [<ffffffff81150dc9>] ? load_elf_binary+0x0/0x17ef
> >  [<ffffffff81150dc9>] ? load_elf_binary+0x0/0x17ef
> >  [<ffffffff81151161>] ? load_elf_binary+0x398/0x17ef
> >  [<ffffffff81042fcf>] ? need_resched+0x23/0x2d
> >
> > [<ffffffff811f463c>] ? process_measurement+0xc0/0xd7
> >  [<ffffffff81150dc9>] ? load_elf_binary+0x0/0x17ef
> >  [<ffffffff81113098>] ? search_binary_handler+0xc8/0x255
> >  [<ffffffff81114366>] ? do_execve+0x1c3/0x29e
> >  [<ffffffff8101155d>] ? sys_execve+0x43/0x5d
> >  [<ffffffff8106fc45>] ? __call_usermodehelper+0x0/0x6f
> >  [<ffffffff81013e28>] ? kernel_execve+0x68/0xd0
> >  [<ffffffff8106fc45>] ? __call_usermodehelper+0x0/0x6f
> >  [<ffffffff8100f8af>] ? xen_restore_fl_direct_end+0x0/0x1
> >  [<ffffffff8106fb64>] ? ____call_usermodehelper+0x113/0x11e
> >  [<ffffffff81013daa>] ? child_rip+0xa/0x20
> >  [<ffffffff8106fc45>] ? __call_usermodehelper+0x0/0x6f
> >  [<ffffffff81012f91>] ? int_ret_from_sys_call+0x7/0x1b
> >  [<ffffffff8101371d>] ? retint_restore_args+0x5/0x6
> >  [<ffffffff81013da0>] ? c
> > hild_rip+0x0/0x20
> > Code: 41 5e 41 5f c9 c3 55 48 89 e5 0f 1f 44 00 00 e8 17 ff ff ff c9 c3 55 48 89 e5 0f 1f 44 00 00 65 8b 04 25 c8 55 01 00 ff c8 75 04 <0f> 0b eb fe 65 48 8b 34 25 c0 55 01 00 48 81 c6 b8 02 00 00 e8
> > RIP  [<ffffffff8103a3cb>] leave_mm+0x15/0x46
> >  RSP <ffff88002803ee48>
> > ---[ end trace 1522f17fdfc9162d ]---
> > Kernel panic - not syncing: Fatal exception in interrupt
> > Pid: 8022, comm: khelper Tainted: G      D    2.6.32.36xen #1
> > Call Trace:
> >  <IRQ>  [<ffffffff8105682e>] panic+0xe0/0x19a
> >  [<ffffffff8144006a>] ? init_amd+0x296/0x37a
> >  [<ffffffff8100f169>] ? xen_force_evtchn_callback+0xd/0xf
> >  [<ffffffff8100f8c2>] ? check_events+0x12/0x20
> >  [<ffffffff8100f8af>] ? xen_restore_fl_direct_end+0x0/0x1
> >  [<ffffffff81056487>] ? print_oops_end_marker+0x23/0x25
> >  [<ffffffff81448165>] oops_end+0xb6/0xc6
> >  [<ffffffff810166e5>] die+0x5a/0x63
> >  [<ffffffff81447a3c>] do_trap+0x115/0x124
> >  [<ffffffff810148e6>] do_invalid_op+0x9c/0xa5
> >  [<ffffffff8103a3cb>] ? leave_mm+0x15/0x46
> >  [<ffffffff8100f6e6>] ? xen_clocksource_read+0x21/0x23
> >  [<ffffffff8100f258>] ? HYPERVISOR_vcpu_op+0xf/0x11
> >  [<ffffffff8100f753>] ? xen_vcpuop_set_next_event+0x52/0x67
> >  [<ffffffff81013b3b>] invalid_op+0x1b/0x20
> >  [<ffffffff81447292>] ? _spin_unlock_irqrestore+0x15/0x17
> >  [<ffffffff8103a3cb>] ? leave_mm+0x15/0x46
> >  [<ffffffff8100e4a4>] drop_other_mm_ref+0x2a/0x53
> >  [<ffffffff81087224>] generic_smp_call_function_single_interrupt+0xd8/0xfc
> >  [<ffffffff810100e8>] xen_call_function_single_interrupt+0x13/0x28
> >  [<ffffffff810a936a>] handle_IRQ_event+0x66/0x120
> >  [<ffffffff810aac5b>] handle_percpu_irq+0x41/0x6e
> >  [<ffffffff8128c1a8>] __xen_evtchn_do_upcall+0x1ab/0x27d
> >  [<ffffffff8128dcf9>] xen_evtchn_do_upcall+0x33/0x46
> >  [<ffffffff81013efe>] xen_do_hypervisor_callback+0x1e/0x30
> >  <EOI>  [<ffffffff81447292>] ? _spin_unlock_irqrestore+0x15/0x17
> >  [<ffffffff8100f8af>] ? xen_restore_fl_direct_end+0x0/0x1
> >  [<ffffffff81113f75>] ? flush_old_exec+0x3ac/0x500
> >  [<ffffffff81150dc9>] ? load_elf_binary+0x0/0x17ef
> >  [<ffffffff81150dc9>] ? load_elf_binary+0x0/0x17ef
> >  [<ffffffff81151161>] ? load_elf_binary+0x398/0x17ef
> >  [<ffffffff81042fcf>] ? need_resched+0x23/0x
> > 2d
> >  [<ffffffff811f463c>] ? process_measurement+0xc0/0xd7
> >  [<ffffffff81150dc9>] ? load_elf_binary+0x0/0x17ef
> >  [<ffffffff81113098>] ? search_binary_handler+0xc8/0x255
> >  [<ffffffff81114366>] ? do_execve+0x1c3/0x29e
> >  [<ffffffff8101155d>] ? sys_execve+0x43/0x5d
> >  [<ffffffff8106fc45>] ? __call_usermodehelper+0x0/0x6f
> >  [<ffffffff81013e28>] ? kernel_execve+0x68/0xd0
> >  [<ffffffff8106fc45>] ? __call_usermodehelper+0x0/0x6f
> >  [<ffffffff8100f8af>] ? xen_restore_fl_direct_end+0x0/0x1
> >  [<ffffffff8106fb64>] ? ____call_usermodehelper+0x113/0x11e
> >  [<ffffffff81013daa>] ? child_rip+0xa/0x20
> >  [<ffffffff8106fc45>] ? __call_usermodehelper+0x0/0x6f
> >  [<ffffffff81012f91>] ? int_ret_from_sys_call+0x7/0x1b
> >  [<ffffffff8101371d>] ? retint_restore_args+0x5/0x6
> >  [<ffffffff81013da0>] ? child_rip+0x0/0x20
> > (XEN) Domain 0 crashed: 'noreboot' set - not rebooting.
> >
> >> Date: Tue, 12 Apr 2011 06:00:00 -0400
> >> From: konrad.wilk@oracle.com
> >> To: tinnycloud@hotmail.com
> >> CC: xen-devel@lists.xensource.com; giamteckchoon@gmail.com;
> >> jeremy@goop.org
> >> Subject: Re: Kernel BUG at arch/x86/mm/tlb.c:61
> >>
> >> On Tue, Apr 12, 2011 at 05:11:51PM +0800, MaoXiaoyun wrote:
> >> >
> >> > Hi :
> >> >
> >> > We are using pvops kernel 2.6.32.36 + xen 4.0.1, but confront a kernel
> >> > panic bug.
> >> >
> >> > 2.6.32.36 Kernel:
> >> > http://git.kernel.org/?p=linux/kernel/git/jeremy/xen.git;a=commit;h=bb1a15e55ec665a64c8a9c6bd699b1f16ac01ff4
> >> > Xen 4.0.1 http://xenbits.xen.org/hg/xen-4.0-testing.hg/rev/b536ebfba183
> >> >
> >> > Our test is simple, 24 HVMS(Win2003 ) on a single host, each HVM loopes
> >> > in restart every 15minutes.
> >>
> >> What is the storage that you are using for your guests? AoE? Local disks?
> >>
> >> > About 17 machines are invovled in the test, after 10 hours run, one
> >> > confrontted a crash at arch/x86/mm/tlb.c:61
> >> >
> >> > Currently I am trying "cpuidle=0 cpufreq=none" tests based on Teck's
> >> > suggestion.
> >> >
> >> > Any comments, thanks.
> >> >


 		 	   		  

[-- Attachment #1.2: Type: text/html, Size: 17127 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

RE: Kernel BUG at arch/x86/mm/tlb.c:61

[MaoXiaoyun]

[-- Attachment #1.1: Type: text/plain, Size: 1166 bytes --]


Hi：

Could the crash  related to this patch ? 
http://git.kernel.org/?p=linux/kernel/git/jeremy/xen.git;a=commitdiff;h=45bfd7bfc6cf32f8e60bb91b32349f0b5090eea3

Since now TLB state  change to TLBSTATE_OK(mmu_context.h:40) is before cpumask_clear_cpu(line 49).
Could it possible that right after execute line 40 of mmu_context.h,  CPU revice IPI from other CPU to 
flush the mm, and when in interrupt, find the TLB state happened to be TLBSTATE_OK. Which conflicts.

Thanks.

arch/x86/include/asm/mmu_context.h
 
33 static inline void switch_mm(struct mm_struct *prev, struct mm_struct *next,
 34 <+++<+++<+++     struct task_struct *tsk)
 35 {
 36 <+++unsigned cpu = smp_processor_id();
 37 
 38 <+++if (likely(prev != next)) {
 39 #ifdef CONFIG_SMP
 40 <+++<+++percpu_write(cpu_tlbstate.state, TLBSTATE_OK);
 41 <+++<+++percpu_write(cpu_tlbstate.active_mm, next);
 42 #endif
 43 <+++<+++cpumask_set_cpu(cpu, mm_cpumask(next));
 44 
 45 <+++<+++/* Re-load page tables */
 46 <+++<+++load_cr3(next->pgd);
 47 
 48 <+++<+++/* stop flush ipis for the previous mm */
 49 <+++<+++cpumask_clear_cpu(cpu, mm_cpumask(prev));  

 		 	   		  

[-- Attachment #1.2: Type: text/html, Size: 3143 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Kernel BUG at arch/x86/mm/tlb.c:61

[Jeremy Fitzhardinge]

On 04/15/2011 05:23 AM, MaoXiaoyun wrote:
> Hi：
>
> Could the crash related to this patch ?
> http://git.kernel.org/?p=linux/kernel/git/jeremy/xen.git;a=commitdiff;h=45bfd7bfc6cf32f8e60bb91b32349f0b5090eea3
>
> Since now TLB state change to TLBSTATE_OK(mmu_context.h:40) is before
> cpumask_clear_cpu(line 49).
> Could it possible that right after execute line 40 of mmu_context.h,
> CPU revice IPI from other CPU to
> flush the mm, and when in interrupt, find the TLB state happened to be
> TLBSTATE_OK. Which conflicts.

Does reverting it help?

J

>
> Thanks.
>
> arch/x86/include/asm/mmu_context.h
>
> 33 static inline void switch_mm(struct mm_struct *prev, struct
> mm_struct *next,
> 34 <+++<+++<+++ struct task_struct *tsk)
> 35 {
> 36 <+++unsigned cpu = smp_processor_id();
> 37
> 38 <+++if (likely(prev != next)) {
> 39 #ifdef CONFIG_SMP
> 40 <+++<+++percpu_write(cpu_tlbstate.state, TLBSTATE_OK);
> 41 <+++<+++percpu_write(cpu_tlbstate.active_mm, next);
> 42 #endif
> 43 <+++<+++cpumask_set_cpu(cpu, mm_cpumask(next));
> 44
> 45 <+++<+++/* Re-load page tables */
> 46 <+++<+++load_cr3(next->pgd);
> 47
> 48 <+++<+++/* stop flush ipis for the previous mm */
> 49 <+++<+++cpumask_clear_cpu(cpu, mm_cpumask(prev));
>
>

RE: Kernel BUG at arch/x86/mm/tlb.c:61

[MaoXiaoyun]

[-- Attachment #1.1: Type: text/plain, Size: 2722 bytes --]



 

> Date: Fri, 15 Apr 2011 14:22:29 -0700
> From: jeremy@goop.org
> To: tinnycloud@hotmail.com
> CC: giamteckchoon@gmail.com; xen-devel@lists.xensource.com; konrad.wilk@oracle.com
> Subject: Re: Kernel BUG at arch/x86/mm/tlb.c:61
> 
> On 04/15/2011 05:23 AM, MaoXiaoyun wrote:
> > Hi：
> >
> > Could the crash related to this patch ?
> > http://git.kernel.org/?p=linux/kernel/git/jeremy/xen.git;a=commitdiff;h=45bfd7bfc6cf32f8e60bb91b32349f0b5090eea3
> >
> > Since now TLB state change to TLBSTATE_OK(mmu_context.h:40) is before
> > cpumask_clear_cpu(line 49).
> > Could it possible that right after execute line 40 of mmu_context.h,
> > CPU revice IPI from other CPU to
> > flush the mm, and when in interrupt, find the TLB state happened to be
> > TLBSTATE_OK. Which conflicts.
> 
> Does reverting it help?
> 
> J
 
Very likely.
 
Previously in 17 machines test, one to three machines will fail in 10hours, very easily.
 
But after reverting, we have 29machines involved the test, 28 successfuly rung 2 days, 1 fail after 28 hours. 
Unfortunately I can't tell wether the failed one related to this bug, since I got no log in messages.  And
the machine was reboot by someone before I can see something from serial port.
 
But in my opinion the fail points to another bug, which I happened to confront before.  
 
Before, one of my develop machine(2.6.32.36kernel+xen4.0.1) completely stop response, 
including serial console. There is no abnormal message in serial port,  looks like Xen runs in deadlock. 
Well, it is rarely happen, since I only met once till now. 
 
Now I am trying to figure out what might cause the deadlock, we never met this before.
I don't have clear thoughts on how to dig it out, but  I think this bug exists in Xen.
since if dom0 hangs, xen should work,  and serial output will response.
 If so, the bug may be introduced between 4.0.0 and 4.0.1.
 
What do you think,  thanks.

> >
> > Thanks.
> >
> > arch/x86/include/asm/mmu_context.h
> >
> > 33 static inline void switch_mm(struct mm_struct *prev, struct
> > mm_struct *next,
> > 34 <+++<+++<+++ struct task_struct *tsk)
> > 35 {
> > 36 <+++unsigned cpu = smp_processor_id();
> > 37
> > 38 <+++if (likely(prev != next)) {
> > 39 #ifdef CONFIG_SMP
> > 40 <+++<+++percpu_write(cpu_tlbstate.state, TLBSTATE_OK);
> > 41 <+++<+++percpu_write(cpu_tlbstate.active_mm, next);
> > 42 #endif
> > 43 <+++<+++cpumask_set_cpu(cpu, mm_cpumask(next));
> > 44
> > 45 <+++<+++/* Re-load page tables */
> > 46 <+++<+++load_cr3(next->pgd);
> > 47
> > 48 <+++<+++/* stop flush ipis for the previous mm */
> > 49 <+++<+++cpumask_clear_cpu(cpu, mm_cpumask(prev));
> >
> >
> 
 		 	   		  

[-- Attachment #1.2: Type: text/html, Size: 3520 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

RE: Kernel BUG at arch/x86/mm/tlb.c:61

[MaoXiaoyun]

[-- Attachment #1.1: Type: text/plain, Size: 2620 bytes --]


 

> Date: Fri, 15 Apr 2011 14:22:29 -0700
> From: jeremy@goop.org
> To: tinnycloud@hotmail.com
> CC: giamteckchoon@gmail.com; xen-devel@lists.xensource.com; konrad.wilk@oracle.com
> Subject: Re: Kernel BUG at arch/x86/mm/tlb.c:61
> 
> On 04/15/2011 05:23 AM, MaoXiaoyun wrote:
> > Hi：
> >
> > Could the crash related to this patch ?
> > http://git.kernel.org/?p=linux/kernel/git/jeremy/xen.git;a=commitdiff;h=45bfd7bfc6cf32f8e60bb91b32349f0b5090eea3
> >
> > Since now TLB state change to TLBSTATE_OK(mmu_context.h:40) is before
> > cpumask_clear_cpu(line 49).
> > Could it possible that right after execute line 40 of mmu_context.h,
> > CPU revice IPI from other CPU to
> > flush the mm, and when in interrupt, find the TLB state happened to be
> > TLBSTATE_OK. Which conflicts.
> 
> Does reverting it help?
> 
> J
 
Hi Jeremy:
 
    The lastest test result shows the reverting didn't help.
    Kernel panic exactly at the same place in tlb.c.
 
    I have question about TLB state, from the stack, 
    xen_do_hypervisor_callback-> xen_evtchn_do_upcall->... ->drop_other_mm_ref
 
    What  cpu_tlbstate.state should be,  could  TLBSTATE_OK or TLBSTATE_LAZY all be possible? 
    That is after a hypercall from userspace, state will be TLBSTATE_OK, and
      if from kernel space, state will be TLBSTATE_LAZE ? 
 
       thanks.
    
 [<ffffffff8100e4a4>] drop_other_mm_ref+0x2a/0x53
 [<ffffffff81087224>] generic_smp_call_function_single_interrupt+0xd8/0xfc
 [<ffffffff810100e8>] xen_call_function_single_interrupt+0x13/0x28
 [<ffffffff810a936a>] handle_IRQ_event+0x66/0x120
 [<ffffffff810aac5b>] handle_percpu_irq+0x41/0x6e
 [<ffffffff8128c1a8>] __xen_evtchn_do_upcall+0x1ab/0x27d
 [<ffffffff8128dcf9>] xen_evtchn_do_upcall+0x33/0x46
 [<ffffffff81013efe>] xen_do_hypervisor_callback+0x1e/0x30


> 
> >
> > Thanks.
> >
> > arch/x86/include/asm/mmu_context.h
> >
> > 33 static inline void switch_mm(struct mm_struct *prev, struct
> > mm_struct *next,
> > 34 <+++<+++<+++ struct task_struct *tsk)
> > 35 {
> > 36 <+++unsigned cpu = smp_processor_id();
> > 37
> > 38 <+++if (likely(prev != next)) {
> > 39 #ifdef CONFIG_SMP
> > 40 <+++<+++percpu_write(cpu_tlbstate.state, TLBSTATE_OK);
> > 41 <+++<+++percpu_write(cpu_tlbstate.active_mm, next);
> > 42 #endif
> > 43 <+++<+++cpumask_set_cpu(cpu, mm_cpumask(next));
> > 44
> > 45 <+++<+++/* Re-load page tables */
> > 46 <+++<+++load_cr3(next->pgd);
> > 47
> > 48 <+++<+++/* stop flush ipis for the previous mm */
> > 49 <+++<+++cpumask_clear_cpu(cpu, mm_cpumask(prev));
> >
> >
> 
 		 	   		  

[-- Attachment #1.2: Type: text/html, Size: 8237 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

RE: RE: Kernel BUG at arch/x86/mm/tlb.c:61

[Tian, Kevin]

[-- Attachment #1: Type: text/plain, Size: 2918 bytes --]

>From: MaoXiaoyun
>Sent: Monday, April 25, 2011 11:15 AM
>> Date: Fri, 15 Apr 2011 14:22:29 -0700
>> From: jeremy@goop.org
>> To: tinnycloud@hotmail.com
>> CC: giamteckchoon@gmail.com; xen-devel@lists.xensource.com; konrad.wilk@oracle.com
>> Subject: Re: Kernel BUG at arch/x86/mm/tlb.c:61
>> 
>> On 04/15/2011 05:23 AM, MaoXiaoyun wrote:
>> > Hi：
>> >
>> > Could the crash related to this patch ?
>> > http://git.kernel.org/?p=linux/kernel/git/jeremy/xen.git;a=commitdiff;h=45bfd7bfc6cf32f8e60bb91b32349f0b5090eea3
>> >
>> > Since now TLB state change to TLBSTATE_OK(mmu_context.h:40) is before
>> > cpumask_clear_cpu(line 49).
>> > Could it possible that right after execute line 40 of mmu_context.h,
>> > CPU revice IPI from other CPU to
>> > flush the mm, and when in interrupt, find the TLB state happened to be
>> > TLBSTATE_OK. Which conflicts.
>> 
>> Does reverting it help?
>> 
>> J
> 
>Hi Jeremy:
> 
>    The lastest test result shows the reverting didn't help.
>    Kernel panic exactly at the same place in tlb.c.
> 
>    I have question about TLB state, from the stack, 
>    xen_do_hypervisor_callback-> xen_evtchn_do_upcall->... ->drop_other_mm_ref
> 
>    What  cpu_tlbstate.state should be,  could  TLBSTATE_OK or TLBSTATE_LAZY all be possible? 
>    That is after a hypercall from userspace, state will be TLBSTATE_OK, and
>      if from kernel space, state will be TLBSTATE_LAZE ? 
> 
>       thanks.

it looks a bug in drop_other_mm_ref implementation, that current TLB state should be checked
before invoking leave_mm(). There's a window between below lines of code:

<xen_drop_mm_ref>
       /* Get the "official" set of cpus referring to our pagetable. */
        if (!alloc_cpumask_var(&mask, GFP_ATOMIC)) {
                for_each_online_cpu(cpu) {
                        if (!cpumask_test_cpu(cpu, mm_cpumask(mm))
                            && per_cpu(xen_current_cr3, cpu) != __pa(mm->pgd))
                                continue;
                        smp_call_function_single(cpu, drop_other_mm_ref, mm, 1);
                }
                return;
        }

there's chance that when smp_call_function_single is invoked, actual TLB state has been
updated in the other cpu. The upstream kernel patch you referred to earlier just makes
this bug exposed more easily. But even without this patch, you may still suffer such issue
which is why reverting the patch doesn't help.

Could you try adding a check in drop_other_mm_ref?

        if (active_mm == mm && percpu_read(cpu_tlbstate.state) != TLBSTATE_OK)
                leave_mm(smp_processor_id());

once the interrupted context has TLBSTATE_OK, it implicates that later it will handle 
the TLB flush and thus no need for leave_mm from interrupt handler, and that's the
assumption of doing leave_mm.

Thanks
Kevin

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

RE: RE: Kernel BUG at arch/x86/mm/tlb.c:61

[MaoXiaoyun]

[-- Attachment #1.1: Type: text/plain, Size: 3725 bytes --]


Many thanks, Kevin.
 
I agree on the race window.
One thing more,  In my understaning, the CPU who send out IPI message, will unpin the pagetable after 
receive all ACKS  from other cpu,  if the CPU who received  IPI message, enter drop_other_mm_ref, and 
has TLBSTATE_OK, does nothing, will it possible it possible confronts with stale pagetable
(that is unpinned by sender CPU)?
 
So do we need flush tlb when its state is TBLSTATE_OK?
 
if (active_mm == mm){
     if (percpu_read(cpu_tlbstate.state) == TLBSTATE_OK)
        load_cr3(mm->pgd)
     else
                leave_mm(smp_processor_id());
 }

 
> From: kevin.tian@intel.com
> To: tinnycloud@hotmail.com; jeremy@goop.org
> CC: xen-devel@lists.xensource.com; giamteckchoon@gmail.com; konrad.wilk@oracle.com
> Date: Tue, 26 Apr 2011 13:52:11 +0800
> Subject: RE: [Xen-devel] RE: Kernel BUG at arch/x86/mm/tlb.c:61
> 
> >From: MaoXiaoyun
> >Sent: Monday, April 25, 2011 11:15 AM
> >> Date: Fri, 15 Apr 2011 14:22:29 -0700
> >> From: jeremy@goop.org
> >> To: tinnycloud@hotmail.com
> >> CC: giamteckchoon@gmail.com; xen-devel@lists.xensource.com; konrad.wilk@oracle.com
> >> Subject: Re: Kernel BUG at arch/x86/mm/tlb.c:61
> >> 
> >> On 04/15/2011 05:23 AM, MaoXiaoyun wrote:
> >> > Hi：
> >> >
> >> > Could the crash related to this patch ?
> >> > http://git.kernel.org/?p=linux/kernel/git/jeremy/xen.git;a=commitdiff;h=45bfd7bfc6cf32f8e60bb91b32349f0b5090eea3
> >> >
> >> > Since now TLB state change to TLBSTATE_OK(mmu_context.h:40) is before
> >> > cpumask_clear_cpu(line 49).
> >> > Could it possible that right after execute line 40 of mmu_context.h,
> >> > CPU revice IPI from other CPU to
> >> > flush the mm, and when in interrupt, find the TLB state happened to be
> >> > TLBSTATE_OK. Which conflicts.
> >> 
> >> Does reverting it help?
> >> 
> >> J
> > 
> >Hi Jeremy:
> > 
> >    The lastest test result shows the reverting didn't help.
> >    Kernel panic exactly at the same place in tlb.c.
> > 
> >    I have question about TLB state, from the stack, 
> >    xen_do_hypervisor_callback-> xen_evtchn_do_upcall->... ->drop_other_mm_ref
> > 
> >    What  cpu_tlbstate.state should be,  could  TLBSTATE_OK or TLBSTATE_LAZY all be possible? 
> >    That is after a hypercall from userspace, state will be TLBSTATE_OK, and
> >      if from kernel space, state will be TLBSTATE_LAZE ? 
> > 
> >       thanks.
> 
> it looks a bug in drop_other_mm_ref implementation, that current TLB state should be checked
> before invoking leave_mm(). There's a window between below lines of code:
> 
> <xen_drop_mm_ref>
> /* Get the "official" set of cpus referring to our pagetable. */
> if (!alloc_cpumask_var(&mask, GFP_ATOMIC)) {
> for_each_online_cpu(cpu) {
> if (!cpumask_test_cpu(cpu, mm_cpumask(mm))
> && per_cpu(xen_current_cr3, cpu) != __pa(mm->pgd))
> continue;
> smp_call_function_single(cpu, drop_other_mm_ref, mm, 1);
> }
> return;
> }
> 
> there's chance that when smp_call_function_single is invoked, actual TLB state has been
> updated in the other cpu. The upstream kernel patch you referred to earlier just makes
> this bug exposed more easily. But even without this patch, you may still suffer such issue
> which is why reverting the patch doesn't help.
> 
> Could you try adding a check in drop_other_mm_ref?
> 
> if (active_mm == mm && percpu_read(cpu_tlbstate.state) != TLBSTATE_OK)
> leave_mm(smp_processor_id());
> 
> once the interrupted context has TLBSTATE_OK, it implicates that later it will handle 
> the TLB flush and thus no need for leave_mm from interrupt handler, and that's the
> assumption of doing leave_mm.
> 
> Thanks
> Kevin
 		 	   		  

[-- Attachment #1.2: Type: text/html, Size: 6181 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

RE: RE: Kernel BUG at arch/x86/mm/tlb.c:61

[Tian, Kevin]

[-- Attachment #1.1: Type: text/plain, Size: 4304 bytes --]

I think that should be fine. note a later check:

        /* If this cpu still has a stale cr3 reference, then make sure
           it has been flushed. */
        if (percpu_read(xen_current_cr3) == __pa(mm->pgd))
                load_cr3(swapper_pg_dir);

this should ensure the stale TLB being flushed if this cpu is still in lazy mode.

Thanks
Kevin

From: MaoXiaoyun [mailto:tinnycloud@hotmail.com]
Sent: Tuesday, April 26, 2011 3:05 PM
To: Tian, Kevin; jeremy@goop.org
Cc: xen devel; giamteckchoon@gmail.com; konrad.wilk@oracle.com
Subject: RE: [Xen-devel] RE: Kernel BUG at arch/x86/mm/tlb.c:61

Many thanks, Kevin.

I agree on the race window.
One thing more,  In my understaning, the CPU who send out IPI message, will unpin the pagetable after
receive all ACKS  from other cpu,  if the CPU who received  IPI message, enter drop_other_mm_ref, and
has TLBSTATE_OK, does nothing, will it possible it possible confronts with stale pagetable
(that is unpinned by sender CPU)?

So do we need flush tlb when its state is TBLSTATE_OK?

if (active_mm == mm){
     if (percpu_read(cpu_tlbstate.state) == TLBSTATE_OK)
        load_cr3(mm->pgd)
     else
                leave_mm(smp_processor_id());
 }

> From: kevin.tian@intel.com
> To: tinnycloud@hotmail.com; jeremy@goop.org
> CC: xen-devel@lists.xensource.com; giamteckchoon@gmail.com; konrad.wilk@oracle.com
> Date: Tue, 26 Apr 2011 13:52:11 +0800
> Subject: RE: [Xen-devel] RE: Kernel BUG at arch/x86/mm/tlb.c:61
>
> >From: MaoXiaoyun
> >Sent: Monday, April 25, 2011 11:15 AM
> >> Date: Fri, 15 Apr 2011 14:22:29 -0700
> >> From: jeremy@goop.org
> >> To: tinnycloud@hotmail.com
> >> CC: giamteckchoon@gmail.com; xen-devel@lists.xensource.com; konrad.wilk@oracle.com
> >> Subject: Re: Kernel BUG at arch/x86/mm/tlb.c:61
> >>
> >> On 04/15/2011 05:23 AM, MaoXiaoyun wrote:
> >> > Hi：
> >> >
> >> > Could the crash related to this patch ?
> >> > http://git.kernel.org/?p=linux/kernel/git/jeremy/xen.git;a=commitdiff;h=45bfd7bfc6cf32f8e60bb91b32349f0b5090eea3
> >> >
> >> > Since now TLB state change to TLBSTATE_OK(mmu_context.h:40) is before
> >> > cpumask_clear_cpu(line 49).
> >> > Could it possible that right after execute line 40 of mmu_context.h,
> >> > CPU revice IPI from other CPU to
> >> > flush the mm, and when in interrupt, find the TLB state happened to be
> >> > TLBSTATE_OK. Which conflicts.
> >>
> >> Does reverting it help?
> >>
> >> J
> >
> >Hi Jeremy:
> >
> >    The lastest test result shows the reverting didn't help.
> >    Kernel panic exactly at the same place in tlb.c.
> >
> >    I have question about TLB state, from the stack,
> >    xen_do_hypervisor_callback-> xen_evtchn_do_upcall->... ->drop_other_mm_ref
> >
> >    What  cpu_tlbstate.state should be,  could  TLBSTATE_OK or TLBSTATE_LAZY all be possible?
> >    That is after a hypercall from userspace, state will be TLBSTATE_OK, and
> >      if from kernel space, state will be TLBSTATE_LAZE ?
> >
> >       thanks.
>
> it looks a bug in drop_other_mm_ref implementation, that current TLB state should be checked
> before invoking leave_mm(). There's a window between below lines of code:
>
> <xen_drop_mm_ref>
> /* Get the "official" set of cpus referring to our pagetable. */
> if (!alloc_cpumask_var(&mask, GFP_ATOMIC)) {
> for_each_online_cpu(cpu) {
> if (!cpumask_test_cpu(cpu, mm_cpumask(mm))
> && per_cpu(xen_current_cr3, cpu) != __pa(mm->pgd))
> continue;
> smp_call_function_single(cpu, drop_other_mm_ref, mm, 1);
> }
> return;
> }
>
> there's chance that when smp_call_function_single is invoked, actual TLB state has been
> updated in the other cpu. The upstream kernel patch you referred to earlier just makes
> this bug exposed more easily. But even without this patch, you may still suffer such issue
> which is why reverting the patch doesn't help.
>
> Could you try adding a check in drop_other_mm_ref?
>
> if (active_mm == mm && percpu_read(cpu_tlbstate.state) != TLBSTATE_OK)
> leave_mm(smp_processor_id());
>
> once the interrupted context has TLBSTATE_OK, it implicates that later it will handle
> the TLB flush and thus no need for leave_mm from interrupt handler, and that's the
> assumption of doing leave_mm.
>
> Thanks
> Kevin

[-- Attachment #1.2: Type: text/html, Size: 11127 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: RE: Kernel BUG at arch/x86/mm/tlb.c:61

[Jeremy Fitzhardinge]

On 04/25/2011 10:52 PM, Tian, Kevin wrote:
>> From: MaoXiaoyun
>> Sent: Monday, April 25, 2011 11:15 AM
>>> Date: Fri, 15 Apr 2011 14:22:29 -0700
>>> From: jeremy@goop.org
>>> To: tinnycloud@hotmail.com
>>> CC: giamteckchoon@gmail.com; xen-devel@lists.xensource.com; konrad.wilk@oracle.com
>>> Subject: Re: Kernel BUG at arch/x86/mm/tlb.c:61
>>>
>>> On 04/15/2011 05:23 AM, MaoXiaoyun wrote:
>>>> Hi：
>>>>
>>>> Could the crash related to this patch ?
>>>> http://git.kernel.org/?p=linux/kernel/git/jeremy/xen.git;a=commitdiff;h=45bfd7bfc6cf32f8e60bb91b32349f0b5090eea3
>>>>
>>>> Since now TLB state change to TLBSTATE_OK(mmu_context.h:40) is before
>>>> cpumask_clear_cpu(line 49).
>>>> Could it possible that right after execute line 40 of mmu_context.h,
>>>> CPU revice IPI from other CPU to
>>>> flush the mm, and when in interrupt, find the TLB state happened to be
>>>> TLBSTATE_OK. Which conflicts.
>>> Does reverting it help?
>>>
>>> J
>>  
>> Hi Jeremy:
>>  
>>     The lastest test result shows the reverting didn't help.
>>     Kernel panic exactly at the same place in tlb.c.
>>  
>>     I have question about TLB state, from the stack, 
>>     xen_do_hypervisor_callback-> xen_evtchn_do_upcall->... ->drop_other_mm_ref
>>  
>>     What  cpu_tlbstate.state should be,  could  TLBSTATE_OK or TLBSTATE_LAZY all be possible? 
>>     That is after a hypercall from userspace, state will be TLBSTATE_OK, and
>>       if from kernel space, state will be TLBSTATE_LAZE ? 
>>  
>>        thanks.
> it looks a bug in drop_other_mm_ref implementation, that current TLB state should be checked
> before invoking leave_mm(). There's a window between below lines of code:
>
> <xen_drop_mm_ref>
>        /* Get the "official" set of cpus referring to our pagetable. */
>         if (!alloc_cpumask_var(&mask, GFP_ATOMIC)) {
>                 for_each_online_cpu(cpu) {
>                         if (!cpumask_test_cpu(cpu, mm_cpumask(mm))
>                             && per_cpu(xen_current_cr3, cpu) != __pa(mm->pgd))
>                                 continue;
>                         smp_call_function_single(cpu, drop_other_mm_ref, mm, 1);
>                 }
>                 return;
>         }
>
> there's chance that when smp_call_function_single is invoked, actual TLB state has been
> updated in the other cpu. The upstream kernel patch you referred to earlier just makes
> this bug exposed more easily. But even without this patch, you may still suffer such issue
> which is why reverting the patch doesn't help.
>
> Could you try adding a check in drop_other_mm_ref?
>
>         if (active_mm == mm && percpu_read(cpu_tlbstate.state) != TLBSTATE_OK)
>                 leave_mm(smp_processor_id());
>
> once the interrupted context has TLBSTATE_OK, it implicates that later it will handle 
> the TLB flush and thus no need for leave_mm from interrupt handler, and that's the
> assumption of doing leave_mm.

That seems reasonable.  MaoXiaoyun, does it fix the bug for you?

Kevin, could you submit this as a proper patch?

Thanks,
    J

RE: RE: Kernel BUG at arch/x86/mm/tlb.c:61

[Tian, Kevin]

[-- Attachment #1: Type: text/plain, Size: 3576 bytes --]

> From: Jeremy Fitzhardinge [mailto:jeremy@goop.org]
> Sent: Friday, April 29, 2011 7:29 AM
> 
> On 04/25/2011 10:52 PM, Tian, Kevin wrote:
> >> From: MaoXiaoyun
> >> Sent: Monday, April 25, 2011 11:15 AM
> >>> Date: Fri, 15 Apr 2011 14:22:29 -0700
> >>> From: jeremy@goop.org
> >>> To: tinnycloud@hotmail.com
> >>> CC: giamteckchoon@gmail.com; xen-devel@lists.xensource.com;
> >>> konrad.wilk@oracle.com
> >>> Subject: Re: Kernel BUG at arch/x86/mm/tlb.c:61
> >>>
> >>> On 04/15/2011 05:23 AM, MaoXiaoyun wrote:
> >>>> Hi：
> >>>>
> >>>> Could the crash related to this patch ?
> >>>> http://git.kernel.org/?p=linux/kernel/git/jeremy/xen.git;a=commitdi
> >>>> ff;h=45bfd7bfc6cf32f8e60bb91b32349f0b5090eea3
> >>>>
> >>>> Since now TLB state change to TLBSTATE_OK(mmu_context.h:40) is
> >>>> before cpumask_clear_cpu(line 49).
> >>>> Could it possible that right after execute line 40 of
> >>>> mmu_context.h, CPU revice IPI from other CPU to flush the mm, and
> >>>> when in interrupt, find the TLB state happened to be TLBSTATE_OK.
> >>>> Which conflicts.
> >>> Does reverting it help?
> >>>
> >>> J
> >>
> >> Hi Jeremy:
> >>
> >>     The lastest test result shows the reverting didn't help.
> >>     Kernel panic exactly at the same place in tlb.c.
> >>
> >>     I have question about TLB state, from the stack,
> >>     xen_do_hypervisor_callback-> xen_evtchn_do_upcall->...
> >> ->drop_other_mm_ref
> >>
> >>     What  cpu_tlbstate.state should be,  could  TLBSTATE_OK or
> TLBSTATE_LAZY all be possible?
> >>     That is after a hypercall from userspace, state will be TLBSTATE_OK,
> and
> >>       if from kernel space, state will be TLBSTATE_LAZE ?
> >>
> >>        thanks.
> > it looks a bug in drop_other_mm_ref implementation, that current TLB
> > state should be checked before invoking leave_mm(). There's a window
> between below lines of code:
> >
> > <xen_drop_mm_ref>
> >        /* Get the "official" set of cpus referring to our pagetable. */
> >         if (!alloc_cpumask_var(&mask, GFP_ATOMIC)) {
> >                 for_each_online_cpu(cpu) {
> >                         if (!cpumask_test_cpu(cpu,
> mm_cpumask(mm))
> >                             && per_cpu(xen_current_cr3, cpu) !=
> __pa(mm->pgd))
> >                                 continue;
> >                         smp_call_function_single(cpu,
> drop_other_mm_ref, mm, 1);
> >                 }
> >                 return;
> >         }
> >
> > there's chance that when smp_call_function_single is invoked, actual
> > TLB state has been updated in the other cpu. The upstream kernel patch
> > you referred to earlier just makes this bug exposed more easily. But
> > even without this patch, you may still suffer such issue which is why reverting
> the patch doesn't help.
> >
> > Could you try adding a check in drop_other_mm_ref?
> >
> >         if (active_mm == mm && percpu_read(cpu_tlbstate.state) !=
> TLBSTATE_OK)
> >                 leave_mm(smp_processor_id());
> >
> > once the interrupted context has TLBSTATE_OK, it implicates that later
> > it will handle the TLB flush and thus no need for leave_mm from
> > interrupt handler, and that's the assumption of doing leave_mm.
> 
> That seems reasonable.  MaoXiaoyun, does it fix the bug for you?
> 
> Kevin, could you submit this as a proper patch?
> 

I'm waiting for Xiaoyun's test result before submitting a proper patch, since this
part of logic is tricky and his test can make sure we don't overlook some corner
cases. :-)

Thanks
Kevin

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

RE: RE: Kernel BUG at arch/x86/mm/tlb.c:61

[MaoXiaoyun]

[-- Attachment #1.1: Type: text/plain, Size: 3927 bytes --]


 

> From: kevin.tian@intel.com
> To: jeremy@goop.org
> CC: tinnycloud@hotmail.com; xen-devel@lists.xensource.com; giamteckchoon@gmail.com; konrad.wilk@oracle.com
> Date: Fri, 29 Apr 2011 08:19:44 +0800
> Subject: RE: [Xen-devel] RE: Kernel BUG at arch/x86/mm/tlb.c:61
> 
> > From: Jeremy Fitzhardinge [mailto:jeremy@goop.org]
> > Sent: Friday, April 29, 2011 7:29 AM
> > 
> > On 04/25/2011 10:52 PM, Tian, Kevin wrote:
> > >> From: MaoXiaoyun
> > >> Sent: Monday, April 25, 2011 11:15 AM
> > >>> Date: Fri, 15 Apr 2011 14:22:29 -0700
> > >>> From: jeremy@goop.org
> > >>> To: tinnycloud@hotmail.com
> > >>> CC: giamteckchoon@gmail.com; xen-devel@lists.xensource.com;
> > >>> konrad.wilk@oracle.com
> > >>> Subject: Re: Kernel BUG at arch/x86/mm/tlb.c:61
> > >>>
> > >>> On 04/15/2011 05:23 AM, MaoXiaoyun wrote:
> > >>>> Hi：
> > >>>>
> > >>>> Could the crash related to this patch ?
> > >>>> http://git.kernel.org/?p=linux/kernel/git/jeremy/xen.git;a=commitdi
> > >>>> ff;h=45bfd7bfc6cf32f8e60bb91b32349f0b5090eea3
> > >>>>
> > >>>> Since now TLB state change to TLBSTATE_OK(mmu_context.h:40) is
> > >>>> before cpumask_clear_cpu(line 49).
> > >>>> Could it possible that right after execute line 40 of
> > >>>> mmu_context.h, CPU revice IPI from other CPU to flush the mm, and
> > >>>> when in interrupt, find the TLB state happened to be TLBSTATE_OK.
> > >>>> Which conflicts.
> > >>> Does reverting it help?
> > >>>
> > >>> J
> > >>
> > >> Hi Jeremy:
> > >>
> > >> The lastest test result shows the reverting didn't help.
> > >> Kernel panic exactly at the same place in tlb.c.
> > >>
> > >> I have question about TLB state, from the stack,
> > >> xen_do_hypervisor_callback-> xen_evtchn_do_upcall->...
> > >> ->drop_other_mm_ref
> > >>
> > >> What cpu_tlbstate.state should be, could TLBSTATE_OK or
> > TLBSTATE_LAZY all be possible?
> > >> That is after a hypercall from userspace, state will be TLBSTATE_OK,
> > and
> > >> if from kernel space, state will be TLBSTATE_LAZE ?
> > >>
> > >> thanks.
> > > it looks a bug in drop_other_mm_ref implementation, that current TLB
> > > state should be checked before invoking leave_mm(). There's a window
> > between below lines of code:
> > >
> > > <xen_drop_mm_ref>
> > > /* Get the "official" set of cpus referring to our pagetable. */
> > > if (!alloc_cpumask_var(&mask, GFP_ATOMIC)) {
> > > for_each_online_cpu(cpu) {
> > > if (!cpumask_test_cpu(cpu,
> > mm_cpumask(mm))
> > > && per_cpu(xen_current_cr3, cpu) !=
> > __pa(mm->pgd))
> > > continue;
> > > smp_call_function_single(cpu,
> > drop_other_mm_ref, mm, 1);
> > > }
> > > return;
> > > }
> > >
> > > there's chance that when smp_call_function_single is invoked, actual
> > > TLB state has been updated in the other cpu. The upstream kernel patch
> > > you referred to earlier just makes this bug exposed more easily. But
> > > even without this patch, you may still suffer such issue which is why reverting
> > the patch doesn't help.
> > >
> > > Could you try adding a check in drop_other_mm_ref?
> > >
> > > if (active_mm == mm && percpu_read(cpu_tlbstate.state) !=
> > TLBSTATE_OK)
> > > leave_mm(smp_processor_id());
> > >
> > > once the interrupted context has TLBSTATE_OK, it implicates that later
> > > it will handle the TLB flush and thus no need for leave_mm from
> > > interrupt handler, and that's the assumption of doing leave_mm.
> > 
> > That seems reasonable. MaoXiaoyun, does it fix the bug for you?
> > 
> > Kevin, could you submit this as a proper patch?
> > 
> 
> I'm waiting for Xiaoyun's test result before submitting a proper patch, since this
> part of logic is tricky and his test can make sure we don't overlook some corner
> cases. :-)
> 
 
I think it works. The test has been running over 70 hours successfully.
My plan is run one week.
 
Thanks. 
 
> Thanks
> Kevin
 		 	   		  

[-- Attachment #1.2: Type: text/html, Size: 5407 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

RE: RE: Kernel BUG at arch/x86/mm/tlb.c:61

[Tian, Kevin]

[-- Attachment #1.1: Type: text/plain, Size: 4603 bytes --]

OK, thanks for the update. I’ll send out the patch then

Thanks
Kevin

From: MaoXiaoyun [mailto:tinnycloud@hotmail.com]
Sent: Friday, April 29, 2011 9:51 AM
To: Tian, Kevin; jeremy@goop.org
Cc: xen devel; giamteckchoon@gmail.com; konrad.wilk@oracle.com
Subject: RE: [Xen-devel] RE: Kernel BUG at arch/x86/mm/tlb.c:61


> From: kevin.tian@intel.com<mailto:kevin.tian@intel.com>
> To: jeremy@goop.org<mailto:jeremy@goop.org>
> CC: tinnycloud@hotmail.com<mailto:tinnycloud@hotmail.com>; xen-devel@lists.xensource.com<mailto:xen-devel@lists.xensource.com>; giamteckchoon@gmail.com<mailto:giamteckchoon@gmail.com>; konrad.wilk@oracle.com<mailto:konrad.wilk@oracle.com>
> Date: Fri, 29 Apr 2011 08:19:44 +0800
> Subject: RE: [Xen-devel] RE: Kernel BUG at arch/x86/mm/tlb.c:61
>
> > From: Jeremy Fitzhardinge [mailto:jeremy@goop.org]<mailto:[mailto:jeremy@goop.org]>
> > Sent: Friday, April 29, 2011 7:29 AM
> >
> > On 04/25/2011 10:52 PM, Tian, Kevin wrote:
> > >> From: MaoXiaoyun
> > >> Sent: Monday, April 25, 2011 11:15 AM
> > >>> Date: Fri, 15 Apr 2011 14:22:29 -0700
> > >>> From: jeremy@goop.org<mailto:jeremy@goop.org>
> > >>> To: tinnycloud@hotmail.com<mailto:tinnycloud@hotmail.com>
> > >>> CC: giamteckchoon@gmail.com<mailto:giamteckchoon@gmail.com>; xen-devel@lists.xensource.com<mailto:xen-devel@lists.xensource.com>;
> > >>> konrad.wilk@oracle.com<mailto:konrad.wilk@oracle.com>
> > >>> Subject: Re: Kernel BUG at arch/x86/mm/tlb.c:61
> > >>>
> > >>> On 04/15/2011 05:23 AM, MaoXiaoyun wrote:
> > >>>> Hi：
> > >>>>
> > >>>> Could the crash related to this patch ?
> > >>>> http://git.kernel.org/?p=linux/kernel/git/jeremy/xen.git;a=commitdi
> > >>>> ff;h=45bfd7bfc6cf32f8e60bb91b32349f0b5090eea3
> > >>>>
> > >>>> Since now TLB state change to TLBSTATE_OK(mmu_context.h:40) is
> > >>>> before cpumask_clear_cpu(line 49).
> > >>>> Could it possible that right after execute line 40 of
> > >>>> mmu_context.h, CPU revice IPI from other CPU to flush the mm, and
> > >>>> when in interrupt, find the TLB state happened to be TLBSTATE_OK.
> > >>>> Which conflicts.
> > >>> Does reverting it help?
> > >>>
> > >>> J
> > >>
> > >> Hi Jeremy:
> > >>
> > >> The lastest test result shows the reverting didn't help.
> > >> Kernel panic exactly at the same place in tlb.c.
> > >>
> > >> I have question about TLB state, from the stack,
> > >> xen_do_hypervisor_callback-> xen_evtchn_do_upcall->...
> > >> ->drop_other_mm_ref
> > >>
> > >> What cpu_tlbstate.state should be, could TLBSTATE_OK or
> > TLBSTATE_LAZY all be possible?
> > >> That is after a hypercall from userspace, state will be TLBSTATE_OK,
> > and
> > >> if from kernel space, state will be TLBSTATE_LAZE ?
> > >>
> > >> thanks.
> > > it looks a bug in drop_other_mm_ref implementation, that current TLB
> > > state should be checked before invoking leave_mm(). There's a window
> > between below lines of code:
> > >
> > > <xen_drop_mm_ref>
> > > /* Get the "official" set of cpus referring to our pagetable. */
> > > if (!alloc_cpumask_var(&mask, GFP_ATOMIC)) {
> > > for_each_online_cpu(cpu) {
> > > if (!cpumask_test_cpu(cpu,
> > mm_cpumask(mm))
> > > && per_cpu(xen_current_cr3, cpu) !=
> > __pa(mm->pgd))
> > > continue;
> > > smp_call_function_single(cpu,
> > drop_other_mm_ref, mm, 1);
> > > }
> > > return;
> > > }
> > >
> > > there's chance that when smp_call_function_single is invoked, actual
> > > TLB state has been updated in the other cpu. The upstream kernel patch
> > > you referred to earlier just makes this bug exposed more easily. But
> > > even without this patch, you may still suffer such issue which is why reverting
> > the patch doesn't help.
> > >
> > > Could you try adding a check in drop_other_mm_ref?
> > >
> > > if (active_mm == mm && percpu_read(cpu_tlbstate.state) !=
> > TLBSTATE_OK)
> > > leave_mm(smp_processor_id());
> > >
> > > once the interrupted context has TLBSTATE_OK, it implicates that later
> > > it will handle the TLB flush and thus no need for leave_mm from
> > > interrupt handler, and that's the assumption of doing leave_mm.
> >
> > That seems reasonable. MaoXiaoyun, does it fix the bug for you?
> >
> > Kevin, could you submit this as a proper patch?
> >
>
> I'm waiting for Xiaoyun's test result before submitting a proper patch, since this
> part of logic is tricky and his test can make sure we don't overlook some corner
> cases. :-)
>

I think it works. The test has been running over 70 hours successfully.
My plan is run one week.

Thanks.

> Thanks
> Kevin

[-- Attachment #1.2: Type: text/html, Size: 9551 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

RE: Kernel BUG at arch/x86/mm/tlb.c:61

[MaoXiaoyun]

[-- Attachment #1.1: Type: text/plain, Size: 2970 bytes --]


I go through the switch_mm more, and come up one more question:
 
Why we don't need to clear prev cpumask in line between line 59 and 60?
 
Say
1)  Context is switch from process A to kernel, then kernel has active_mm-> A's mm
2)  Context is switch from kernel to A, in sched.c oldmm = A's mm; mm = A's mm
3)  it will call arch/x86/include/asm/mmu_context.h:60, since prev = next;
     if another CPU flush A's mm, but this cpu don't clear CPU mask, it might enter IPI interrput
     routine, and also find cpu_tlbstate.state is TLBSTATE_OK.
 
Could this possible?
 
kernel/sched.c
 
2999 context_switch(struct rq *rq, struct task_struct *prev,
 3000            struct task_struct *next)
 3001 {
 3002     struct mm_struct *mm, *oldmm;
 3003 
 3004     prepare_task_switch(rq, prev, next);
 3005     trace_sched_switch(rq, prev, next);
 3006     mm = next->mm;
 3007     oldmm = prev->active_mm;
 3008     /*
 3009      * For paravirt, this is coupled with an exit in switch_to to
 3010      * combine the page table reload and the switch backend into
 3011      * one hypercall.
 3012      */
 3013     arch_start_context_switch(prev);
 3014 
 3015     if (unlikely(!mm)) {
 3016         next->active_mm = oldmm;
 3017         atomic_inc(&oldmm->mm_count);
 3018         enter_lazy_tlb(oldmm, next);
 3019     } else
 3020         switch_mm(oldmm, mm, next);
 3021 
 3022     if (unlikely(!prev->mm)) {
 3023         prev->active_mm = NULL;
 3024         rq->prev_mm = oldmm;
 3025     }
 
 
 33 static inline void switch_mm(struct mm_struct *prev, struct mm_struct *next,
 34                  struct task_struct *tsk)
 35 {
 36     unsigned cpu = smp_processor_id();
 37 
 38     if (likely(prev != next)) {
 39         /* stop flush ipis for the previous mm */
 40         cpumask_clear_cpu(cpu, mm_cpumask(prev));
 41 
 42 
 43 #ifdef CONFIG_SMP
 44         percpu_write(cpu_tlbstate.state, TLBSTATE_OK);
 45         percpu_write(cpu_tlbstate.active_mm, next);
 46 #endif
 47         cpumask_set_cpu(cpu, mm_cpumask(next));
 48 
 49         /* Re-load page tables */
 50         load_cr3(next->pgd);
 51 
 52         /*
 53          * load the LDT, if the LDT is different:
 54          */
 55         if (unlikely(prev->context.ldt != next->context.ldt))
 56             load_LDT_nolock(&next->context);
 57     }
 58 #ifdef CONFIG_SMP
 59     else {
 60         percpu_write(cpu_tlbstate.state, TLBSTATE_OK);
 61         BUG_ON(percpu_read(cpu_tlbstate.active_mm) != next);
 62 
 63         if (!cpumask_test_and_set_cpu(cpu, mm_cpumask(next))) {
 64             /* We were in lazy tlb mode and leave_mm disabled
 65              * tlb flush IPI delivery. We must reload CR3
 66              * to make sure to use no freed page tables.
 67              */
 68             load_cr3(next->pgd);
 69             load_LDT_nolock(&next->context);
 70         }
 71     } 
 		 	   		  

[-- Attachment #1.2: Type: text/html, Size: 6837 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

RE: Kernel BUG at arch/x86/mm/tlb.c:61

[MaoXiaoyun]

[-- Attachment #1.1: Type: text/plain, Size: 7257 bytes --]


Add some debug info in drop_other_mm_ref(line 1516), get on machine crash.
log attached, pity I lost prink info.
 
Does current->mm indicates userspace?
Thanks.
 
============================
1502 #ifdef CONFIG_SMP
1503 /* Another cpu may still have their %cr3 pointing at the pagetable, so
1504    we need to repoint it somewhere else before we can unpin it. */
1505 static void drop_other_mm_ref(void *info)
1506 {
1507 <+++struct mm_struct *mm = info;
1508 <+++struct mm_struct *active_mm;
1509 
1510 <+++active_mm = percpu_read(cpu_tlbstate.active_mm);
1511 
1512 <+++if (active_mm == mm){
1513         if(current->mm){
1514 <+++<+++    printk("in userspace active_mm %p mm %p curr_mm %p tlbstate%d\n",                                                                           
1515                    active_mm, mm, current->mm, percpu_read(cpu_tlbstate.state));
1516             BUG();
1517         }
1518 <+++<+++leave_mm(smp_processor_id());
1519     }
1520 
 
 
============================
 
Starting udev: ------------[ cut here ]------------
kernel BUG at arch/x86/xen/mmu.c:1516!
invalid opcode: 0000 [#1] SMP 
last sysfs file: /sys/class/raw/rawctl/dev
CPU 2 
Modules linked in: snd_seq_dummy bnx2 snd_seq_oss(+) snd_seq_midi_event snd_seq 
snd_seq_device serio_raw snd_pcm_oss snd_mixer_oss snd_pcm snd_timer i2c_i801 i2c_core iTCO_wdt snd pata_acpi iTCO_vendor_support ata_generic soundcore 
snd_page_alloc pcspkr ata_piix shpchp mptsas mptscsih mptbase                           
Pid: 1126, comm: khelper Not tainted 2.6.32.36xen #1 Tecal RH2285          
RIP: e030:[<ffffffff8100e4c0>]  [<ffffffff8100e4c0>] drop_other_mm_ref+0x46/0x80
RSP: e02b:ffff880028078e58  EFLAGS: 00010092
RAX: 0000000000000015 RBX: 0000000000000001 RCX: 00000000ffff0075
RDX: 0000000000009f9f RSI: ffffffff8144006a RDI: 0000000000000004
RBP: ffff880028078e68 R08: 0000000000000001 R09: 0000000000000000
R10: 0000000028078cf8 R11: 0000000000000246 R12: ffff88012c032680
R13: ffff880028080020 R14: 00000000000004f1 R15: 0000000000000000
FS:  00007f01adcf8710(0000) GS:ffff880028075000(0000) knlGS:0000000000000000
CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007f01adf20648 CR3: 000000012a546000 CR4: 0000000000002660
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process khelper (pid: 1126, threadinfo ffff88012d80e000, task ffff88012b880000)
Stack:
 0000000000000001 ffff88012bb9bb88 ffff880028078e98 ffffffff81087224
<0> ffff880028078e78 ffff880028078e78 ffff88015f808540 00000000000004f1
<0> ffff880028078ea8 ffffffff81010118 ffff880028078ee8 ffffffff810a936a
Call Trace:
 <IRQ> 
 [<ffffffff81087224>] generic_smp_call_function_single_interrupt+0xd8/0xfc
 [<ffffffff81010118>] xen_call_function_single_interrupt+0x13/0x28
 [<ffffffff810a936a>] handle_IRQ_event+0x66/0x120
 [<ffffffff810aac5b>] handle_percpu_irq+0x41/0x6e
 [<ffffffff8128c1a8>] __xen_evtchn_do_upcall+0x1ab/0x27d
 [<ffffffff8128dcf9>] xen_evtchn_do_upcall+0x33/0x46
 [<ffffffff81013efe>] xen_do_hypervisor_callback+0x1e/0x30
 <EOI> 
 [<ffffffff8100f8df>] ? xen_restore_fl_direct_end+0x0/0x1
 [<ffffffff8100922a>] ? hypercall_page+0x22a/0x1000
 [<ffffffff8100922a>] ? hypercall_page+0x22a/0x1000
 [<ffffffff81447292>] ? _spin_unlock_irqrestore+0x15/0x17
 [<ffffffff8100f195>] ? xen_force_evtchn_callback+0xd/0xf
 [<ffffffff8100f8f2>] ? check_events+0x12/0x20
 [<ffffffff81447292>] ? _spin_unlock_irqrestore+0x15/0x17
 [<ffffffff8100f8df>] ? xen_restore_fl_direct_end+0x0/0x1
 [<ffffffff8100f8df>] ? xen_restore_fl_direct_end+0x0/0x1
 [<ffffffff8100d47f>] ? xen_mc_issue+0x2e/0x33
 [<ffffffff8100e42f>] ? __xen_pgd_pin+0xc1/0xc9
 [<ffffffff8100e449>] ? xen_pgd_pin+0x12/0x14
 [<ffffffff8100e470>] ? xen_activate_mm+0x25/0x2f
 [<ffffffff81113f59>] ? flush_old_exec+0x390/0x500
 [<ffffffff81150dc9>] ? load_elf_binary+0x0/0x17ef
 [<ffffffff81150dc9>] ? load_elf_binary+0x0/0x17ef
 [<ffffffff81151161>] ? load_elf_binary+0x398/0x17ef
 [<ffffffff81042fcf>] ? need_resched+0x23/0x2d
 [<ffffffff811f463c>] ? process_measurement+0xc0/0xd7
 [<ffffffff81150dc9>] ? load_elf_binary+0x0/0x17ef
 [<ffffffff81113098>] ? search_binary_handler+0xc8/0x255
 [<ffffffff81114366>] ? do_execve+0x1c3/0x29e
 [<ffffffff8101155d>] ? sys_execve+0x43/0x5d
 [<ffffffff8106fc45>] ? __call_usermodehelper+0x0/0x6f
 [<ffffffff81013e28>] ? kernel_execve+0x68/0xd0
 [<ffffffff8106fc45>] ? __call_usermodehelper+0x0/0x6f
 [<ffffffff8100f8df>] ? xen_restore_fl_direct_end+0x0/0x1
 [<ffffffff8106fb64>] ? ____call_usermodehelper+0x113/0x11e
 [<ffffffff81013daa>] ? child_rip+0xa/0x20
 [<ffffffff8106fc45>] ? __call_usermodehelper+0x0/0x6f
 [<ffffffff81012f91>] ? int_ret_from_sys_call+0x7/0x1b
 [<ffffffff8101371d>] ? retint_restore_args+0x5/0x6
 [<ffffffff81013da0>] ? child_rip+0x0/0x20
Code: 75 3a 65 48 8b 04 25 c0 cb 00 00 48 83 b8 78 02 00 00 00 74 1a 65 8b 34 25 c8 55 01 00 48 c7 c7 06 98 5b 81 31 c0 e8 d9 90 04 00 <0f> 0b eb fe 65 8b 3c 
25 78 e3 00 00 e8 e5 be 02 00 65 48 8b 1c                                         
RIP  [<ffffffff8100e4c0>] drop_other_mm_ref+0x46/0x80
 RSP <ffff880028078e58>
[<ffffffff8144006a>] ? init_amd+0x296/0x37a
 [<ffffffff8100f195>] ? xen_force_evtchn_callback+0xd/0xf
 [<ffffffff8100f8f2>] ? check_events+0x12/0x20
 [<ffffffff81056487>] ? print_oops_end_marker+0x23/0x25
 [<ffffffff81448165>] oops_end+0xb6/0xc6
 [<ffffffff810166e5>] die+0x5a/0x63
 [<ffffffff81447a3c>] do_trap+0x115/0x124
 [<ffffffff810148e6>] do_invalid_op+0x9c/0xa5
 [<ffffffff8100e4c0>] ? drop_other_mm_ref+0x46/0x80
 [<ffffffff81057640>] ? printk+0xa7/0xa9
 [<ffffffff81013b3b>] invalid_op+0x1b/0x20
 [<ffffffff8144006a>] ? init_amd+0x296/0x37a
 [<ffffffff8100e4c0>] ? drop_other_mm_ref+0x46/0x80
 [<ffffffff8100e4c0>] ? drop_other_mm_ref+0x46/0x80
 [<ffffffff81087224>] generic_smp_call_function_single_interrupt+0xd8/0xfc
 [<ffffffff81010118>] xen_call_function_single_interrupt+0x13/0x28
 [<ffffffff810a936a>] handle_IRQ_event+0x66/0x120
 [<ffffffff810aac5b>] handle_percpu_irq+0x41/0x6e
 [<ffffffff8128c1a8>] __xen_evtchn_do_upcall+0x1ab/0x27d
 [<ffffffff8128dcf9>] xen_evtchn_do_upcall+0x33/0x46
 [<ffffffff81013efe>] xen_do_hypervisor_callback+0x1e/0x30
 <EOI>  [<ffffffff8100f8df>] ? xen_restore_fl_direct_end+0x0/0x1
 [<ffffffff8100922a>] ? hypercall_page+0x22a/0x1000
 [<ffffffff8100922a>] ? hypercall_page+0x22a/0x1000
 [<ffffffff81447292>] ? _spin_unlock_irqrestore+0x15/0x17
 [<ffffffff8100f195>] ? xen_force_evtchn_callback+0xd/0xf
 [<ffffffff8100f8f2>] ? check_events+0x12/0x20
 [<ffffffff81447292>] ? _spin_unlock_irqrestore+0x15/0x17
 [<ffffffff8100f8df>] ? xen_restore_fl_direct_end+0x0/0x1
 [<ffffffff8100f8df>] ? xen_restore_fl_direct_end+0x0/0x1
 [<ffffffff8100d47f>] ? xen_mc_issue+0x2e/0x33
 [<ffffffff8100e42f>] ? __xen_pgd_pin+0xc1/0xc9
 [<ffffffff8100e449>] ? xen_pgd_pin+0x12/0x14
 [<ffffffff8100e470>] ? xen_activate_mm+0x25/0x2f
 [<ffffffff81113f59>] ? flush_old_exec+0x390/0x500
 [<ffffffff81150dc9>] ? load_elf_binary+0x0/0x17ef
 [<ffffffff81150dc9>] ? load_elf_binary+0x0/0x17ef
 [<ffffffff81151161>] ? load_elf_binary+0x398/0x17ef
 		 	   		  

[-- Attachment #1.2: Type: text/html, Size: 9845 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

RE: Kernel BUG at arch/x86/mm/tlb.c:61

[MaoXiaoyun]

[-- Attachment #1.1: Type: text/plain, Size: 541 bytes --]


 
>From: tinnycloud@hotmail.com
>To: jeremy@goop.org
>CC: giamteckchoon@gmail.com; xen-devel@lists.xensource.com; konrad.wilk@oracle.com
>Subject: RE: Kernel BUG at arch/x86/mm/tlb.c:61
>Date: Mon, 25 Apr 2011 20:54:54 +0800




>Add some debug info in drop_other_mm_ref(line 1516), get on machine crash.
>log attached, pity I lost prink info.

printk info: in userspace active_mm ffff8800a3669f80 mm ffff8800a3669f80 curr_mm ffff88008d73c000 tlbstate 2  

>Does current->mm indicates userspace?
>Thanks.
 

 		 	   		  

[-- Attachment #1.2: Type: text/html, Size: 1026 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

RE: Kernel BUG at arch/x86/mm/tlb.c:61

[MaoXiaoyun]

[-- Attachment #1.1: Type: text/plain, Size: 715 bytes --]


Please ignore my last two mails, I just learnt that Current is meanless in irq context.
 
Just come up one whole assumption:
 
In my opinion:
 
1) CPU running in switch_mm has the possiblity of receiving IPI message and enter interrupt
2) Before revert that patch, not matter the if statement is true or not, the cpu_tlbstate.state
could be changed to TLBSTATE_OK, right before enter irq routhine
3) Since the cpu_tlbstate is per CPU variable, before calling leave_mm(), test cpu_tlbstate.state
in drop_other_mm_ref is feasible and nessary
4) If I am right, strange thing is the code of 2.6.32.36 is same as 2.6.31.x, which we never met tlb bug before.
 
any comments?
 
Many thanks.
  		 	   		  

[-- Attachment #1.2: Type: text/html, Size: 1017 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

RE: RE: Kernel BUG at arch/x86/mm/tlb.c:61

[Tian, Kevin]

[-- Attachment #1.1: Type: text/plain, Size: 1232 bytes --]

the race window is always there, but whether it will be triggered is not determined. It's possible that you never met this bug on 2.6.31.x now, but it doesn't mean you won't meet it in long run in the future. :)

Thanks
Kevin

From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of MaoXiaoyun
Sent: Monday, April 25, 2011 11:05 PM
To: jeremy@goop.org
Cc: xen devel; giamteckchoon@gmail.com; konrad.wilk@oracle.com
Subject: [Xen-devel] RE: Kernel BUG at arch/x86/mm/tlb.c:61

Please ignore my last two mails, I just learnt that Current is meanless in irq context.

Just come up one whole assumption:

In my opinion:

1) CPU running in switch_mm has the possiblity of receiving IPI message and enter interrupt
2) Before revert that patch, not matter the if statement is true or not, the cpu_tlbstate.state
could be changed to TLBSTATE_OK, right before enter irq routhine
3) Since the cpu_tlbstate is per CPU variable, before calling leave_mm(), test cpu_tlbstate.state
in drop_other_mm_ref is feasible and nessary
4) If I am right, strange thing is the code of 2.6.32.36 is same as 2.6.31.x, which we never met tlb bug before.

any comments?

Many thanks.


[-- Attachment #1.2: Type: text/html, Size: 4985 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: kernel BUG at arch/x86/xen/mmu.c:1872

[Teck Choon Giam]

2011/4/12 Jeremy Fitzhardinge <jeremy@goop.org>:
> On 04/11/2011 05:31 AM, MaoXiaoyun wrote:
>> Hi:
>>
>> I believe this is the fix at much extent.
>> Since I have my own test cases which with this patch, my test case
>> will success in 30 rounds run.
>> Every round takes 8hours. While without this patch, tests fail evey
>> round in 15minutes.
>>
>> So this really means fix most of the things.
>>
>> But during running, I met another crash, from the log it it looks like
>> has relation with
>> this BUG, since the crash log shows it is tlb related and this BUG
>> also tlb related.
>>
>> Well, I'm also have poor knowledge of kernel.
>> Hope someone from Xen Devel offer some help.
>
> Thanks for confirming; it makes sense and explains the symptoms, so I'm
> glad it also works ;)
>
>
> J
>

Thanks Jeremy, I can see the needed backport patch is in your
xen/next-2.6.32 tree now ;)

Kindest regards,
Giam Teck Choon

Re: kernel BUG at arch/x86/xen/mmu.c:1860!

[Joerg Stephan]

Hey guys,

i just wanne report that our debian squeeze xen machines are also 
influenced by this bug, and we tried the workaround with the debug option 
by building our own

linux-image-2.6.32-5-xen-amd64_2.6.32-31_amd64.deb

but this isnt a workaround really, we tested it on an 40 vmachines dom0 
and it just dont work, of course the lvm works but the machine is slow as 
hell and runs into several kernel errors within a few hours

[46566.052925] BUG: soft lockup - CPU#0 stuck for 61s! [qemu-dm:6292]

in this case xentop shows an 500% CPU usage by the dom0, we experience an 
85%st value.

So, i think this should be fixed soon, so what is the status of the issue?

Best whishes

Joerg

kernel BUG at arch/x86/xen/mmu.c:1860!

[Teck Choon Giam]

[-- Attachment #1.1: Type: text/plain, Size: 4396 bytes --]

Hi,

Information: CentOS 5.5 x86_64
dom0: latest xen/stable-2.6.32.x pvops git commit
75cc13f5aa29b4f3227d269ca165dfa8937c94fe
xen version: xen-4.0.2-rc1-pre from xen-4.0-testing.hg changeset 21422

While doing LVM snapshot for migration and get the following:

Dec 26 15:58:29 xen01 kernel: ------------[ cut here ]------------
Dec 26 15:58:29 xen01 kernel: kernel BUG at arch/x86/xen/mmu.c:1860!
Dec 26 15:58:29 xen01 kernel: invalid opcode: 0000 [#1] SMP
Dec 26 15:58:29 xen01 kernel: last sysfs file: /sys/block/dm-26/dev
Dec 26 15:58:29 xen01 kernel: CPU 0
Dec 26 15:58:29 xen01 kernel: Modules linked in: ipt_MASQUERADE iptable_nat
nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJECT
xt_tcpudp xt_physdev iptable_filter ip_tables x_tables bridge stp be2iscsi
iscsi_tcp bnx2i cnic uio ipv6 cxgb3i cxgb3 mdio libiscsi_tcp libiscsi
scsi_transport_iscsi loop dm_multipath scsi_dh video backlight output sbs
sbshc power_meter hwmon battery acpi_memhotplug xen_acpi_memhotplug ac
parport_pc lp parport sg tpm_tis tpm tpm_bios button i2c_i801 i2c_core
iTCO_wdt e1000e shpchp pcspkr dm_snapshot dm_zero dm_mirror dm_region_hash
dm_log dm_mod usb_storage ahci libata sd_mod scsi_mod raid1 ext3 jbd
uhci_hcd ohci_hcd ehci_hcd [last unloaded: microcode]
Dec 26 15:58:29 xen01 kernel: Pid: 27998, comm: udevd Not tainted
2.6.32.27-0.xen.pvops.choon.centos5 #1 S3420GP
Dec 26 15:58:29 xen01 kernel: RIP: e030:[<ffffffff8100cb5b>]
[<ffffffff8100cb5b>] pin_pagetable_pfn+0x53/0x59
Dec 26 15:58:29 xen01 kernel: RSP: e02b:ffff88003bc3bdc8  EFLAGS: 00010282
Dec 26 15:58:29 xen01 kernel: RAX: 00000000ffffffea RBX: 0000000000017605
RCX: 00000000000000bb
Dec 26 15:58:29 xen01 kernel: RDX: 00000000deadbeef RSI: 00000000deadbeef
RDI: 00000000deadbeef
Dec 26 15:58:29 xen01 kernel: RBP: ffff88003bc3bde8 R08: 0000000000000028
R09: ffff880000000000
Dec 26 15:58:29 xen01 kernel: R10: 00000000deadbeef R11: 00007fdb5665e600
R12: 0000000000000003
Dec 26 15:58:30 xen01 kernel: R13: 0000000000017605 R14: ffff880012ee0780
R15: 00007fdb56224268
Dec 26 15:58:30 xen01 kernel: FS:  00007fdb56fed710(0000)
GS:ffff88002804f000(0000) knlGS:0000000000000000
Dec 26 15:58:30 xen01 kernel: CS:  e033 DS: 0000 ES: 0000 CR0:
000000008005003b
Dec 26 15:58:30 xen01 kernel: CR2: 00007fdb56224268 CR3: 000000003addb000
CR4: 0000000000002660
Dec 26 15:58:30 xen01 kernel: DR0: 0000000000000000 DR1: 0000000000000000
DR2: 0000000000000000
Dec 26 15:58:30 xen01 kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0
DR7: 0000000000000400
Dec 26 15:58:30 xen01 kernel: Process udevd (pid: 27998, threadinfo
ffff88003bc3a000, task ffff880012ee0780)
Dec 26 15:58:30 xen01 kernel: Stack:
Dec 26 15:58:30 xen01 kernel:  0000000000000000 0000000000424121
000000013f00ae20 0000000000017605
Dec 26 15:58:30 xen01 kernel: <0> ffff88003bc3be08 ffffffff8100e07c
ffff88003a3c2580 ffff880034bb6588
Dec 26 15:58:30 xen01 kernel: <0> ffff88003bc3be18 ffffffff8100e0af
ffff88003bc3be58 ffffffff810a402f
Dec 26 15:58:31 xen01 kernel: Call Trace:
Dec 26 15:58:31 xen01 kernel:  [<ffffffff8100e07c>]
xen_alloc_ptpage+0x64/0x69
Dec 26 15:58:31 xen01 kernel:  [<ffffffff8100e0af>] xen_alloc_pte+0xe/0x10
Dec 26 15:58:31 xen01 kernel:  [<ffffffff810a402f>] __pte_alloc+0x70/0xce
Dec 26 15:58:31 xen01 kernel:  [<ffffffff810a41cd>]
handle_mm_fault+0x140/0x8b9
Dec 26 15:58:31 xen01 kernel:  [<ffffffff810d2ecc>] ? d_kill+0x3a/0x42
Dec 26 15:58:31 xen01 kernel:  [<ffffffff810c4cd1>] ? __fput+0x1cb/0x1da
Dec 26 15:58:31 xen01 kernel:  [<ffffffff8131be4d>]
do_page_fault+0x252/0x2e2
Dec 26 15:58:31 xen01 kernel:  [<ffffffff81319dd5>] page_fault+0x25/0x30
Dec 26 15:58:31 xen01 kernel: Code: 48 b8 ff ff ff ff ff ff ff 7f 48 21 c2
48 89 55 e8 48 8d 7d e0 be 01 00 00 00 31 d2 41 ba f0 7f 00 00 e8 e9 c7 ff
ff 85 c0 74 04 <0f> 0b eb fe c9 c3 55 40 f6 c7 01 48 89 e5 53 48 89 fb 74 5b
48
Dec 26 15:58:31 xen01 kernel: RIP  [<ffffffff8100cb5b>]
pin_pagetable_pfn+0x53/0x59
Dec 26 15:58:31 xen01 kernel:  RSP <ffff88003bc3bdc8>
Dec 26 15:58:31 xen01 kernel: ---[ end trace 540bcf6f0170242d ]---

Triggered BUG() in line 1860:

static void pin_pagetable_pfn(unsigned cmd, unsigned long pfn)
{
        struct mmuext_op op;
        op.cmd = cmd;
        op.arg1.mfn = pfn_to_mfn(pfn);
           if (HYPERVISOR_mmuext_op(&op, 1, NULL, DOMID_SELF))
                   BUG(); <<THIS ONE?
}

Any idea?

Thanks.

Kindest regards,
Giam Teck Choon

[-- Attachment #1.2: Type: text/html, Size: 4731 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: kernel BUG at arch/x86/xen/mmu.c:1860!

[Konrad Rzeszutek Wilk]

On Sun, Dec 26, 2010 at 04:16:16PM +0800, Teck Choon Giam wrote:
> Hi,
> 
> Information: CentOS 5.5 x86_64
> dom0: latest xen/stable-2.6.32.x pvops git commit
> 75cc13f5aa29b4f3227d269ca165dfa8937c94fe
> xen version: xen-4.0.2-rc1-pre from xen-4.0-testing.hg changeset 21422
> 
> While doing LVM snapshot for migration and get the following:
> 
> Dec 26 15:58:29 xen01 kernel: ------------[ cut here ]------------
> Dec 26 15:58:29 xen01 kernel: kernel BUG at arch/x86/xen/mmu.c:1860!
> Dec 26 15:58:29 xen01 kernel: invalid opcode: 0000 [#1] SMP
> Dec 26 15:58:29 xen01 kernel: last sysfs file: /sys/block/dm-26/dev
> Dec 26 15:58:29 xen01 kernel: CPU 0
> Dec 26 15:58:29 xen01 kernel: Modules linked in: ipt_MASQUERADE iptable_nat
> nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJECT
> xt_tcpudp xt_physdev iptable_filter ip_tables x_tables bridge stp be2iscsi
> iscsi_tcp bnx2i cnic uio ipv6 cxgb3i cxgb3 mdio libiscsi_tcp libiscsi
> scsi_transport_iscsi loop dm_multipath scsi_dh video backlight output sbs
> sbshc power_meter hwmon battery acpi_memhotplug xen_acpi_memhotplug ac
> parport_pc lp parport sg tpm_tis tpm tpm_bios button i2c_i801 i2c_core
> iTCO_wdt e1000e shpchp pcspkr dm_snapshot dm_zero dm_mirror dm_region_hash
> dm_log dm_mod usb_storage ahci libata sd_mod scsi_mod raid1 ext3 jbd
> uhci_hcd ohci_hcd ehci_hcd [last unloaded: microcode]
> Dec 26 15:58:29 xen01 kernel: Pid: 27998, comm: udevd Not tainted
> 2.6.32.27-0.xen.pvops.choon.centos5 #1 S3420GP
> Dec 26 15:58:29 xen01 kernel: RIP: e030:[<ffffffff8100cb5b>]
> [<ffffffff8100cb5b>] pin_pagetable_pfn+0x53/0x59
> Dec 26 15:58:29 xen01 kernel: RSP: e02b:ffff88003bc3bdc8  EFLAGS: 00010282
> Dec 26 15:58:29 xen01 kernel: RAX: 00000000ffffffea RBX: 0000000000017605
> RCX: 00000000000000bb
> Dec 26 15:58:29 xen01 kernel: RDX: 00000000deadbeef RSI: 00000000deadbeef
> RDI: 00000000deadbeef
> Dec 26 15:58:29 xen01 kernel: RBP: ffff88003bc3bde8 R08: 0000000000000028
> R09: ffff880000000000
> Dec 26 15:58:29 xen01 kernel: R10: 00000000deadbeef R11: 00007fdb5665e600
> R12: 0000000000000003
> Dec 26 15:58:30 xen01 kernel: R13: 0000000000017605 R14: ffff880012ee0780
> R15: 00007fdb56224268
> Dec 26 15:58:30 xen01 kernel: FS:  00007fdb56fed710(0000)
> GS:ffff88002804f000(0000) knlGS:0000000000000000
> Dec 26 15:58:30 xen01 kernel: CS:  e033 DS: 0000 ES: 0000 CR0:
> 000000008005003b
> Dec 26 15:58:30 xen01 kernel: CR2: 00007fdb56224268 CR3: 000000003addb000
> CR4: 0000000000002660
> Dec 26 15:58:30 xen01 kernel: DR0: 0000000000000000 DR1: 0000000000000000
> DR2: 0000000000000000
> Dec 26 15:58:30 xen01 kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0
> DR7: 0000000000000400
> Dec 26 15:58:30 xen01 kernel: Process udevd (pid: 27998, threadinfo
> ffff88003bc3a000, task ffff880012ee0780)
> Dec 26 15:58:30 xen01 kernel: Stack:
> Dec 26 15:58:30 xen01 kernel:  0000000000000000 0000000000424121
> 000000013f00ae20 0000000000017605
> Dec 26 15:58:30 xen01 kernel: <0> ffff88003bc3be08 ffffffff8100e07c
> ffff88003a3c2580 ffff880034bb6588
> Dec 26 15:58:30 xen01 kernel: <0> ffff88003bc3be18 ffffffff8100e0af
> ffff88003bc3be58 ffffffff810a402f
> Dec 26 15:58:31 xen01 kernel: Call Trace:
> Dec 26 15:58:31 xen01 kernel:  [<ffffffff8100e07c>]
> xen_alloc_ptpage+0x64/0x69
> Dec 26 15:58:31 xen01 kernel:  [<ffffffff8100e0af>] xen_alloc_pte+0xe/0x10
> Dec 26 15:58:31 xen01 kernel:  [<ffffffff810a402f>] __pte_alloc+0x70/0xce
> Dec 26 15:58:31 xen01 kernel:  [<ffffffff810a41cd>]
> handle_mm_fault+0x140/0x8b9
> Dec 26 15:58:31 xen01 kernel:  [<ffffffff810d2ecc>] ? d_kill+0x3a/0x42
> Dec 26 15:58:31 xen01 kernel:  [<ffffffff810c4cd1>] ? __fput+0x1cb/0x1da
> Dec 26 15:58:31 xen01 kernel:  [<ffffffff8131be4d>]
> do_page_fault+0x252/0x2e2
> Dec 26 15:58:31 xen01 kernel:  [<ffffffff81319dd5>] page_fault+0x25/0x30
> Dec 26 15:58:31 xen01 kernel: Code: 48 b8 ff ff ff ff ff ff ff 7f 48 21 c2
> 48 89 55 e8 48 8d 7d e0 be 01 00 00 00 31 d2 41 ba f0 7f 00 00 e8 e9 c7 ff
> ff 85 c0 74 04 <0f> 0b eb fe c9 c3 55 40 f6 c7 01 48 89 e5 53 48 89 fb 74 5b
> 48
> Dec 26 15:58:31 xen01 kernel: RIP  [<ffffffff8100cb5b>]
> pin_pagetable_pfn+0x53/0x59
> Dec 26 15:58:31 xen01 kernel:  RSP <ffff88003bc3bdc8>
> Dec 26 15:58:31 xen01 kernel: ---[ end trace 540bcf6f0170242d ]---
> 
> Triggered BUG() in line 1860:
> 
> static void pin_pagetable_pfn(unsigned cmd, unsigned long pfn)
> {
>         struct mmuext_op op;
>         op.cmd = cmd;
>         op.arg1.mfn = pfn_to_mfn(pfn);
>            if (HYPERVISOR_mmuext_op(&op, 1, NULL, DOMID_SELF))
>                    BUG(); <<THIS ONE?

Yup.
> }
> 
> Any idea?

Do you get to see this every time you do LVM migrate?
> 
> Thanks.
> 
> Kindest regards,
> Giam Teck Choon

> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: kernel BUG at arch/x86/xen/mmu.c:1860!

[Teck Choon Giam]

[-- Attachment #1.1: Type: text/plain, Size: 5740 bytes --]

On Mon, Dec 27, 2010 at 11:53 PM, Konrad Rzeszutek Wilk <
konrad.wilk@oracle.com> wrote:

> On Sun, Dec 26, 2010 at 04:16:16PM +0800, Teck Choon Giam wrote:
> > Hi,
> >
> > Information: CentOS 5.5 x86_64
> > dom0: latest xen/stable-2.6.32.x pvops git commit
> > 75cc13f5aa29b4f3227d269ca165dfa8937c94fe
> > xen version: xen-4.0.2-rc1-pre from xen-4.0-testing.hg changeset 21422
> >
> > While doing LVM snapshot for migration and get the following:
> >
> > Dec 26 15:58:29 xen01 kernel: ------------[ cut here ]------------
> > Dec 26 15:58:29 xen01 kernel: kernel BUG at arch/x86/xen/mmu.c:1860!
> > Dec 26 15:58:29 xen01 kernel: invalid opcode: 0000 [#1] SMP
> > Dec 26 15:58:29 xen01 kernel: last sysfs file: /sys/block/dm-26/dev
> > Dec 26 15:58:29 xen01 kernel: CPU 0
> > Dec 26 15:58:29 xen01 kernel: Modules linked in: ipt_MASQUERADE
> iptable_nat
> > nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJECT
> > xt_tcpudp xt_physdev iptable_filter ip_tables x_tables bridge stp
> be2iscsi
> > iscsi_tcp bnx2i cnic uio ipv6 cxgb3i cxgb3 mdio libiscsi_tcp libiscsi
> > scsi_transport_iscsi loop dm_multipath scsi_dh video backlight output sbs
> > sbshc power_meter hwmon battery acpi_memhotplug xen_acpi_memhotplug ac
> > parport_pc lp parport sg tpm_tis tpm tpm_bios button i2c_i801 i2c_core
> > iTCO_wdt e1000e shpchp pcspkr dm_snapshot dm_zero dm_mirror
> dm_region_hash
> > dm_log dm_mod usb_storage ahci libata sd_mod scsi_mod raid1 ext3 jbd
> > uhci_hcd ohci_hcd ehci_hcd [last unloaded: microcode]
> > Dec 26 15:58:29 xen01 kernel: Pid: 27998, comm: udevd Not tainted
> > 2.6.32.27-0.xen.pvops.choon.centos5 #1 S3420GP
> > Dec 26 15:58:29 xen01 kernel: RIP: e030:[<ffffffff8100cb5b>]
> > [<ffffffff8100cb5b>] pin_pagetable_pfn+0x53/0x59
> > Dec 26 15:58:29 xen01 kernel: RSP: e02b:ffff88003bc3bdc8  EFLAGS:
> 00010282
> > Dec 26 15:58:29 xen01 kernel: RAX: 00000000ffffffea RBX: 0000000000017605
> > RCX: 00000000000000bb
> > Dec 26 15:58:29 xen01 kernel: RDX: 00000000deadbeef RSI: 00000000deadbeef
> > RDI: 00000000deadbeef
> > Dec 26 15:58:29 xen01 kernel: RBP: ffff88003bc3bde8 R08: 0000000000000028
> > R09: ffff880000000000
> > Dec 26 15:58:29 xen01 kernel: R10: 00000000deadbeef R11: 00007fdb5665e600
> > R12: 0000000000000003
> > Dec 26 15:58:30 xen01 kernel: R13: 0000000000017605 R14: ffff880012ee0780
> > R15: 00007fdb56224268
> > Dec 26 15:58:30 xen01 kernel: FS:  00007fdb56fed710(0000)
> > GS:ffff88002804f000(0000) knlGS:0000000000000000
> > Dec 26 15:58:30 xen01 kernel: CS:  e033 DS: 0000 ES: 0000 CR0:
> > 000000008005003b
> > Dec 26 15:58:30 xen01 kernel: CR2: 00007fdb56224268 CR3: 000000003addb000
> > CR4: 0000000000002660
> > Dec 26 15:58:30 xen01 kernel: DR0: 0000000000000000 DR1: 0000000000000000
> > DR2: 0000000000000000
> > Dec 26 15:58:30 xen01 kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0
> > DR7: 0000000000000400
> > Dec 26 15:58:30 xen01 kernel: Process udevd (pid: 27998, threadinfo
> > ffff88003bc3a000, task ffff880012ee0780)
> > Dec 26 15:58:30 xen01 kernel: Stack:
> > Dec 26 15:58:30 xen01 kernel:  0000000000000000 0000000000424121
> > 000000013f00ae20 0000000000017605
> > Dec 26 15:58:30 xen01 kernel: <0> ffff88003bc3be08 ffffffff8100e07c
> > ffff88003a3c2580 ffff880034bb6588
> > Dec 26 15:58:30 xen01 kernel: <0> ffff88003bc3be18 ffffffff8100e0af
> > ffff88003bc3be58 ffffffff810a402f
> > Dec 26 15:58:31 xen01 kernel: Call Trace:
> > Dec 26 15:58:31 xen01 kernel:  [<ffffffff8100e07c>]
> > xen_alloc_ptpage+0x64/0x69
> > Dec 26 15:58:31 xen01 kernel:  [<ffffffff8100e0af>]
> xen_alloc_pte+0xe/0x10
> > Dec 26 15:58:31 xen01 kernel:  [<ffffffff810a402f>] __pte_alloc+0x70/0xce
> > Dec 26 15:58:31 xen01 kernel:  [<ffffffff810a41cd>]
> > handle_mm_fault+0x140/0x8b9
> > Dec 26 15:58:31 xen01 kernel:  [<ffffffff810d2ecc>] ? d_kill+0x3a/0x42
> > Dec 26 15:58:31 xen01 kernel:  [<ffffffff810c4cd1>] ? __fput+0x1cb/0x1da
> > Dec 26 15:58:31 xen01 kernel:  [<ffffffff8131be4d>]
> > do_page_fault+0x252/0x2e2
> > Dec 26 15:58:31 xen01 kernel:  [<ffffffff81319dd5>] page_fault+0x25/0x30
> > Dec 26 15:58:31 xen01 kernel: Code: 48 b8 ff ff ff ff ff ff ff 7f 48 21
> c2
> > 48 89 55 e8 48 8d 7d e0 be 01 00 00 00 31 d2 41 ba f0 7f 00 00 e8 e9 c7
> ff
> > ff 85 c0 74 04 <0f> 0b eb fe c9 c3 55 40 f6 c7 01 48 89 e5 53 48 89 fb 74
> 5b
> > 48
> > Dec 26 15:58:31 xen01 kernel: RIP  [<ffffffff8100cb5b>]
> > pin_pagetable_pfn+0x53/0x59
> > Dec 26 15:58:31 xen01 kernel:  RSP <ffff88003bc3bdc8>
> > Dec 26 15:58:31 xen01 kernel: ---[ end trace 540bcf6f0170242d ]---
> >
> > Triggered BUG() in line 1860:
> >
> > static void pin_pagetable_pfn(unsigned cmd, unsigned long pfn)
> > {
> >         struct mmuext_op op;
> >         op.cmd = cmd;
> >         op.arg1.mfn = pfn_to_mfn(pfn);
> >            if (HYPERVISOR_mmuext_op(&op, 1, NULL, DOMID_SELF))
> >                    BUG(); <<THIS ONE?
>
> Yup.
> > }
> >
> > Any idea?
>
> Do you get to see this every time you do LVM migrate?
>

My method of migration is to do below without using xen xm migrate... ...

1. LVM snapshot of domU
2. mount LVM snapshot domU
3. rsync over to the target host
4. umount LVM snapshot domU
5. remove LVM snapshot domU
6. shutdown domU
7. mount LVM domU
8. rsync mounted LVM domU over to target host
9. start domU in the new target host

Actually server will crash if I do daily LVM snapshot to backup domUs not
just for migration.  And this happen almost daily :(

Method of backup domU:

1. LVM snapshot domU
2. mount LVM snapshot domU
3. rsync to disk as backup
4. umount LVM snapshot domU
5. remove LVM snapshot domU

Even I use combined ionice and nice... ... yet still crash... ...

Maybe it is time to roll back to XenLinux 2.6.18.8... ...

Thanks.

Kindest regards,
Giam Teck Choon

[-- Attachment #1.2: Type: text/html, Size: 6812 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: kernel BUG at arch/x86/xen/mmu.c:1860!

[Pasi Kärkkäinen]

On Tue, Dec 28, 2010 at 06:14:16AM +0800, Teck Choon Giam wrote:
>    On Mon, Dec 27, 2010 at 11:53 PM, Konrad Rzeszutek Wilk
>    <[1]konrad.wilk@oracle.com> wrote:
> 
>      On Sun, Dec 26, 2010 at 04:16:16PM +0800, Teck Choon Giam wrote:
>      > Hi,
>      >
>      > Information: CentOS 5.5 x86_64
>      > dom0: latest xen/stable-2.6.32.x pvops git commit
>      > 75cc13f5aa29b4f3227d269ca165dfa8937c94fe
>      > xen version: xen-4.0.2-rc1-pre from xen-4.0-testing.hg changeset 21422
>      >
>      > While doing LVM snapshot for migration and get the following:
>      >
>      > Dec 26 15:58:29 xen01 kernel: ------------[ cut here ]------------
>      > Dec 26 15:58:29 xen01 kernel: kernel BUG at arch/x86/xen/mmu.c:1860!
>      > Dec 26 15:58:29 xen01 kernel: invalid opcode: 0000 [#1] SMP
>      > Dec 26 15:58:29 xen01 kernel: last sysfs file: /sys/block/dm-26/dev
>      > Dec 26 15:58:29 xen01 kernel: CPU 0
>      > Dec 26 15:58:29 xen01 kernel: Modules linked in: ipt_MASQUERADE
>      iptable_nat
>      > nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack
>      ipt_REJECT
>      > xt_tcpudp xt_physdev iptable_filter ip_tables x_tables bridge stp
>      be2iscsi
>      > iscsi_tcp bnx2i cnic uio ipv6 cxgb3i cxgb3 mdio libiscsi_tcp libiscsi
>      > scsi_transport_iscsi loop dm_multipath scsi_dh video backlight output
>      sbs
>      > sbshc power_meter hwmon battery acpi_memhotplug xen_acpi_memhotplug ac
>      > parport_pc lp parport sg tpm_tis tpm tpm_bios button i2c_i801 i2c_core
>      > iTCO_wdt e1000e shpchp pcspkr dm_snapshot dm_zero dm_mirror
>      dm_region_hash
>      > dm_log dm_mod usb_storage ahci libata sd_mod scsi_mod raid1 ext3 jbd
>      > uhci_hcd ohci_hcd ehci_hcd [last unloaded: microcode]
>      > Dec 26 15:58:29 xen01 kernel: Pid: 27998, comm: udevd Not tainted
>      > 2.6.32.27-0.xen.pvops.choon.centos5 #1 S3420GP
>      > Dec 26 15:58:29 xen01 kernel: RIP: e030:[<ffffffff8100cb5b>]
>      > [<ffffffff8100cb5b>] pin_pagetable_pfn+0x53/0x59
>      > Dec 26 15:58:29 xen01 kernel: RSP: e02b:ffff88003bc3bdc8  EFLAGS:
>      00010282
>      > Dec 26 15:58:29 xen01 kernel: RAX: 00000000ffffffea RBX:
>      0000000000017605
>      > RCX: 00000000000000bb
>      > Dec 26 15:58:29 xen01 kernel: RDX: 00000000deadbeef RSI:
>      00000000deadbeef
>      > RDI: 00000000deadbeef
>      > Dec 26 15:58:29 xen01 kernel: RBP: ffff88003bc3bde8 R08:
>      0000000000000028
>      > R09: ffff880000000000
>      > Dec 26 15:58:29 xen01 kernel: R10: 00000000deadbeef R11:
>      00007fdb5665e600
>      > R12: 0000000000000003
>      > Dec 26 15:58:30 xen01 kernel: R13: 0000000000017605 R14:
>      ffff880012ee0780
>      > R15: 00007fdb56224268
>      > Dec 26 15:58:30 xen01 kernel: FS:  00007fdb56fed710(0000)
>      > GS:ffff88002804f000(0000) knlGS:0000000000000000
>      > Dec 26 15:58:30 xen01 kernel: CS:  e033 DS: 0000 ES: 0000 CR0:
>      > 000000008005003b
>      > Dec 26 15:58:30 xen01 kernel: CR2: 00007fdb56224268 CR3:
>      000000003addb000
>      > CR4: 0000000000002660
>      > Dec 26 15:58:30 xen01 kernel: DR0: 0000000000000000 DR1:
>      0000000000000000
>      > DR2: 0000000000000000
>      > Dec 26 15:58:30 xen01 kernel: DR3: 0000000000000000 DR6:
>      00000000ffff0ff0
>      > DR7: 0000000000000400
>      > Dec 26 15:58:30 xen01 kernel: Process udevd (pid: 27998, threadinfo
>      > ffff88003bc3a000, task ffff880012ee0780)
>      > Dec 26 15:58:30 xen01 kernel: Stack:
>      > Dec 26 15:58:30 xen01 kernel:  0000000000000000 0000000000424121
>      > 000000013f00ae20 0000000000017605
>      > Dec 26 15:58:30 xen01 kernel: <0> ffff88003bc3be08 ffffffff8100e07c
>      > ffff88003a3c2580 ffff880034bb6588
>      > Dec 26 15:58:30 xen01 kernel: <0> ffff88003bc3be18 ffffffff8100e0af
>      > ffff88003bc3be58 ffffffff810a402f
>      > Dec 26 15:58:31 xen01 kernel: Call Trace:
>      > Dec 26 15:58:31 xen01 kernel:  [<ffffffff8100e07c>]
>      > xen_alloc_ptpage+0x64/0x69
>      > Dec 26 15:58:31 xen01 kernel:  [<ffffffff8100e0af>]
>      xen_alloc_pte+0xe/0x10
>      > Dec 26 15:58:31 xen01 kernel:  [<ffffffff810a402f>]
>      __pte_alloc+0x70/0xce
>      > Dec 26 15:58:31 xen01 kernel:  [<ffffffff810a41cd>]
>      > handle_mm_fault+0x140/0x8b9
>      > Dec 26 15:58:31 xen01 kernel:  [<ffffffff810d2ecc>] ? d_kill+0x3a/0x42
>      > Dec 26 15:58:31 xen01 kernel:  [<ffffffff810c4cd1>] ?
>      __fput+0x1cb/0x1da
>      > Dec 26 15:58:31 xen01 kernel:  [<ffffffff8131be4d>]
>      > do_page_fault+0x252/0x2e2
>      > Dec 26 15:58:31 xen01 kernel:  [<ffffffff81319dd5>]
>      page_fault+0x25/0x30
>      > Dec 26 15:58:31 xen01 kernel: Code: 48 b8 ff ff ff ff ff ff ff 7f 48
>      21 c2
>      > 48 89 55 e8 48 8d 7d e0 be 01 00 00 00 31 d2 41 ba f0 7f 00 00 e8 e9
>      c7 ff
>      > ff 85 c0 74 04 <0f> 0b eb fe c9 c3 55 40 f6 c7 01 48 89 e5 53 48 89 fb
>      74 5b
>      > 48
>      > Dec 26 15:58:31 xen01 kernel: RIP  [<ffffffff8100cb5b>]
>      > pin_pagetable_pfn+0x53/0x59
>      > Dec 26 15:58:31 xen01 kernel:  RSP <ffff88003bc3bdc8>
>      > Dec 26 15:58:31 xen01 kernel: ---[ end trace 540bcf6f0170242d ]---
>      >
>      > Triggered BUG() in line 1860:
>      >
>      > static void pin_pagetable_pfn(unsigned cmd, unsigned long pfn)
>      > {
>      >         struct mmuext_op op;
>      >         op.cmd = cmd;
>      >         op.arg1.mfn = pfn_to_mfn(pfn);
>      >            if (HYPERVISOR_mmuext_op(&op, 1, NULL, DOMID_SELF))
>      >                    BUG(); <<THIS ONE?
> 
>      Yup.
>      > }
>      >
>      > Any idea?
> 
>      Do you get to see this every time you do LVM migrate?
> 
>    My method of migration is to do below without using xen xm migrate... ...
> 
>    1. LVM snapshot of domU
>    2. mount LVM snapshot domU
>    3. rsync over to the target host
>    4. umount LVM snapshot domU
>    5. remove LVM snapshot domU
>    6. shutdown domU
>    7. mount LVM domU
>    8. rsync mounted LVM domU over to target host
>    9. start domU in the new target host
> 
>    Actually server will crash if I do daily LVM snapshot to backup domUs not
>    just for migration.  And this happen almost daily :(
> 
>    Method of backup domU:
> 
>    1. LVM snapshot domU
>    2. mount LVM snapshot domU
>    3. rsync to disk as backup
>    4. umount LVM snapshot domU
>    5. remove LVM snapshot domU
> 
>    Even I use combined ionice and nice... ... yet still crash... ...
> 
>    Maybe it is time to roll back to XenLinux 2.6.18.8... ...
> 

It would be very good to track this down and get it fixed.. 
hopefully you're able to help a bit and try some things to debug it.

Konrad maybe has some ideas to try.. 

-- Pasi

Re: kernel BUG at arch/x86/xen/mmu.c:1860!

[Teck Choon Giam]

[-- Attachment #1.1: Type: text/plain, Size: 18214 bytes --]

> It would be very good to track this down and get it fixed..
> hopefully you're able to help a bit and try some things to debug it.
>
> Konrad maybe has some ideas to try..
>
>
I would love to track this down and have it fixed or else this is a stopper
for majority of us to deploy pvops stable kernel 2.6.32.x in production.
Well at least for me... ...

More related logs and it seems all is related to LVM since all will have the
last sysfs file to /dev/block/dm-??/dev... ?

/var/log/messages.2:Dec 16 01:00:04 xen01 kernel: ------------[ cut here
]------------
/var/log/messages.2-Dec 16 01:00:04 xen01 kernel: kernel BUG at
arch/x86/xen/mmu.c:1860!
/var/log/messages.2-Dec 16 01:00:04 xen01 kernel: invalid opcode: 0000 [#1]
SMP
/var/log/messages.2-Dec 16 01:00:04 xen01 kernel: last sysfs file:
/sys/block/dm-17/dev
/var/log/messages.2-Dec 16 01:00:04 xen01 kernel: CPU 5
/var/log/messages.2-Dec 16 01:00:04 xen01 kernel: Modules linked in:
dm_snapshot ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4
nf_defrag_ipv4 xt_state nf_conntrack ipt_REJECT xt_tcpudp xt_physdev
iptable_filter ip_tables x_tables bridge stp be2iscsi iscsi_tcp bnx2i cnic
uio ipv6 cxgb3i cxgb3 mdio libiscsi_tcp libiscsi scsi_transport_iscsi loop
dm_mirror dm_multipath scsi_dh video backlight output sbs sbshc power_meter
hwmon battery acpi_memhotplug xen_acpi_memhotplug ac parport_pc lp parport
sg tpm_tis tpm tpm_bios button i2c_i801 i2c_core iTCO_wdt e1000e shpchp
pcspkr dm_region_hash dm_log dm_mod usb_storage ahci libata sd_mod scsi_mod
raid1 ext3 jbd uhci_hcd ohci_hcd ehci_hcd [last unloaded: microcode]
/var/log/messages.2-Dec 16 01:00:04 xen01 kernel: Pid: 26100, comm: udevd
Not tainted 2.6.32.26-3.xen.pvops.choon.centos5 #1 S3420GP
/var/log/messages.2-Dec 16 01:00:04 xen01 kernel: RIP:
e030:[<ffffffff8100cb5b>]  [<ffffffff8100cb5b>] pin_pagetable_pfn+0x53/0x59
/var/log/messages.2-Dec 16 01:00:04 xen01 kernel: RSP:
e02b:ffff8800086abdc8  EFLAGS: 00010282
/var/log/messages.2-Dec 16 01:00:04 xen01 kernel: RAX: 00000000ffffffea RBX:
000000000003cf00 RCX: 00000000000001e7
/var/log/messages.2-Dec 16 01:00:04 xen01 kernel: RDX: 00000000deadbeef RSI:
00000000deadbeef RDI: 00000000deadbeef
/var/log/messages.2-Dec 16 01:00:04 xen01 kernel: RBP: ffff8800086abde8 R08:
0000000000000800 R09: ffff880000000000
/var/log/messages.2-Dec 16 01:00:04 xen01 kernel: R10: 00000000deadbeef R11:
00007fe3d2a72600 R12: 0000000000000003
/var/log/messages.2-Dec 16 01:00:04 xen01 kernel: R13: 000000000003cf00 R14:
ffff88003384c140 R15: 00007fe3d2638268
/var/log/messages.2-Dec 16 01:00:04 xen01 kernel: FS:
00007fe3d3401710(0000) GS:ffff8800280e0000(0000) knlGS:0000000000000000
/var/log/messages.2-Dec 16 01:00:04 xen01 kernel: CS:  e033 DS: 0000 ES:
0000 CR0: 000000008005003b
/var/log/messages.2-Dec 16 01:00:04 xen01 kernel: CR2: 00007fe3d2638268 CR3:
0000000014ff7000 CR4: 0000000000002660
/var/log/messages.2-Dec 16 01:00:04 xen01 kernel: DR0: 0000000000000000 DR1:
0000000000000000 DR2: 0000000000000000
/var/log/messages.2-Dec 16 01:00:04 xen01 kernel: DR3: 0000000000000000 DR6:
00000000ffff0ff0 DR7: 0000000000000400
/var/log/messages.2-Dec 16 01:00:04 xen01 kernel: Process udevd (pid: 26100,
threadinfo ffff8800086aa000, task ffff88003384c140)
/var/log/messages.2-Dec 16 01:00:04 xen01 kernel: Stack:
/var/log/messages.2-Dec 16 01:00:04 xen01 kernel:  0000000000000000
000000000024f026 000000013e766520 000000000003cf00
/var/log/messages.2-Dec 16 01:00:04 xen01 kernel: <0> ffff8800086abe08
ffffffff8100e07c ffff88003ad77ac0 ffff88003ce03498
/var/log/messages.2-Dec 16 01:00:04 xen01 kernel: <0> ffff8800086abe18
ffffffff8100e0af ffff8800086abe58 ffffffff810a4013
/var/log/messages.2-Dec 16 01:00:04 xen01 kernel: Call Trace:
/var/log/messages.2-Dec 16 01:00:04 xen01 kernel:  [<ffffffff8100e07c>]
xen_alloc_ptpage+0x64/0x69
/var/log/messages.2-Dec 16 01:00:04 xen01 kernel:  [<ffffffff8100e0af>]
xen_alloc_pte+0xe/0x10
/var/log/messages.2-Dec 16 01:00:04 xen01 kernel:  [<ffffffff810a4013>]
__pte_alloc+0x70/0xce
/var/log/messages.2-Dec 16 01:00:04 xen01 kernel:  [<ffffffff810a41b1>]
handle_mm_fault+0x140/0x8b9
/var/log/messages.2-Dec 16 01:00:04 xen01 kernel:  [<ffffffff810d2ea0>] ?
d_kill+0x3a/0x42
/var/log/messages.2-Dec 16 01:00:04 xen01 kernel:  [<ffffffff8131ba4d>]
do_page_fault+0x252/0x2e2
/var/log/messages.2-Dec 16 01:00:04 xen01 kernel:  [<ffffffff813199d5>]
page_fault+0x25/0x30
/var/log/messages.2-Dec 16 01:00:04 xen01 kernel: Code: 48 b8 ff ff ff ff ff
ff ff 7f 48 21 c2 48 89 55 e8 48 8d 7d e0 be 01 00 00 00 31 d2 41 ba f0 7f
00 00 e8 e9 c7 ff ff 85 c0 74 04 <0f> 0b eb fe c9 c3 55 40 f6 c7 01 48 89 e5
53 48 89 fb 74 5b 48
/var/log/messages.2-Dec 16 01:00:04 xen01 kernel: RIP  [<ffffffff8100cb5b>]
pin_pagetable_pfn+0x53/0x59
/var/log/messages.2-Dec 16 01:00:04 xen01 kernel:  RSP <ffff8800086abdc8>
/var/log/messages.2-Dec 16 01:00:04 xen01 kernel: ---[ end trace
6828227b20a6a7c6 ]---

/var/log/messages.1:Dec 25 21:45:20 xen01 kernel: ------------[ cut here
]------------
/var/log/messages.1-Dec 25 21:45:20 xen01 kernel: kernel BUG at
arch/x86/xen/mmu.c:1860!
/var/log/messages.1-Dec 25 21:45:20 xen01 kernel: invalid opcode: 0000 [#1]
SMP
/var/log/messages.1-Dec 25 21:45:20 xen01 kernel: last sysfs file:
/sys/block/dm-26/dev
/var/log/messages.1-Dec 25 21:45:20 xen01 kernel: CPU 7
/var/log/messages.1-Dec 25 21:45:20 xen01 kernel: Modules linked in:
ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state
nf_conntrack ipt_REJECT xt_tcpudp xt_physdev iptable_filter ip_tables
x_tables bridge stp be2iscsi iscsi_tcp bnx2i cnic uio ipv6 cxgb3i cxgb3 mdio
libiscsi_tcp libiscsi scsi_transport_iscsi loop dm_multipath scsi_dh video
backlight output sbs sbshc power_meter hwmon battery acpi_memhotplug
xen_acpi_memhotplug ac parport_pc lp parport sg tpm_tis tpm tpm_bios button
i2c_i801 i2c_core iTCO_wdt shpchp e1000e pcspkr dm_snapshot dm_zero
dm_mirror dm_region_hash dm_log dm_mod usb_storage ahci libata sd_mod
scsi_mod raid1 ext3 jbd uhci_hcd ohci_hcd ehci_hcd [last unloaded:
microcode]
/var/log/messages.1-Dec 25 21:45:20 xen01 kernel: Pid: 20985, comm: udevd
Not tainted 2.6.32.26-3.xen.pvops.choon.centos5 #1 S3420GP
/var/log/messages.1-Dec 25 21:45:20 xen01 kernel: RIP:
e030:[<ffffffff8100cb5b>]  [<ffffffff8100cb5b>] pin_pagetable_pfn+0x53/0x59
/var/log/messages.1-Dec 25 21:45:20 xen01 kernel: RSP:
e02b:ffff880035c91dc8  EFLAGS: 00010282
/var/log/messages.1-Dec 25 21:45:20 xen01 kernel: RAX: 00000000ffffffea RBX:
0000000000036540 RCX: 00000000000001b2
/var/log/messages.1-Dec 25 21:45:20 xen01 kernel: RDX: 00000000deadbeef RSI:
00000000deadbeef RDI: 00000000deadbeef
/var/log/messages.1-Dec 25 21:45:20 xen01 kernel: RBP: ffff880035c91de8 R08:
0000000000000a00 R09: ffff880000000000
/var/log/messages.1-Dec 25 21:45:20 xen01 kernel: R10: 00000000deadbeef R11:
00007f8988e49600 R12: 0000000000000003
/var/log/messages.1-Dec 25 21:45:21 xen01 kernel: R13: 0000000000036540 R14:
ffff880036768540 R15: 00007f8988a0f268
/var/log/messages.1-Dec 25 21:45:21 xen01 kernel: FS:
00007f89897d8710(0000) GS:ffff88002811a000(0000) knlGS:0000000000000000
/var/log/messages.1-Dec 25 21:45:21 xen01 kernel: CS:  e033 DS: 0000 ES:
0000 CR0: 000000008005003b
/var/log/messages.1-Dec 25 21:45:21 xen01 kernel: CR2: 00007f8988a0f268 CR3:
00000000375b8000 CR4: 0000000000002660
/var/log/messages.1-Dec 25 21:45:21 xen01 kernel: DR0: 0000000000000000 DR1:
0000000000000000 DR2: 0000000000000000
/var/log/messages.1-Dec 25 21:45:21 xen01 kernel: DR3: 0000000000000000 DR6:
00000000ffff0ff0 DR7: 0000000000000400
/var/log/messages.1-Dec 25 21:45:21 xen01 kernel: Process udevd (pid: 20985,
threadinfo ffff880035c90000, task ffff880036768540)
/var/log/messages.1-Dec 25 21:45:21 xen01 kernel: Stack:
/var/log/messages.1-Dec 25 21:45:21 xen01 kernel:  0000000000000000
00000000004431e6 000000013dba6e20 0000000000036540
/var/log/messages.1-Dec 25 21:45:21 xen01 kernel: <0> ffff880035c91e08
ffffffff8100e07c ffff8800375abac0 ffff88000665e228
/var/log/messages.1-Dec 25 21:45:21 xen01 kernel: <0> ffff880035c91e18
ffffffff8100e0af ffff880035c91e58 ffffffff810a4013
/var/log/messages.1-Dec 25 21:45:21 xen01 kernel: Call Trace:
/var/log/messages.1-Dec 25 21:45:21 xen01 kernel:  [<ffffffff8100e07c>]
xen_alloc_ptpage+0x64/0x69
/var/log/messages.1-Dec 25 21:45:21 xen01 kernel:  [<ffffffff8100e0af>]
xen_alloc_pte+0xe/0x10
/var/log/messages.1-Dec 25 21:45:21 xen01 kernel:  [<ffffffff810a4013>]
__pte_alloc+0x70/0xce
/var/log/messages.1-Dec 25 21:45:21 xen01 kernel:  [<ffffffff810a41b1>]
handle_mm_fault+0x140/0x8b9
/var/log/messages.1-Dec 25 21:45:21 xen01 kernel:  [<ffffffff8131ba4d>]
do_page_fault+0x252/0x2e2
/var/log/messages.1-Dec 25 21:45:21 xen01 kernel:  [<ffffffff813199d5>]
page_fault+0x25/0x30
/var/log/messages.1-Dec 25 21:45:21 xen01 kernel: Code: 48 b8 ff ff ff ff ff
ff ff 7f 48 21 c2 48 89 55 e8 48 8d 7d e0 be 01 00 00 00 31 d2 41 ba f0 7f
00 00 e8 e9 c7 ff ff 85 c0 74 04 <0f> 0b eb fe c9 c3 55 40 f6 c7 01 48 89 e5
53 48 89 fb 74 5b 48
/var/log/messages.1-Dec 25 21:45:21 xen01 kernel: RIP  [<ffffffff8100cb5b>]
pin_pagetable_pfn+0x53/0x59
/var/log/messages.1-Dec 25 21:45:21 xen01 kernel:  RSP <ffff880035c91dc8>
/var/log/messages.1-Dec 25 21:45:21 xen01 kernel: ---[ end trace
f394f31a52cfbac7 ]---

/var/log/messages:Dec 28 01:20:44 xen01 kernel: ------------[ cut here
]------------
/var/log/messages-Dec 28 01:20:44 xen01 kernel: kernel BUG at
arch/x86/xen/mmu.c:1860!
/var/log/messages-Dec 28 01:20:44 xen01 kernel: invalid opcode: 0000 [#1]
SMP
/var/log/messages-Dec 28 01:20:44 xen01 kernel: last sysfs file:
/sys/block/dm-26/dev
/var/log/messages-Dec 28 01:20:44 xen01 kernel: CPU 2
/var/log/messages-Dec 28 01:20:44 xen01 kernel: Modules linked in:
ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state
nf_conntrack ipt_REJECT xt_tcpudp xt_physdev iptable_filter ip_tables
x_tables bridge stp be2iscsi iscsi_tcp bnx2i cnic uio ipv6 cxgb3i cxgb3 mdio
libiscsi_tcp libiscsi scsi_transport_iscsi loop dm_multipath scsi_dh video
backlight output sbs sbshc power_meter hwmon battery acpi_memhotplug
xen_acpi_memhotplug ac parport_pc lp parport sg tpm_tis tpm tpm_bios button
i2c_i801 i2c_core iTCO_wdt e1000e shpchp pcspkr dm_snapshot dm_zero
dm_mirror dm_region_hash dm_log dm_mod usb_storage ahci libata sd_mod
scsi_mod raid1 ext3 jbd uhci_hcd ohci_hcd ehci_hcd [last unloaded:
microcode]
/var/log/messages-Dec 28 01:20:44 xen01 kernel: Pid: 30295, comm: sh Not
tainted 2.6.32.27-0.xen.pvops.choon.centos5 #1 S3420GP
/var/log/messages-Dec 28 01:20:44 xen01 kernel: RIP:
e030:[<ffffffff8100cb5b>]  [<ffffffff8100cb5b>] pin_pagetable_pfn+0x53/0x59
/var/log/messages-Dec 28 01:20:44 xen01 kernel: RSP: e02b:ffff880036661dc8
EFLAGS: 00010282
/var/log/messages-Dec 28 01:20:44 xen01 kernel: RAX: 00000000ffffffea RBX:
000000000003be66 RCX: 00000000000001df
/var/log/messages-Dec 28 01:20:44 xen01 kernel: RDX: 00000000deadbeef RSI:
00000000deadbeef RDI: 00000000deadbeef
/var/log/messages-Dec 28 01:20:44 xen01 kernel: RBP: ffff880036661de8 R08:
0000000000000330 R09: ffff880000000000
/var/log/messages-Dec 28 01:20:44 xen01 kernel: R10: 00000000deadbeef R11:
0000000000000246 R12: 0000000000000003
/var/log/messages-Dec 28 01:20:44 xen01 kernel: R13: 000000000003be66 R14:
ffff88001845e180 R15: 00000037e629a4d5
/var/log/messages-Dec 28 01:20:44 xen01 kernel: FS:  00007f240f1b96e0(0000)
GS:ffff880028089000(0000) knlGS:0000000000000000
/var/log/messages-Dec 28 01:20:44 xen01 kernel: CS:  e033 DS: 0000 ES: 0000
CR0: 000000008005003b
/var/log/messages-Dec 28 01:20:44 xen01 kernel: CR2: 00000037e629a4d5 CR3:
0000000031765000 CR4: 0000000000002660
/var/log/messages-Dec 28 01:20:44 xen01 kernel: DR0: 0000000000000000 DR1:
0000000000000000 DR2: 0000000000000000
/var/log/messages-Dec 28 01:20:44 xen01 kernel: DR3: 0000000000000000 DR6:
00000000ffff0ff0 DR7: 0000000000000400
/var/log/messages-Dec 28 01:20:44 xen01 kernel: Process sh (pid: 30295,
threadinfo ffff880036660000, task ffff88001845e180)
/var/log/messages-Dec 28 01:20:44 xen01 kernel: Stack:
/var/log/messages-Dec 28 01:20:44 xen01 kernel:  0000000000000000
00000000004508c0 000000013dae4820 000000000003be66
/var/log/messages-Dec 28 01:20:44 xen01 kernel: <0> ffff880036661e08
ffffffff8100e07c ffff880035da1580 ffff880035c48988
/var/log/messages-Dec 28 01:20:44 xen01 kernel: <0> ffff880036661e18
ffffffff8100e0af ffff880036661e58 ffffffff810a402f
/var/log/messages-Dec 28 01:20:44 xen01 kernel: Call Trace:
/var/log/messages-Dec 28 01:20:44 xen01 kernel:  [<ffffffff8100e07c>]
xen_alloc_ptpage+0x64/0x69
/var/log/messages-Dec 28 01:20:44 xen01 kernel:  [<ffffffff8100e0af>]
xen_alloc_pte+0xe/0x10
/var/log/messages-Dec 28 01:20:44 xen01 kernel:  [<ffffffff810a402f>]
__pte_alloc+0x70/0xce
/var/log/messages-Dec 28 01:20:44 xen01 kernel:  [<ffffffff810a41cd>]
handle_mm_fault+0x140/0x8b9
/var/log/messages-Dec 28 01:20:44 xen01 kernel:  [<ffffffff81319dd5>] ?
page_fault+0x25/0x30
/var/log/messages-Dec 28 01:20:44 xen01 kernel:  [<ffffffff8131be4d>]
do_page_fault+0x252/0x2e2
/var/log/messages-Dec 28 01:20:44 xen01 kernel:  [<ffffffff8116dd7d>] ?
__put_user_4+0x1d/0x30
/var/log/messages-Dec 28 01:20:44 xen01 kernel:  [<ffffffff81319dd5>]
page_fault+0x25/0x30
/var/log/messages-Dec 28 01:20:44 xen01 kernel: Code: 48 b8 ff ff ff ff ff
ff ff 7f 48 21 c2 48 89 55 e8 48 8d 7d e0 be 01 00 00 00 31 d2 41 ba f0 7f
00 00 e8 e9 c7 ff ff 85 c0 74 04 <0f> 0b eb fe c9 c3 55 40 f6 c7 01 48 89 e5
53 48 89 fb 74 5b 48
/var/log/messages-Dec 28 01:20:45 xen01 kernel: RIP  [<ffffffff8100cb5b>]
pin_pagetable_pfn+0x53/0x59
/var/log/messages-Dec 28 01:20:45 xen01 kernel:  RSP <ffff880036661dc8>
/var/log/messages-Dec 28 01:20:45 xen01 kernel: ---[ end trace
b65ec9b025b586cf ]---

/var/log/messages:Dec 26 15:58:29 xen01 kernel: ------------[ cut here
]------------
/var/log/messages-Dec 26 15:58:29 xen01 kernel: kernel BUG at
arch/x86/xen/mmu.c:1860!
/var/log/messages-Dec 26 15:58:29 xen01 kernel: invalid opcode: 0000 [#1]
SMP
/var/log/messages-Dec 26 15:58:29 xen01 kernel: last sysfs file:
/sys/block/dm-26/dev
/var/log/messages-Dec 26 15:58:29 xen01 kernel: CPU 0
/var/log/messages-Dec 26 15:58:29 xen01 kernel: Modules linked in:
ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state
nf_conntrack ipt_REJECT xt_tcpudp xt_physdev iptable_filter ip_tables
x_tables bridge stp be2iscsi iscsi_tcp bnx2i cnic uio ipv6 cxgb3i cxgb3 mdio
libiscsi_tcp libiscsi scsi_transport_iscsi loop dm_multipath scsi_dh video
backlight output sbs sbshc power_meter hwmon battery acpi_memhotplug
xen_acpi_memhotplug ac parport_pc lp parport sg tpm_tis tpm tpm_bios button
i2c_i801 i2c_core iTCO_wdt e1000e shpchp pcspkr dm_snapshot dm_zero
dm_mirror dm_region_hash dm_log dm_mod usb_storage ahci libata sd_mod
scsi_mod raid1 ext3 jbd uhci_hcd ohci_hcd ehci_hcd [last unloaded:
microcode]
/var/log/messages-Dec 26 15:58:29 xen01 kernel: Pid: 27998, comm: udevd Not
tainted 2.6.32.27-0.xen.pvops.choon.centos5 #1 S3420GP
/var/log/messages-Dec 26 15:58:29 xen01 kernel: RIP:
e030:[<ffffffff8100cb5b>]  [<ffffffff8100cb5b>] pin_pagetable_pfn+0x53/0x59
/var/log/messages-Dec 26 15:58:29 xen01 kernel: RSP: e02b:ffff88003bc3bdc8
EFLAGS: 00010282
/var/log/messages-Dec 26 15:58:29 xen01 kernel: RAX: 00000000ffffffea RBX:
0000000000017605 RCX: 00000000000000bb
/var/log/messages-Dec 26 15:58:29 xen01 kernel: RDX: 00000000deadbeef RSI:
00000000deadbeef RDI: 00000000deadbeef
/var/log/messages-Dec 26 15:58:29 xen01 kernel: RBP: ffff88003bc3bde8 R08:
0000000000000028 R09: ffff880000000000
/var/log/messages-Dec 26 15:58:29 xen01 kernel: R10: 00000000deadbeef R11:
00007fdb5665e600 R12: 0000000000000003
/var/log/messages-Dec 26 15:58:30 xen01 kernel: R13: 0000000000017605 R14:
ffff880012ee0780 R15: 00007fdb56224268
/var/log/messages-Dec 26 15:58:30 xen01 kernel: FS:  00007fdb56fed710(0000)
GS:ffff88002804f000(0000) knlGS:0000000000000000
/var/log/messages-Dec 26 15:58:30 xen01 kernel: CS:  e033 DS: 0000 ES: 0000
CR0: 000000008005003b
/var/log/messages-Dec 26 15:58:30 xen01 kernel: CR2: 00007fdb56224268 CR3:
000000003addb000 CR4: 0000000000002660
/var/log/messages-Dec 26 15:58:30 xen01 kernel: DR0: 0000000000000000 DR1:
0000000000000000 DR2: 0000000000000000
/var/log/messages-Dec 26 15:58:30 xen01 kernel: DR3: 0000000000000000 DR6:
00000000ffff0ff0 DR7: 0000000000000400
/var/log/messages-Dec 26 15:58:30 xen01 kernel: Process udevd (pid: 27998,
threadinfo ffff88003bc3a000, task ffff880012ee0780)
/var/log/messages-Dec 26 15:58:30 xen01 kernel: Stack:
/var/log/messages-Dec 26 15:58:30 xen01 kernel:  0000000000000000
0000000000424121 000000013f00ae20 0000000000017605
/var/log/messages-Dec 26 15:58:30 xen01 kernel: <0> ffff88003bc3be08
ffffffff8100e07c ffff88003a3c2580 ffff880034bb6588
/var/log/messages-Dec 26 15:58:30 xen01 kernel: <0> ffff88003bc3be18
ffffffff8100e0af ffff88003bc3be58 ffffffff810a402f
/var/log/messages-Dec 26 15:58:31 xen01 kernel: Call Trace:
/var/log/messages-Dec 26 15:58:31 xen01 kernel:  [<ffffffff8100e07c>]
xen_alloc_ptpage+0x64/0x69
/var/log/messages-Dec 26 15:58:31 xen01 kernel:  [<ffffffff8100e0af>]
xen_alloc_pte+0xe/0x10
/var/log/messages-Dec 26 15:58:31 xen01 kernel:  [<ffffffff810a402f>]
__pte_alloc+0x70/0xce
/var/log/messages-Dec 26 15:58:31 xen01 kernel:  [<ffffffff810a41cd>]
handle_mm_fault+0x140/0x8b9
/var/log/messages-Dec 26 15:58:31 xen01 kernel:  [<ffffffff810d2ecc>] ?
d_kill+0x3a/0x42
/var/log/messages-Dec 26 15:58:31 xen01 kernel:  [<ffffffff810c4cd1>] ?
__fput+0x1cb/0x1da
/var/log/messages-Dec 26 15:58:31 xen01 kernel:  [<ffffffff8131be4d>]
do_page_fault+0x252/0x2e2
/var/log/messages-Dec 26 15:58:31 xen01 kernel:  [<ffffffff81319dd5>]
page_fault+0x25/0x30
/var/log/messages-Dec 26 15:58:31 xen01 kernel: Code: 48 b8 ff ff ff ff ff
ff ff 7f 48 21 c2 48 89 55 e8 48 8d 7d e0 be 01 00 00 00 31 d2 41 ba f0 7f
00 00 e8 e9 c7 ff ff 85 c0 74 04 <0f> 0b eb fe c9 c3 55 40 f6 c7 01 48 89 e5
53 48 89 fb 74 5b 48
/var/log/messages-Dec 26 15:58:31 xen01 kernel: RIP  [<ffffffff8100cb5b>]
pin_pagetable_pfn+0x53/0x59
/var/log/messages-Dec 26 15:58:31 xen01 kernel:  RSP <ffff88003bc3bdc8>
/var/log/messages-Dec 26 15:58:31 xen01 kernel: ---[ end trace
540bcf6f0170242d ]---

Thanks.

Kindest regards,
Giam Teck Choon

[-- Attachment #1.2: Type: text/html, Size: 19389 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: kernel BUG at arch/x86/xen/mmu.c:1860!

[Teck Choon Giam]

[-- Attachment #1.1: Type: text/plain, Size: 20345 bytes --]

On Wed, Dec 29, 2010 at 2:01 AM, Teck Choon Giam <giamteckchoon@gmail.com>wrote:

>
> It would be very good to track this down and get it fixed..
>> hopefully you're able to help a bit and try some things to debug it.
>>
>> Konrad maybe has some ideas to try..
>>
>>
> I would love to track this down and have it fixed or else this is a stopper
> for majority of us to deploy pvops stable kernel 2.6.32.x in production.
> Well at least for me... ...
>
> More related logs and it seems all is related to LVM since all will have
> the last sysfs file to /dev/block/dm-??/dev... ?
>


While waiting for others to offer any assistance... I created a script to
basically test lvm create snapshot, mount, umount and remove snapshot... ...
the result is quite sad as I am able to reproduce the crash with loop count
100 easily on two of my servers with different specs... ...

If anyone is willing to run the script to test whether can reproduce the
crash on their deveopment xen servers or any LVM host servers would be
great.  The below script assume that those snapshot LVM can be mounted as
personally I am running all domUs.  If your LV Group Name is different, just
ammend the LVGroupName=VolGroup or anything to suit your environment.

----------8<----------8<----------8<----------8<----------8<----------8<----------8<----------
#!/bin/sh
#
# This script is to create lvm snapshot, mount it, umount it and remove in a
# specified number of loops to test whether it will crash the host server.
#
# Created by Giam Teck Choon
#

# The LV name and for this case we are using XenGroup
LVGroupName=XenGroup

# return 1 if is mounted otherwise return 0
check_mount() {
    local checkdir=${1}
    if [ -n "$checkdir" ] ; then
        local check=`grep "$checkdir" /proc/mounts`
        if [ -n "$check" ] ; then
            return 1
        fi
    fi
    return 0
}

do_lvm_create_remove() {
    # number of loops default is 1
    local loopcountlimit=${1:-1}
    # snapshot size default is 1G
    local snapshotsize=${2:-1G}
    # implement a sleep between create, mount, umount and remove (default is
0 which is no pause)
    local pauseinterval=${3:-0}
    # We filter out control, snapshot and swap
    local count=0
    if [ -d "/dev/${LVGroupName}" ] ; then
        while [ "$count" -lt "$loopcountlimit" ]
        do
            count=`expr $count + 1`
            echo "${count} ... ... "
            for i in `ls /dev/${LVGroupName} | grep -Ev 'snapshot$' | grep
-Ev 'swap$'`; do
                if [ -h "/dev/${LVGroupName}/${i}" ] ; then
                    echo -n "lvcreate -s -v -n ${i}-snapshot -L
${snapshotsize} /dev/${LVGroupName}/${i} ... ... "
                    lvcreate -s -v -n ${i}-snapshot -L ${snapshotsize}
/dev/${LVGroupName}/${i}
                    echo "done."
                    sleep ${pauseinterval}
                    mkdir -p /mnt/testlvm/${i}
                    if [ -h "/dev/${LVGroupName}/${i}-snapshot" ] ; then
                        check_mount /mnt/testlvm/${i}
                        local ismount=$?
                        if [ "$ismount" -eq 0 ] ; then
                            echo -n "mount /dev/${LVGroupName}/${i}-snapshot
/mnt/testlvm/${i} ... ... "
                            mount /dev/${LVGroupName}/${i}-snapshot
/mnt/testlvm/${i}
                            echo "done."
                            sleep ${pauseinterval}
                        fi
                        check_mount /mnt/testlvm/${i}
                        local ismount2=$?
                        if [ "$ismount2" -eq 1 ] ; then
                            echo -n "umount /mnt/testlvm/${i} ... ... "
                            umount /mnt/testlvm/${i}
                            echo "done."
                            sleep ${pauseinterval}
                        fi
                    fi
                    rm -rf /mnt/testlvm/${i}
                    echo -n "lvremove -f /dev/${LVGroupName}/${i}-snapshot
... ... "
                    lvremove -f /dev/${LVGroupName}/${i}-snapshot
                    echo "done."
                    sleep ${pauseinterval}
                fi
            done
            rm -fr /mnt/testlvm
        done
    else
        echo "/dev/${LVGroupName} directory not found!"
        exit 1
    fi
}

case $1 in
    loop)    shift
        do_lvm_create_remove "$@"
        ;;
    *)    echo "Usage: $0 loop loopcountlimit(default is 1)
snapshotsize(default is 1GB) pauseinterval (default is 0)"
        ;;
esac
----------8<----------8<----------8<----------8<----------8<----------8<----------8<----------

I name the above script as test.sh then run:

Below is without pause
# sh test.sh loop 100

Then I will get something printed to output as below:

Message from syslogd@ at Wed Dec 29 12:13:19 2010 ...
xen03 kernel: ------------[ cut here ]------------

Message from syslogd@ at Wed Dec 29 12:13:19 2010 ...
xen03 kernel: invalid opcode: 0000 [#1] SMP

Message from syslogd@ at Wed Dec 29 12:13:19 2010 ...
xen03 kernel: last sysfs file: /sys/block/dm-16/dev

Message from syslogd@ at Wed Dec 29 12:13:19 2010 ...
xen03 kernel: Stack:

Message from syslogd@ at Wed Dec 29 12:13:19 2010 ...
xen03 kernel: Call Trace:

Message from syslogd@ at Wed Dec 29 12:13:19 2010 ...
xen03 kernel: Code: 48 b8 ff ff ff ff ff ff ff 7f 48 21 c2 48 89 55 e8 48 8d
7d e0 be 01 00 00 00 31 d2 41 ba f0 7f 00 00 e8 e9 c7 ff ff 85 c0 74 04 <0f>
0b eb fe c9 c3 55 40 f6 c7 01 48 89 e5 53 48 89 fb 74 5b 48
  Unable to read from event server.

Then I can run the below command to get the log after it is back online:

# grep -A 50 'cut here' /var/log/messages
Dec 29 11:47:32 xen03 kernel: ------------[ cut here ]------------
Dec 29 11:47:32 xen03 kernel: kernel BUG at arch/x86/xen/mmu.c:1860!
Dec 29 11:47:32 xen03 kernel: invalid opcode: 0000 [#2] SMP
Dec 29 11:47:32 xen03 kernel: last sysfs file: /sys/block/dm-16/dev
Dec 29 11:47:32 xen03 kernel: CPU 3
Dec 29 11:47:32 xen03 kernel: Modules linked in: ext4 jbd2 crc16 xt_physdev
iptable_filter ip_tables x_tables bridge stp be2iscsi iscsi_tcp bnx2i cnic
uio ipv6 cxgb3i cxgb3 mdio libiscsi_tcp libiscsi scsi_transport_iscsi loop
dm_multipath scsi_dh video backlight output sbs sbshc power_meter hwmon
battery acpi_memhotplug xen_acpi_memhotplug ac parport_pc lp parport sg tg3
libphy ide_cd_mod cdrom button serio_raw tpm_tis tpm tpm_bios pcspkr
iTCO_wdt i2c_i801 shpchp i2c_core dm_snapshot dm_zero dm_mirror
dm_region_hash dm_log dm_mod ata_piix libata sd_mod scsi_mod raid1 ext3 jbd
uhci_hcd ohci_hcd ehci_hcd [last unloaded: microcode]
Dec 29 11:47:32 xen03 kernel: Pid: 30576, comm: mpath_wait Tainted: G
D    2.6.32.27-0.xen.pvops.choon.centos5 #1 PowerEdge 860
Dec 29 11:47:32 xen03 kernel: RIP: e030:[<ffffffff8100cb5b>]
[<ffffffff8100cb5b>] pin_pagetable_pfn+0x53/0x59
Dec 29 11:47:32 xen03 kernel: RSP: e02b:ffff880030a7fdc8  EFLAGS: 00010282
Dec 29 11:47:32 xen03 kernel: RAX: 00000000ffffffea RBX: 000000000002445a
RCX: 0000000000000122
Dec 29 11:47:32 xen03 kernel: RDX: 00000000deadbeef RSI: 00000000deadbeef
RDI: 00000000deadbeef
Dec 29 11:47:32 xen03 kernel: RBP: ffff880030a7fde8 R08: 00000000000002d0
R09: ffff880000000000
Dec 29 11:47:32 xen03 kernel: R10: 00000000deadbeef R11: 0000000000000246
R12: 0000000000000003
Dec 29 11:47:32 xen03 kernel: R13: 000000000002445a R14: ffff88002456e480
R15: 0000003db6c9a4d5
Dec 29 11:47:32 xen03 kernel: FS:  00007f45a01586e0(0000)
GS:ffff8800280a6000(0000) knlGS:0000000000000000
Dec 29 11:47:32 xen03 kernel: CS:  e033 DS: 0000 ES: 0000 CR0:
000000008005003b
Dec 29 11:47:32 xen03 kernel: CR2: 0000003db6c9a4d5 CR3: 000000001cc2d000
CR4: 0000000000002660
Dec 29 11:47:32 xen03 kernel: DR0: 0000000000000000 DR1: 0000000000000000
DR2: 0000000000000000
Dec 29 11:47:32 xen03 kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0
DR7: 0000000000000400
Dec 29 11:47:32 xen03 kernel: Process mpath_wait (pid: 30576, threadinfo
ffff880030a7e000, task ffff88002456e480)
Dec 29 11:47:32 xen03 kernel: Stack:
Dec 29 11:47:32 xen03 kernel:  0000000000000000 00000000001f667f
000000013e6f3e18 000000000002445a
Dec 29 11:47:32 xen03 kernel: <0> ffff880030a7fe08 ffffffff8100e07c
ffff88003d2fc040 ffff8800246c9db0
Dec 29 11:47:32 xen03 kernel: <0> ffff880030a7fe18 ffffffff8100e0af
ffff880030a7fe58 ffffffff810a402f
Dec 29 11:47:32 xen03 kernel: Call Trace:
Dec 29 11:47:32 xen03 kernel:  [<ffffffff8100e07c>]
xen_alloc_ptpage+0x64/0x69
Dec 29 11:47:32 xen03 kernel:  [<ffffffff8100e0af>] xen_alloc_pte+0xe/0x10
Dec 29 11:47:32 xen03 kernel:  [<ffffffff810a402f>] __pte_alloc+0x70/0xce
Dec 29 11:47:32 xen03 kernel:  [<ffffffff810a41cd>]
handle_mm_fault+0x140/0x8b9
Dec 29 11:47:32 xen03 kernel:  [<ffffffff81319dd5>] ? page_fault+0x25/0x30
Dec 29 11:47:32 xen03 kernel:  [<ffffffff8131be4d>]
do_page_fault+0x252/0x2e2
Dec 29 11:47:32 xen03 kernel:  [<ffffffff8116dd7d>] ? __put_user_4+0x1d/0x30
Dec 29 11:47:32 xen03 kernel:  [<ffffffff81319dd5>] page_fault+0x25/0x30
Dec 29 11:47:32 xen03 kernel: Code: 48 b8 ff ff ff ff ff ff ff 7f 48 21 c2
48 89 55 e8 48 8d 7d e0 be 01 00 00 00 31 d2 41 ba f0 7f 00 00 e8 e9 c7 ff
ff 85 c0 74 04 <0f> 0b eb fe c9 c3 55 40 f6 c7 01 48 89 e5 53 48 89 fb 74 5b
48
Dec 29 11:47:32 xen03 kernel: RIP  [<ffffffff8100cb5b>]
pin_pagetable_pfn+0x53/0x59
Dec 29 11:47:32 xen03 kernel:  RSP <ffff880030a7fdc8>
Dec 29 11:47:32 xen03 kernel: ---[ end trace 900e639a50e97057 ]---
Dec 29 11:50:38 xen03 syslogd 1.4.1: restart.
Dec 29 11:50:38 xen03 kernel: klogd 1.4.1, log source = /proc/kmsg started.
Dec 29 11:50:38 xen03 kernel: Linux version
2.6.32.27-0.xen.pvops.choon.centos5 (mockbuild@builder5.choon.net) (gcc
version 4.1.2 20080704 (Red Hat 4.1.2-48)) #1 SMP Sat Dec 25 09:19:44 SGT
2010
Dec 29 11:50:38 xen03 kernel: Command line: ro root=/dev/md1 panic=5
panic_timeout=5
Dec 29 11:50:38 xen03 kernel: KERNEL supported cpus:
Dec 29 11:50:38 xen03 kernel:   Intel GenuineIntel
Dec 29 11:50:38 xen03 kernel:   AMD AuthenticAMD
Dec 29 11:50:38 xen03 kernel:   Centaur CentaurHauls
Dec 29 11:50:38 xen03 kernel: released 0 pages of unused memory
Dec 29 11:50:38 xen03 kernel: BIOS-provided physical RAM map:
Dec 29 11:50:38 xen03 kernel:  Xen: 0000000000000000 - 00000000000a0000
(usable)
Dec 29 11:50:38 xen03 kernel:  Xen: 00000000000a0000 - 0000000000100000
(reserved)
Dec 29 11:50:38 xen03 kernel:  Xen: 0000000000100000 - 0000000040000000
(usable)
Dec 29 11:50:38 xen03 kernel:  Xen: 00000000dffc0000 - 00000000dffcfc00
(ACPI data)
--
Dec 29 11:52:30 xen03 kernel: ------------[ cut here ]------------
Dec 29 11:52:30 xen03 kernel: kernel BUG at arch/x86/xen/mmu.c:1860!
Dec 29 11:52:30 xen03 kernel: invalid opcode: 0000 [#1] SMP
Dec 29 11:52:30 xen03 kernel: last sysfs file:
/sys/devices/system/cpu/cpu3/cache/index2/shared_cpu_map
Dec 29 11:52:30 xen03 kernel: CPU 3
Dec 29 11:52:30 xen03 kernel: Modules linked in: xt_physdev iptable_filter
ip_tables x_tables bridge stp be2iscsi iscsi_tcp bnx2i cnic uio ipv6 cxgb3i
cxgb3 mdio libiscsi_tcp libiscsi scsi_transport_iscsi loop dm_multipath
scsi_dh video backlight output sbs sbshc power_meter hwmon battery
acpi_memhotplug xen_acpi_memhotplug ac parport_pc lp parport sg tg3 libphy
ide_cd_mod cdrom serio_raw button tpm_tis tpm tpm_bios i2c_i801 i2c_core
iTCO_wdt pcspkr shpchp dm_snapshot dm_zero dm_mirror dm_region_hash dm_log
dm_mod ata_piix libata sd_mod scsi_mod raid1 ext3 jbd uhci_hcd ohci_hcd
ehci_hcd [last unloaded: microcode]
Dec 29 11:52:30 xen03 kernel: Pid: 8000, comm: udevd Not tainted
2.6.32.27-0.xen.pvops.choon.centos5 #1 PowerEdge 860
Dec 29 11:52:30 xen03 kernel: RIP: e030:[<ffffffff8100cb5b>]
[<ffffffff8100cb5b>] pin_pagetable_pfn+0x53/0x59
Dec 29 11:52:30 xen03 kernel: RSP: e02b:ffff88001f0c9dc8  EFLAGS: 00010282
Dec 29 11:52:30 xen03 kernel: RAX: 00000000ffffffea RBX: 0000000000037fc2
RCX: 00000000000001bf
Dec 29 11:52:30 xen03 kernel: RDX: 00000000deadbeef RSI: 00000000deadbeef
RDI: 00000000deadbeef
Dec 29 11:52:30 xen03 kernel: RBP: ffff88001f0c9de8 R08: 0000000000000e10
R09: ffff880000000000
Dec 29 11:52:30 xen03 kernel: R10: 00000000deadbeef R11: 00007fb810051ce0
R12: 0000000000000003
Dec 29 11:52:30 xen03 kernel: R13: 0000000000037fc2 R14: ffff88003257e100
R15: 00007fb80fc14158
Dec 29 11:52:30 xen03 kernel: FS:  00007fb81024e710(0000)
GS:ffff8800280a6000(0000) knlGS:0000000000000000
Dec 29 11:52:30 xen03 kernel: CS:  e033 DS: 0000 ES: 0000 CR0:
000000008005003b
Dec 29 11:52:30 xen03 kernel: CR2: 00007fb80fc14158 CR3: 0000000032ee7000
CR4: 0000000000002660
Dec 29 11:52:30 xen03 kernel: DR0: 0000000000000000 DR1: 0000000000000000
DR2: 0000000000000000
Dec 29 11:52:30 xen03 kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0
DR7: 0000000000000400
Dec 29 11:52:30 xen03 kernel: Process udevd (pid: 8000, threadinfo
ffff88001f0c8000, task ffff88003257e100)
Dec 29 11:52:30 xen03 kernel: Stack:
Dec 29 11:52:30 xen03 kernel:  0000000000000000 000000000020a314
000000013e6f3e18 0000000000037fc2
Dec 29 11:52:30 xen03 kernel: <0> ffff88001f0c9e08 ffffffff8100e07c
ffff880032c2a040 ffff880032aaa3f0
Dec 29 11:52:30 xen03 kernel: <0> ffff88001f0c9e18 ffffffff8100e0af
ffff88001f0c9e58 ffffffff810a402f
Dec 29 11:52:30 xen03 kernel: Call Trace:
Dec 29 11:52:30 xen03 kernel:  [<ffffffff8100e07c>]
xen_alloc_ptpage+0x64/0x69
Dec 29 11:52:30 xen03 kernel:  [<ffffffff8100e0af>] xen_alloc_pte+0xe/0x10
Dec 29 11:52:30 xen03 kernel:  [<ffffffff810a402f>] __pte_alloc+0x70/0xce
Dec 29 11:52:30 xen03 kernel:  [<ffffffff810a41cd>]
handle_mm_fault+0x140/0x8b9
Dec 29 11:52:30 xen03 kernel:  [<ffffffff810d2ecc>] ? d_kill+0x3a/0x42
Dec 29 11:52:30 xen03 kernel:  [<ffffffff810c4cd1>] ? __fput+0x1cb/0x1da
Dec 29 11:52:30 xen03 kernel:  [<ffffffff8131be4d>]
do_page_fault+0x252/0x2e2
Dec 29 11:52:30 xen03 kernel:  [<ffffffff81319dd5>] page_fault+0x25/0x30
Dec 29 11:52:30 xen03 kernel: Code: 48 b8 ff ff ff ff ff ff ff 7f 48 21 c2
48 89 55 e8 48 8d 7d e0 be 01 00 00 00 31 d2 41 ba f0 7f 00 00 e8 e9 c7 ff
ff 85 c0 74 04 <0f> 0b eb fe c9 c3 55 40 f6 c7 01 48 89 e5 53 48 89 fb 74 5b
48
Dec 29 11:52:30 xen03 kernel: RIP  [<ffffffff8100cb5b>]
pin_pagetable_pfn+0x53/0x59
Dec 29 11:52:30 xen03 kernel:  RSP <ffff88001f0c9dc8>
Dec 29 11:52:30 xen03 kernel: ---[ end trace 2b8ce24b81556aa1 ]---
Dec 29 11:52:40 xen03 kernel: dmeventd[4328]: segfault at 90 ip
0000003db740bcd0 sp 00000000408080c8 error 4 in libpthread-2.5.so
[3db7400000+16000]
Dec 29 11:55:35 xen03 syslogd 1.4.1: restart.
Dec 29 11:55:35 xen03 kernel: klogd 1.4.1, log source = /proc/kmsg started.
Dec 29 11:55:35 xen03 kernel: Linux version
2.6.32.27-0.xen.pvops.choon.centos5 (mockbuild@builder5.choon.net) (gcc
version 4.1.2 20080704 (Red Hat 4.1.2-48)) #1 SMP Sat Dec 25 09:19:44 SGT
2010
Dec 29 11:55:35 xen03 kernel: Command line: ro root=/dev/md1 panic=5
panic_timeout=5
Dec 29 11:55:35 xen03 kernel: KERNEL supported cpus:
Dec 29 11:55:35 xen03 kernel:   Intel GenuineIntel
Dec 29 11:55:35 xen03 kernel:   AMD AuthenticAMD
Dec 29 11:55:35 xen03 kernel:   Centaur CentaurHauls
Dec 29 11:55:35 xen03 kernel: released 0 pages of unused memory
Dec 29 11:55:35 xen03 kernel: BIOS-provided physical RAM map:
Dec 29 11:55:35 xen03 kernel:  Xen: 0000000000000000 - 00000000000a0000
(usable)
Dec 29 11:55:35 xen03 kernel:  Xen: 00000000000a0000 - 0000000000100000
(reserved)
Dec 29 11:55:35 xen03 kernel:  Xen: 0000000000100000 - 0000000040000000
(usable)
--
Dec 29 12:13:19 xen03 kernel: ------------[ cut here ]------------
Dec 29 12:13:19 xen03 kernel: kernel BUG at arch/x86/xen/mmu.c:1860!
Dec 29 12:13:19 xen03 kernel: invalid opcode: 0000 [#1] SMP
Dec 29 12:13:19 xen03 kernel: last sysfs file: /sys/block/dm-16/dev
Dec 29 12:13:19 xen03 kernel: CPU 1
Dec 29 12:13:19 xen03 kernel: Modules linked in: ext4 jbd2 crc16 xt_physdev
iptable_filter ip_tables x_tables bridge stp be2iscsi iscsi_tcp bnx2i cnic
uio ipv6 cxgb3i cxgb3 mdio libiscsi_tcp libiscsi scsi_transport_iscsi loop
dm_multipath scsi_dh video backlight output sbs sbshc power_meter hwmon
battery acpi_memhotplug xen_acpi_memhotplug ac parport_pc lp parport sg
ide_cd_mod cdrom serio_raw tg3 button libphy tpm_tis tpm tpm_bios iTCO_wdt
i2c_i801 pcspkr i2c_core shpchp dm_snapshot dm_zero dm_mirror dm_region_hash
dm_log dm_mod ata_piix libata sd_mod scsi_mod raid1 ext3 jbd uhci_hcd
ohci_hcd ehci_hcd [last unloaded: microcode]
Dec 29 12:13:19 xen03 kernel: Pid: 15285, comm: udevd Not tainted
2.6.32.27-0.xen.pvops.choon.centos5 #1 PowerEdge 860
Dec 29 12:13:19 xen03 kernel: RIP: e030:[<ffffffff8100cb5b>]
[<ffffffff8100cb5b>] pin_pagetable_pfn+0x53/0x59
Dec 29 12:13:19 xen03 kernel: RSP: e02b:ffff880025db5dc8  EFLAGS: 00010282
Dec 29 12:13:19 xen03 kernel: RAX: 00000000ffffffea RBX: 0000000000035093
RCX: 00000000000001a8
Dec 29 12:13:19 xen03 kernel: RDX: 00000000deadbeef RSI: 00000000deadbeef
RDI: 00000000deadbeef
Dec 29 12:13:19 xen03 kernel: RBP: ffff880025db5de8 R08: 0000000000000498
R09: ffff880000000000
Dec 29 12:13:19 xen03 kernel: R10: 00000000deadbeef R11: 00007f7824d70600
R12: 0000000000000003
Dec 29 12:13:19 xen03 kernel: R13: 0000000000035093 R14: ffff880039ff0540
R15: 00007f7824936268
Dec 29 12:13:19 xen03 kernel: FS:  00007f78256fe710(0000)
GS:ffff88002806c000(0000) knlGS:0000000000000000
Dec 29 12:13:19 xen03 kernel: CS:  e033 DS: 0000 ES: 0000 CR0:
000000008005003b
Dec 29 12:13:19 xen03 kernel: CR2: 00007f7824936268 CR3: 0000000039fef000
CR4: 0000000000002660
Dec 29 12:13:19 xen03 kernel: DR0: 0000000000000000 DR1: 0000000000000000
DR2: 0000000000000000
Dec 29 12:13:19 xen03 kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0
DR7: 0000000000000400
Dec 29 12:13:19 xen03 kernel: Process udevd (pid: 15285, threadinfo
ffff880025db4000, task ffff880039ff0540)
Dec 29 12:13:19 xen03 kernel: Stack:
Dec 29 12:13:19 xen03 kernel:  0000000000000000 0000000000207243
000000013e64d518 0000000000035093
Dec 29 12:13:19 xen03 kernel: <0> ffff880025db5e08 ffffffff8100e07c
ffff88003a50fac0 ffff8800352a6920
Dec 29 12:13:19 xen03 kernel: <0> ffff880025db5e18 ffffffff8100e0af
ffff880025db5e58 ffffffff810a402f
Dec 29 12:13:19 xen03 kernel: Call Trace:
Dec 29 12:13:19 xen03 kernel:  [<ffffffff8100e07c>]
xen_alloc_ptpage+0x64/0x69
Dec 29 12:13:19 xen03 kernel:  [<ffffffff8100e0af>] xen_alloc_pte+0xe/0x10
Dec 29 12:13:19 xen03 kernel:  [<ffffffff810a402f>] __pte_alloc+0x70/0xce
Dec 29 12:13:19 xen03 kernel:  [<ffffffff810a41cd>]
handle_mm_fault+0x140/0x8b9
Dec 29 12:13:19 xen03 kernel:  [<ffffffff813199ff>] ?
_spin_unlock_irqrestore+0x11/0x13
Dec 29 12:13:19 xen03 kernel:  [<ffffffff8131be4d>]
do_page_fault+0x252/0x2e2
Dec 29 12:13:19 xen03 kernel:  [<ffffffff81319dd5>] page_fault+0x25/0x30
Dec 29 12:13:19 xen03 kernel: Code: 48 b8 ff ff ff ff ff ff ff 7f 48 21 c2
48 89 55 e8 48 8d 7d e0 be 01 00 00 00 31 d2 41 ba f0 7f 00 00 e8 e9 c7 ff
ff 85 c0 74 04 <0f> 0b eb fe c9 c3 55 40 f6 c7 01 48 89 e5 53 48 89 fb 74 5b
48
Dec 29 12:13:19 xen03 kernel: RIP  [<ffffffff8100cb5b>]
pin_pagetable_pfn+0x53/0x59
Dec 29 12:13:19 xen03 kernel:  RSP <ffff880025db5dc8>
Dec 29 12:13:19 xen03 kernel: ---[ end trace aa0f33d0cdc0d845 ]---
Dec 29 12:16:25 xen03 syslogd 1.4.1: restart.
Dec 29 12:16:25 xen03 kernel: klogd 1.4.1, log source = /proc/kmsg started.
Dec 29 12:16:25 xen03 kernel: Linux version
2.6.32.27-0.xen.pvops.choon.centos5 (mockbuild@builder5.choon.net) (gcc
version 4.1.2 20080704 (Red Hat 4.1.2-48)) #1 SMP Sat Dec 25 09:19:44 SGT
2010
Dec 29 12:16:25 xen03 kernel: Command line: ro root=/dev/md1 panic=5
panic_timeout=5
Dec 29 12:16:25 xen03 kernel: KERNEL supported cpus:
Dec 29 12:16:25 xen03 kernel:   Intel GenuineIntel
Dec 29 12:16:25 xen03 kernel:   AMD AuthenticAMD
Dec 29 12:16:25 xen03 kernel:   Centaur CentaurHauls
Dec 29 12:16:25 xen03 kernel: released 0 pages of unused memory
Dec 29 12:16:25 xen03 kernel: BIOS-provided physical RAM map:
Dec 29 12:16:25 xen03 kernel:  Xen: 0000000000000000 - 00000000000a0000
(usable)
Dec 29 12:16:25 xen03 kernel:  Xen: 00000000000a0000 - 0000000000100000
(reserved)
Dec 29 12:16:25 xen03 kernel:  Xen: 0000000000100000 - 0000000040000000
(usable)
Dec 29 12:16:25 xen03 kernel:  Xen: 00000000dffc0000 - 00000000dffcfc00
(ACPI data)
Dec 29 12:16:25 xen03 kernel:  Xen: 00000000dffcfc00 - 00000000dffff000
(reserved)

My next crash test for this will be with a pause... to see whether I can
reproduce the crash or not... then after that I will implement a sync before
each crash to test crash... ...

Thanks.

Kindest regards,
Giam Teck Choon

[-- Attachment #1.2: Type: text/html, Size: 22634 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: kernel BUG at arch/x86/xen/mmu.c:1860!

[Teck Choon Giam]

[-- Attachment #1.1: Type: text/plain, Size: 5201 bytes --]

Below is my latest test crash script:

----------8<----------8<----------8<----------8<----------8<----------8<----------8<----------8<----------
#!/bin/sh
#
# This script is to create lvm snapshot, mount it, umount it and remove in a
# specified number of loops to test whether it will crash the host server.
# All LVM snapshots assumed can be mounted like if you are running a PV
domU.
#
# Created by Giam Teck Choon
#

# The LV name and for this case we are using XenGroup
LVGroupName=XenGroup

# return 1 if is mounted otherwise return 0
check_mount() {
    local checkdir=${1}
    if [ -n "$checkdir" ] ; then
        local check=`grep "$checkdir" /proc/mounts`
        if [ -n "$check" ] ; then
            return 1
        fi
    fi
    return 0
}

do_lvm_create_remove() {
    # number of loops default is 1
    local loopcountlimit=${1:-1}
    # snapshot size default is 1G
    local snapshotsize=${2:-1G}
    # implement a sleep between create, mount, umount and remove (default is
0 which is no pause)
    local pauseinterval=${3:-0}
    # execute commands after each pause/sleep such as sync or anything that
you want to test
    local commands=${4}
    # We filter out snapshot and swap
    local count=0
    if [ -d "/dev/${LVGroupName}" ] ; then
        while [ "$count" -lt "$loopcountlimit" ]
        do
            count=`expr $count + 1`
            echo "${count} ... ... "
            for i in `ls /dev/${LVGroupName} | grep -Ev 'snapshot$' | grep
-Ev 'swap$'`; do
                if [ -h "/dev/${LVGroupName}/${i}" ] ; then
                    echo -n "lvcreate -s -v -n ${i}-snapshot -L
${snapshotsize} /dev/${LVGroupName}/${i} ... ... "
                    lvcreate -s -v -n ${i}-snapshot -L ${snapshotsize}
/dev/${LVGroupName}/${i}
                    echo "done."
                    sleep ${pauseinterval}
                    if [ -n "$commands" ] ; then
                        echo -n "${commands} ... ... "
                        $commands
                        echo "done."
                    fi
                    mkdir -p /mnt/testlvm/${i}
                    if [ -h "/dev/${LVGroupName}/${i}-snapshot" ] ; then
                        check_mount /mnt/testlvm/${i}
                        local ismount=$?
                        if [ "$ismount" -eq 0 ] ; then
                            echo -n "mount /dev/${LVGroupName}/${i}-snapshot
/mnt/testlvm/${i} ... ... "
                            mount /dev/${LVGroupName}/${i}-snapshot
/mnt/testlvm/${i}
                            echo "done."
                            sleep ${pauseinterval}
                            if [ -n "$commands" ] ; then
                                echo -n "${commands} ... ... "
                                $commands
                                echo "done."
                            fi
                        fi
                        check_mount /mnt/testlvm/${i}
                        local ismount2=$?
                        if [ "$ismount2" -eq 1 ] ; then
                            echo -n "umount /mnt/testlvm/${i} ... ... "
                            umount /mnt/testlvm/${i}
                            echo "done."
                            sleep ${pauseinterval}
                            if [ -n "$commands" ] ; then
                                echo -n "${commands} ... ... "
                                $commands
                                echo "done."
                            fi
                        fi
                    fi
                    rm -rf /mnt/testlvm/${i}
                    echo -n "lvremove -f /dev/${LVGroupName}/${i}-snapshot
... ... "
                    lvremove -f /dev/${LVGroupName}/${i}-snapshot
                    echo "done."
                    sleep ${pauseinterval}
                    if [ -n "$commands" ] ; then
                        echo -n "${commands} ... ... "
                        $commands
                        echo "done."
                    fi
                fi
            done
            rm -fr /mnt/testlvm
        done
    else
        echo "/dev/${LVGroupName} directory not found!"
        exit 1
    fi
}

case $1 in
    loop)    shift
        do_lvm_create_remove "$@"
        ;;
    *)    cat <<HELP
Usage: $0 loop loopcountlimit snapshotsize pauseinterval commands
Where:
    loopcountlimit is default to 1
    snapshotsize is default to 1G
    pauseinterval is default to 0
    commands is default to none

Example to run with 100 loops without pause/sleep:
    $0 loop 100

Example to run with 100 loops with pause/sleep of 5 seconds:
    $0 loop 100 1G 5

Example to run with 100 loops with snapshot size of 2G instead of 1G:
    $0 loop 100 2G

Example to run with 50 loops, 1G snapshot size, 5 seconds pause and with
sync:
command with each pause/sleep
    $0 loop 50 1G 5 sync

Example to run with 50 loops, 1G snapshot size, no pause and with sync:
command with each pause/sleep
    $0 loop 50 1G 0 sync

Example to run your own commands:
    $0 loop 100 1G 5 "echo hi && sync"

HELP
        ;;
esac
----------8<----------8<----------8<----------8<----------8<----------8<----------8<----------8<----------

Thanks.

Kindest regards,
Giam Teck Choon

[-- Attachment #1.2: Type: text/html, Size: 6076 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: kernel BUG at arch/x86/xen/mmu.c:1860!

[Konrad Rzeszutek Wilk]

On Wed, Dec 29, 2010 at 12:58:15PM +0800, Teck Choon Giam wrote:
> Below is my latest test crash script:

This test script is still valid? Or do you have a more updated one?
> 
> ----------8<----------8<----------8<----------8<----------8<----------8<----------8<----------8<----------
> #!/bin/sh
> #
> # This script is to create lvm snapshot, mount it, umount it and remove in a
> # specified number of loops to test whether it will crash the host server.
> # All LVM snapshots assumed can be mounted like if you are running a PV
> domU.
> #
> # Created by Giam Teck Choon
> #
> 
> # The LV name and for this case we are using XenGroup
> LVGroupName=XenGroup
> 
> # return 1 if is mounted otherwise return 0
> check_mount() {
>     local checkdir=${1}
>     if [ -n "$checkdir" ] ; then
>         local check=`grep "$checkdir" /proc/mounts`
>         if [ -n "$check" ] ; then
>             return 1
>         fi
>     fi
>     return 0
> }
> 
> do_lvm_create_remove() {
>     # number of loops default is 1
>     local loopcountlimit=${1:-1}
>     # snapshot size default is 1G
>     local snapshotsize=${2:-1G}
>     # implement a sleep between create, mount, umount and remove (default is
> 0 which is no pause)
>     local pauseinterval=${3:-0}
>     # execute commands after each pause/sleep such as sync or anything that
> you want to test
>     local commands=${4}
>     # We filter out snapshot and swap
>     local count=0
>     if [ -d "/dev/${LVGroupName}" ] ; then
>         while [ "$count" -lt "$loopcountlimit" ]
>         do
>             count=`expr $count + 1`
>             echo "${count} ... ... "
>             for i in `ls /dev/${LVGroupName} | grep -Ev 'snapshot$' | grep
> -Ev 'swap$'`; do
>                 if [ -h "/dev/${LVGroupName}/${i}" ] ; then
>                     echo -n "lvcreate -s -v -n ${i}-snapshot -L
> ${snapshotsize} /dev/${LVGroupName}/${i} ... ... "
>                     lvcreate -s -v -n ${i}-snapshot -L ${snapshotsize}
> /dev/${LVGroupName}/${i}
>                     echo "done."
>                     sleep ${pauseinterval}
>                     if [ -n "$commands" ] ; then
>                         echo -n "${commands} ... ... "
>                         $commands
>                         echo "done."
>                     fi
>                     mkdir -p /mnt/testlvm/${i}
>                     if [ -h "/dev/${LVGroupName}/${i}-snapshot" ] ; then
>                         check_mount /mnt/testlvm/${i}
>                         local ismount=$?
>                         if [ "$ismount" -eq 0 ] ; then
>                             echo -n "mount /dev/${LVGroupName}/${i}-snapshot
> /mnt/testlvm/${i} ... ... "
>                             mount /dev/${LVGroupName}/${i}-snapshot
> /mnt/testlvm/${i}
>                             echo "done."
>                             sleep ${pauseinterval}
>                             if [ -n "$commands" ] ; then
>                                 echo -n "${commands} ... ... "
>                                 $commands
>                                 echo "done."
>                             fi
>                         fi
>                         check_mount /mnt/testlvm/${i}
>                         local ismount2=$?
>                         if [ "$ismount2" -eq 1 ] ; then
>                             echo -n "umount /mnt/testlvm/${i} ... ... "
>                             umount /mnt/testlvm/${i}
>                             echo "done."
>                             sleep ${pauseinterval}
>                             if [ -n "$commands" ] ; then
>                                 echo -n "${commands} ... ... "
>                                 $commands
>                                 echo "done."
>                             fi
>                         fi
>                     fi
>                     rm -rf /mnt/testlvm/${i}
>                     echo -n "lvremove -f /dev/${LVGroupName}/${i}-snapshot
> ... ... "
>                     lvremove -f /dev/${LVGroupName}/${i}-snapshot
>                     echo "done."
>                     sleep ${pauseinterval}
>                     if [ -n "$commands" ] ; then
>                         echo -n "${commands} ... ... "
>                         $commands
>                         echo "done."
>                     fi
>                 fi
>             done
>             rm -fr /mnt/testlvm
>         done
>     else
>         echo "/dev/${LVGroupName} directory not found!"
>         exit 1
>     fi
> }
> 
> case $1 in
>     loop)    shift
>         do_lvm_create_remove "$@"
>         ;;
>     *)    cat <<HELP
> Usage: $0 loop loopcountlimit snapshotsize pauseinterval commands
> Where:
>     loopcountlimit is default to 1
>     snapshotsize is default to 1G
>     pauseinterval is default to 0
>     commands is default to none
> 
> Example to run with 100 loops without pause/sleep:
>     $0 loop 100
> 
> Example to run with 100 loops with pause/sleep of 5 seconds:
>     $0 loop 100 1G 5
> 
> Example to run with 100 loops with snapshot size of 2G instead of 1G:
>     $0 loop 100 2G
> 
> Example to run with 50 loops, 1G snapshot size, 5 seconds pause and with
> sync:
> command with each pause/sleep
>     $0 loop 50 1G 5 sync
> 
> Example to run with 50 loops, 1G snapshot size, no pause and with sync:
> command with each pause/sleep
>     $0 loop 50 1G 0 sync
> 
> Example to run your own commands:
>     $0 loop 100 1G 5 "echo hi && sync"
> 
> HELP
>         ;;
> esac
> ----------8<----------8<----------8<----------8<----------8<----------8<----------8<----------8<----------
> 
> Thanks.
> 
> Kindest regards,
> Giam Teck Choon

> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: kernel BUG at arch/x86/xen/mmu.c:1860!

[Teck Choon Giam]

[-- Attachment #1.1: Type: text/plain, Size: 7158 bytes --]

On Fri, Jan 14, 2011 at 11:20 PM, Konrad Rzeszutek Wilk <
konrad.wilk@oracle.com> wrote:

> On Wed, Dec 29, 2010 at 12:58:15PM +0800, Teck Choon Giam wrote:
> > Below is my latest test crash script:
>
> This test script is still valid? Or do you have a more updated one?
>

Below is the more updated one.  Please note that this will mostly not
support VG in CLVM since clustered LVM doesn't support snapshot in my CentOS
5 testing... ...

----------8<----------8<----------8<----------8<----------8<----------8<----------8<----------8<----------8<----------
#!/bin/sh
#
# This script is to create lvm snapshot, mount it, umount it and remove in a
# specified number of loops to test whether it will crash the host server.
# All LVM snapshots assumed can be mounted like if you are running a PV
domU.
#
# Created by Giam Teck Choon
#

# The LV name and for this case we are using the first in vgdisplay output.
# Change the variable if you want other VG Name or change head -n 1 to
# tail -n 1 if you prefer to use last VG instead of first if you happen to
have
# more than one VG
LVGroupName=`vgdisplay | grep 'VG Name' | awk '{print $3}' | head -n 1`

if [ ! -n "$LVGroupName" ] && [ ! -d "/dev/${LVGroupName}" ] ; then
    echo "Unable to detect VG Name!"
    exit 1
fi

# return 1 if is mounted otherwise return 0
check_mount() {
    local checkdir=${1}
    if [ -n "$checkdir" ] ; then
        local check=`grep "$checkdir" /proc/mounts`
        if [ -n "$check" ] ; then
            return 1
        fi
    fi
    return 0
}

# We will create 5 testcrash LV in $LVGroupName each with 5GB size
# and format it as ext3
do_lvm_create_testcrash() {
    local lvname=${1:-testcrash}
    local lvsize=${2:-5G}
    local count=1
    local limit=5
    while [ "$count" -le "$limit" ]
    do
        if [ ! -h "/dev/${LVGroupName}/${lvname}${count}" ] ; then
            echo "lvcreate -v -n ${lvname}${count} -L ${lvsize}
${LVGroupName} ... ... "
            lvcreate -v -n ${lvname}${count} -L ${lvsize} ${LVGroupName}
            echo "lvcreate -v -n ${lvname}${count} -L ${lvsize}
${LVGroupName} completed!"
            if [ -h "/dev/${LVGroupName}/${lvname}${count}" ] ; then
                echo "mke2fs -F -j /dev/${LVGroupName}/${lvname}${count} ...
... "
                mke2fs -F -j /dev/${LVGroupName}/${lvname}${count}
                echo "mke2fs -F -j /dev/${LVGroupName}/${lvname}${count}
completed!"
            else
                echo "/dev/${LVGroupName}/${lvname}${count} not found!"
            fi
        fi
        count=`expr $count + 1`
    done
}

do_lvm_create_remove() {
    # number of loops default is 1
    local loopcountlimit=${1:-1}
    # snapshot size default is 1G
    local snapshotsize=${2:-1G}
    # implement a sleep between create, mount, umount and remove (default is
0 which is no pause)
    local pauseinterval=${3:-0}
    # execute commands after each pause/sleep such as sync or anything that
you want to test
    local commands=${4}
    # We filter out snapshot and swap
    local count=0
    if [ -d "/dev/${LVGroupName}" ] ; then
        while [ "$count" -lt "$loopcountlimit" ]
        do
            count=`expr $count + 1`
            echo "${count} ... ... "
            for i in `ls /dev/${LVGroupName} | grep -Ev 'snapshot$' | grep
-Ev 'swap$'`; do
                if [ -h "/dev/${LVGroupName}/${i}" ] ; then
                    echo -n "lvcreate -s -v -n ${i}-snapshot -L
${snapshotsize} /dev/${LVGroupName}/${i} ... ... "
                    lvcreate -s -v -n ${i}-snapshot -L ${snapshotsize}
/dev/${LVGroupName}/${i}
                    echo "done."
                    sleep ${pauseinterval}
                    if [ -n "$commands" ] ; then
                        echo -n "${commands} ... ... "
                        $commands
                        echo "done."
                    fi
                    mkdir -p /mnt/testlvm/${i}
                    if [ -h "/dev/${LVGroupName}/${i}-snapshot" ] ; then
                        check_mount /mnt/testlvm/${i}
                        local ismount=$?
                        if [ "$ismount" -eq 0 ] ; then
                            echo -n "mount /dev/${LVGroupName}/${i}-snapshot
/mnt/testlvm/${i} ... ... "
                            mount /dev/${LVGroupName}/${i}-snapshot
/mnt/testlvm/${i}
                            echo "done."
                            sleep ${pauseinterval}
                            if [ -n "$commands" ] ; then
                                echo -n "${commands} ... ... "
                                $commands
                                echo "done."
                            fi
                        fi
                        check_mount /mnt/testlvm/${i}
                        local ismount2=$?
                        if [ "$ismount2" -eq 1 ] ; then
                            echo -n "umount /mnt/testlvm/${i} ... ... "
                            umount /mnt/testlvm/${i}
                            echo "done."
                            sleep ${pauseinterval}
                            if [ -n "$commands" ] ; then
                                echo -n "${commands} ... ... "
                                $commands
                                echo "done."
                            fi
                        fi
                    fi
                    rm -rf /mnt/testlvm/${i}
                    echo -n "lvremove -f /dev/${LVGroupName}/${i}-snapshot
... ... "
                    lvremove -f /dev/${LVGroupName}/${i}-snapshot
                    echo "done."
                    sleep ${pauseinterval}
                    if [ -n "$commands" ] ; then
                        echo -n "${commands} ... ... "
                        $commands
                        echo "done."
                    fi
                fi
            done
            rm -fr /mnt/testlvm
        done
    else
        echo "/dev/${LVGroupName} directory not found!"
        exit 1
    fi
}

case $1 in
    setup)    shift
        do_lvm_create_testcrash "$@"
        ;;
    loop)    shift
        do_lvm_create_remove "$@"
        ;;
    *)    cat <<HELP
Usage: $0 loop loopcountlimit snapshotsize pauseinterval commands
Where:
    loopcountlimit is default to 1
    snapshotsize is default to 1G
    pauseinterval is default to 0
    commands is default to none

Example to run with 100 loops without pause/sleep:
    $0 loop 100

Example to run with 100 loops with pause/sleep of 5 seconds:
    $0 loop 100 1G 5

Example to run with 100 loops with snapshot size of 2G instead of 1G:
    $0 loop 100 2G

Example to run with 50 loops, 1G snapshot size, 5 seconds pause and with
sync:
command with each pause/sleep
    $0 loop 50 1G 5 sync

Example to run your own commands:
    $0 loop 100 1G 5 "echo hi && sync"

If this is the first time you are running and do not have any LV in your VG,
run:
    $0 setup
This will create 5 testcrash LV in your VG with 5GB size each (default) and
format
to ext3.

HELP
        ;;
esac

----------8<----------8<----------8<----------8<----------8<----------8<----------8<----------8<----------8<----------


Thanks.

Kindest regards,
Giam Teck Choon

[-- Attachment #1.2: Type: text/html, Size: 8667 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: kernel BUG at arch/x86/xen/mmu.c:1860!

[Konrad Rzeszutek Wilk]

On Sat, Jan 15, 2011 at 03:25:55AM +0800, Teck Choon Giam wrote:
> On Fri, Jan 14, 2011 at 11:20 PM, Konrad Rzeszutek Wilk <
> konrad.wilk@oracle.com> wrote:
> 
> > On Wed, Dec 29, 2010 at 12:58:15PM +0800, Teck Choon Giam wrote:
> > > Below is my latest test crash script:
> >
> > This test script is still valid? Or do you have a more updated one?
> >
> 
> Below is the more updated one.  Please note that this will mostly not

Can you send it as attachment? The copy-n-paste broke it.

Re: kernel BUG at arch/x86/xen/mmu.c:1860!

[Teck Choon Giam]

[-- Attachment #1.1: Type: text/plain, Size: 654 bytes --]

On Sat, Jan 15, 2011 at 3:44 AM, Konrad Rzeszutek Wilk <
konrad.wilk@oracle.com> wrote:

> On Sat, Jan 15, 2011 at 03:25:55AM +0800, Teck Choon Giam wrote:
> > On Fri, Jan 14, 2011 at 11:20 PM, Konrad Rzeszutek Wilk <
> > konrad.wilk@oracle.com> wrote:
> >
> > > On Wed, Dec 29, 2010 at 12:58:15PM +0800, Teck Choon Giam wrote:
> > > > Below is my latest test crash script:
> > >
> > > This test script is still valid? Or do you have a more updated one?
> > >
> >
> > Below is the more updated one.  Please note that this will mostly not
>
> Can you send it as attachment? The copy-n-paste broke it.
>

Sure ;)

Thanks.

Kindest regards,
Giam Teck Choon

[-- Attachment #1.2: Type: text/html, Size: 1092 bytes --]

[-- Attachment #2: testcrash.sh --]
[-- Type: application/x-sh, Size: 5146 bytes --]

[-- Attachment #3: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: kernel BUG at arch/x86/xen/mmu.c:1860!

[Teck Choon Giam]

[-- Attachment #1.1: Type: text/plain, Size: 4229 bytes --]

My latest test crash with a little different result:

Jan 15 04:27:16 xen06 kernel: last sysfs file:
/sys/devices/system/cpu/cpu3/cache/index2/shared_cpu_map

Normally is related to LVM/DM but now it is related to shared_cpu_map cache?

Jan 15 04:27:16 xen06 kernel: ------------[ cut here ]------------
Jan 15 04:27:16 xen06 kernel: kernel BUG at arch/x86/xen/mmu.c:1860!
Jan 15 04:27:16 xen06 kernel: invalid opcode: 0000 [#1] SMP
Jan 15 04:27:16 xen06 kernel: last sysfs file:
/sys/devices/system/cpu/cpu3/cache/index2/shared_cpu_map << SEE THIS?
Jan 15 04:27:16 xen06 kernel: CPU 1
Jan 15 04:27:16 xen06 kernel: Modules linked in: dm_snapshot ipt_MASQUERADE
iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack
ipt_REJECT xt_tcpudp gfs2 dlm configfs xt_physdev iptable_filter ip_tables
x_tables bridge stp be2iscsi iscsi_tcp bnx2i cnic uio ipv6 cxgb3i cxgb3 mdio
libiscsi_tcp libiscsi scsi_transport_iscsi dm_mirror dm_multipath scsi_dh
video backlight output sbs sbshc power_meter hwmon battery acpi_memhotplug
xen_acpi_memhotplug ac parport_pc lp parport sg tg3 libphy ide_cd_mod cdrom
serio_raw button tpm_tis tpm tpm_bios i2c_i801 pcspkr i2c_core shpchp
iTCO_wdt dm_region_hash dm_log dm_mod ata_piix libata sd_mod scsi_mod raid1
ext3 jbd uhci_hcd ohci_hcd ehci_hcd [last unloaded: microcode]
Jan 15 04:27:16 xen06 kernel: Pid: 26392, comm: mpath_wait Not tainted
2.6.32.27-0.xen.pvops.choon.centos5 #1 PowerEdge 860
Jan 15 04:27:16 xen06 kernel: RIP: e030:[<ffffffff8100cb5b>]
[<ffffffff8100cb5b>] pin_pagetable_pfn+0x53/0x59
Jan 15 04:27:16 xen06 kernel: RSP: e02b:ffff88001aae5dc8  EFLAGS: 00010282
Jan 15 04:27:16 xen06 kernel: RAX: 00000000ffffffea RBX: 000000000003cf04
RCX: 00000000000001e7
Jan 15 04:27:16 xen06 kernel: RDX: 00000000deadbeef RSI: 00000000deadbeef
RDI: 00000000deadbeef
Jan 15 04:27:16 xen06 kernel: RBP: ffff88001aae5de8 R08: 0000000000000820
R09: ffff880000000000
Jan 15 04:27:16 xen06 kernel: R10: 00000000deadbeef R11: 0000000000000246
R12: 0000000000000003
Jan 15 04:27:16 xen06 kernel: R13: 000000000003cf04 R14: ffff88001aa6a7c0
R15: 00000033e1a9a4d5
Jan 15 04:27:16 xen06 kernel: FS:  00007f9b8dbdf6e0(0000)
GS:ffff88002806c000(0000) knlGS:0000000000000000
Jan 15 04:27:16 xen06 kernel: CS:  e033 DS: 0000 ES: 0000 CR0:
000000008005003b
Jan 15 04:27:16 xen06 kernel: CR2: 00000033e1a9a4d5 CR3: 000000001ab61000
CR4: 0000000000002660
Jan 15 04:27:16 xen06 kernel: DR0: 0000000000000000 DR1: 0000000000000000
DR2: 0000000000000000
Jan 15 04:27:16 xen06 kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0
DR7: 0000000000000400
Jan 15 04:27:16 xen06 kernel: Process mpath_wait (pid: 26392, threadinfo
ffff88001aae4000, task ffff88001aa6a7c0)
Jan 15 04:27:16 xen06 kernel: Stack:
Jan 15 04:27:16 xen06 kernel:  0000000000000000 000000000020f3d2
000000013e64d518 000000000003cf04
Jan 15 04:27:16 xen06 kernel: <0> ffff88001aae5e08 ffffffff8100e07c
ffff880025db0040 ffff880025fda868
Jan 15 04:27:16 xen06 kernel: <0> ffff88001aae5e18 ffffffff8100e0af
ffff88001aae5e58 ffffffff810a402f
Jan 15 04:27:16 xen06 kernel: Call Trace:
Jan 15 04:27:16 xen06 kernel:  [<ffffffff8100e07c>]
xen_alloc_ptpage+0x64/0x69
Jan 15 04:27:16 xen06 kernel:  [<ffffffff8100e0af>] xen_alloc_pte+0xe/0x10
Jan 15 04:27:16 xen06 kernel:  [<ffffffff810a402f>] __pte_alloc+0x70/0xce
Jan 15 04:27:16 xen06 kernel:  [<ffffffff810a41cd>]
handle_mm_fault+0x140/0x8b9
Jan 15 04:27:16 xen06 kernel:  [<ffffffff81319dd5>] ? page_fault+0x25/0x30
Jan 15 04:27:16 xen06 kernel:  [<ffffffff8131be4d>]
do_page_fault+0x252/0x2e2
Jan 15 04:27:16 xen06 kernel:  [<ffffffff8116dd7d>] ? __put_user_4+0x1d/0x30
Jan 15 04:27:16 xen06 kernel:  [<ffffffff81319dd5>] page_fault+0x25/0x30
Jan 15 04:27:16 xen06 kernel: Code: 48 b8 ff ff ff ff ff ff ff 7f 48 21 c2
48 89 55 e8 48 8d 7d e0 be 01 00 00 00 31 d2 41 ba f0 7f 00 00 e8 e9 c7 ff
ff 85 c0 74 04 <0f> 0b eb fe c9 c3 55 40 f6 c7 01 48 89 e5 53 48 89 fb 74 5b
48
Jan 15 04:27:16 xen06 kernel: RIP  [<ffffffff8100cb5b>]
pin_pagetable_pfn+0x53/0x59
Jan 15 04:27:16 xen06 kernel:  RSP <ffff88001aae5dc8>
Jan 15 04:27:16 xen06 kernel: ---[ end trace 865f1d440d090f4f ]---

Sorry, this server doesn't have serial console though :(

Thanks.

Kindest regards,
Giam Teck Choon

[-- Attachment #1.2: Type: text/html, Size: 4517 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: kernel BUG at arch/x86/xen/mmu.c:1860!

[Teck Choon Giam]

[-- Attachment #1.1: Type: text/plain, Size: 205 bytes --]

Sorry, does anyone able to solve this bug?  My prevention doing sleep 5 and
sync method only last me longest 20+- days of uptime then the same bug will
appear :(

Thanks.

Kindest regards,
Giam Teck Choon

[-- Attachment #1.2: Type: text/html, Size: 225 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: kernel BUG at arch/x86/xen/mmu.c:1860!

[Konrad Rzeszutek Wilk]

On Mon, Jan 24, 2011 at 09:42:02AM +0800, Teck Choon Giam wrote:
> Sorry, does anyone able to solve this bug?  My prevention doing sleep 5 and
> sync method only last me longest 20+- days of uptime then the same bug will
> appear :(

You have to give more details. Which kernel and what does the back-trace look like?

> 
> Thanks.
> 
> Kindest regards,
> Giam Teck Choon

> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: kernel BUG at arch/x86/xen/mmu.c:1860!

[Teck Choon Giam]

[-- Attachment #1.1: Type: text/plain, Size: 1050 bytes --]

On Mon, Jan 24, 2011 at 10:36 PM, Konrad Rzeszutek Wilk <
konrad.wilk@oracle.com> wrote:

> On Mon, Jan 24, 2011 at 09:42:02AM +0800, Teck Choon Giam wrote:
> > Sorry, does anyone able to solve this bug?  My prevention doing sleep 5
> and
> > sync method only last me longest 20+- days of uptime then the same bug
> will
> > appear :(
>
> You have to give more details. Which kernel and what does the back-trace
> look like?
>

Thanks for your prompt reply.

Kernel is from
http://git.kernel.org/?p=linux/kernel/git/jeremy/xen.git;a=summary latest
stable/2.6.32.x

Xen version is from
http://xenbits.xensource.com/staging/xen-4.0-testing.hglatest
changeset 21439 in fact any changeset 214xx I believe.

I have posted console output for one of my xen servers and maybe my serial
console configuration not right... ...

http://lists.xensource.com/archives/html/xen-devel/2011-01/msg00177.html

How do I back-trace?  You mean I should execute my test crash script with
strace and post the output during crash?

Thanks.

Kindest regards,
Giam Teck Choon

[-- Attachment #1.2: Type: text/html, Size: 1658 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: kernel BUG at arch/x86/xen/mmu.c:1860!

[Konrad Rzeszutek Wilk]

On Mon, Jan 24, 2011 at 11:56:21PM +0800, Teck Choon Giam wrote:
> On Mon, Jan 24, 2011 at 10:36 PM, Konrad Rzeszutek Wilk <
> konrad.wilk@oracle.com> wrote:
> 
> > On Mon, Jan 24, 2011 at 09:42:02AM +0800, Teck Choon Giam wrote:
> > > Sorry, does anyone able to solve this bug?  My prevention doing sleep 5
> > and
> > > sync method only last me longest 20+- days of uptime then the same bug
> > will
> > > appear :(
> >
> > You have to give more details. Which kernel and what does the back-trace
> > look like?
> >
> 
> Thanks for your prompt reply.
> 
> Kernel is from
> http://git.kernel.org/?p=linux/kernel/git/jeremy/xen.git;a=summary latest
> stable/2.6.32.x
> 
> Xen version is from
> http://xenbits.xensource.com/staging/xen-4.0-testing.hglatest
> changeset 21439 in fact any changeset 214xx I believe.
> 
> I have posted console output for one of my xen servers and maybe my serial
> console configuration not right... ...
> 
> http://lists.xensource.com/archives/html/xen-devel/2011-01/msg00177.html
> 
> How do I back-trace?  You mean I should execute my test crash script with
> strace and post the output during crash?


No that is OK. I now remember this one - I am poking at the code to get an idea
of what might be happening. No data yet.

Re: kernel BUG at arch/x86/xen/mmu.c:1860!

[Konrad Rzeszutek Wilk]

On Tue, Jan 25, 2011 at 09:48:09AM -0500, Konrad Rzeszutek Wilk wrote:
> On Mon, Jan 24, 2011 at 11:56:21PM +0800, Teck Choon Giam wrote:
> > On Mon, Jan 24, 2011 at 10:36 PM, Konrad Rzeszutek Wilk <
> > konrad.wilk@oracle.com> wrote:
> > 
> > > On Mon, Jan 24, 2011 at 09:42:02AM +0800, Teck Choon Giam wrote:
> > > > Sorry, does anyone able to solve this bug?  My prevention doing sleep 5
> > > and
> > > > sync method only last me longest 20+- days of uptime then the same bug
> > > will
> > > > appear :(
> > >
> > > You have to give more details. Which kernel and what does the back-trace
> > > look like?
> > >
> > 
> > Thanks for your prompt reply.
> > 
> > Kernel is from
> > http://git.kernel.org/?p=linux/kernel/git/jeremy/xen.git;a=summary latest
> > stable/2.6.32.x

What does git log show? What is the latest commit you have there?

> > 
> > Xen version is from
> > http://xenbits.xensource.com/staging/xen-4.0-testing.hglatest
> > changeset 21439 in fact any changeset 214xx I believe.
> > 
> > I have posted console output for one of my xen servers and maybe my serial
> > console configuration not right... ...
> > 
> > http://lists.xensource.com/archives/html/xen-devel/2011-01/msg00177.html
> > 
> > How do I back-trace?  You mean I should execute my test crash script with
> > strace and post the output during crash?
> 
> 
> No that is OK. I now remember this one - I am poking at the code to get an idea
> of what might be happening. No data yet.
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: kernel BUG at arch/x86/xen/mmu.c:1860!

[Teck Choon Giam]

[-- Attachment #1.1: Type: text/plain, Size: 2575 bytes --]

On Wed, Jan 26, 2011 at 10:31 PM, Konrad Rzeszutek Wilk <
konrad.wilk@oracle.com> wrote:

> On Tue, Jan 25, 2011 at 09:48:09AM -0500, Konrad Rzeszutek Wilk wrote:
> > On Mon, Jan 24, 2011 at 11:56:21PM +0800, Teck Choon Giam wrote:
> > > On Mon, Jan 24, 2011 at 10:36 PM, Konrad Rzeszutek Wilk <
> > > konrad.wilk@oracle.com> wrote:
> > >
> > > > On Mon, Jan 24, 2011 at 09:42:02AM +0800, Teck Choon Giam wrote:
> > > > > Sorry, does anyone able to solve this bug?  My prevention doing
> sleep 5
> > > > and
> > > > > sync method only last me longest 20+- days of uptime then the same
> bug
> > > > will
> > > > > appear :(
> > > >
> > > > You have to give more details. Which kernel and what does the
> back-trace
> > > > look like?
> > > >
> > >
> > > Thanks for your prompt reply.
> > >
> > > Kernel is from
> > > http://git.kernel.org/?p=linux/kernel/git/jeremy/xen.git;a=summarylatest
> > > stable/2.6.32.x
>
> What does git log show? What is the latest commit you have there?
>

commit 75cc13f5aa29b4f3227d269ca165dfa8937c94fe
Merge: 2607c07 a386bf7
Author: Jeremy Fitzhardinge <jeremy.fitzhardinge@citrix.com>
Date:   Thu Dec 9 17:16:16 2010 -0800

    Merge commit 'v2.6.32.27' into xen/next-2.6.32

    * commit 'v2.6.32.27': (128 commits)
      Linux 2.6.32.27
      x86: uv: xpc NULL deref when mesq becomes empty
      X86: uv: xpc_make_first_contact hang due to not accepting ACTIVE state
      x86: uv: XPC receive message reuse triggers invalid BUG_ON()
      UV - XPC: pass nasid instead of nid to gru_create_message_queue
      net sched: fix some kernel memory leaks
      act_nat: use stack variable
      nmi: fix clock comparator revalidation
      net: Limit socket I/O iovec total length to INT_MAX.
      net: Truncate recvfrom and sendto length to INT_MAX.
      rds: Integer overflow in RDS cmsg handling
      econet: fix CVE-2010-3850
      econet: disallow NULL remote addr for sendmsg(), fixes CVE-2010-3849
      x86-32: Fix dummy trampoline-related inline stubs
      x86, mm: Fix CONFIG_VMSPLIT_1G and 2G_OPT trampoline
      x86-32: Separate 1:1 pagetables from swapper_pg_dir
      crypto: padlock - Fix AES-CBC handling on odd-block-sized input
      x25: Prevent crashing when parsing bad X.25 facilities
      V4L/DVB: ivtvfb: prevent reading uninitialized stack memory
      can-bcm: fix minor heap overflow
      ...

    Conflicts:
        drivers/xen/events.c


Anything that I can test or you need me to test... feel free to ask so that
I can speed up in tracking/hunting this bug down :)

Thanks.

Kindest regards,
Giam Teck Choon

[-- Attachment #1.2: Type: text/html, Size: 3450 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: kernel BUG at arch/x86/xen/mmu.c:1860!

[Konrad Rzeszutek Wilk]

> > What does git log show? What is the latest commit you have there?
> >
> 
> commit 75cc13f5aa29b4f3227d269ca165dfa8937c94fe
> Merge: 2607c07 a386bf7
> Author: Jeremy Fitzhardinge <jeremy.fitzhardinge@citrix.com>
> Date:   Thu Dec 9 17:16:16 2010 -0800
> 
>     Merge commit 'v2.6.32.27' into xen/next-2.6.32
> 
>     * commit 'v2.6.32.27': (128 commits)
>       Linux 2.6.32.27

.. snip..
> 
> Anything that I can test or you need me to test... feel free to ask so that
> I can speed up in tracking/hunting this bug down :)

You got the latest one so you should have the fixes. But something is not
working.. just to make sure I am not missing anything can you send me your .config
file?

Re: kernel BUG at arch/x86/xen/mmu.c:1860!

[Teck Choon Giam]

[-- Attachment #1.1: Type: text/plain, Size: 893 bytes --]

On Fri, Jan 28, 2011 at 4:32 AM, Konrad Rzeszutek Wilk <
konrad.wilk@oracle.com> wrote:

> > > What does git log show? What is the latest commit you have there?
> > >
> >
> > commit 75cc13f5aa29b4f3227d269ca165dfa8937c94fe
> > Merge: 2607c07 a386bf7
> > Author: Jeremy Fitzhardinge <jeremy.fitzhardinge@citrix.com>
> > Date:   Thu Dec 9 17:16:16 2010 -0800
> >
> >     Merge commit 'v2.6.32.27' into xen/next-2.6.32
> >
> >     * commit 'v2.6.32.27': (128 commits)
> >       Linux 2.6.32.27
>
> .. snip..
> >
> > Anything that I can test or you need me to test... feel free to ask so
> that
> > I can speed up in tracking/hunting this bug down :)
>
> You got the latest one so you should have the fixes. But something is not
> working.. just to make sure I am not missing anything can you send me your
> .config
> file?
>

Ok... please see attached.

Thanks.

Kindest regards,
Giam Teck Choon

[-- Attachment #1.2: Type: text/html, Size: 1415 bytes --]

[-- Attachment #2: config-2.6.32.27-0.xen.pvops.choon.centos5 --]
[-- Type: application/octet-stream, Size: 73091 bytes --]

#
# Automatically generated make config: don't edit
# Linux kernel version: 2.6.32.27-0.xen.pvops.choon.centos5
# Sat Dec 25 09:16:08 2010
#
CONFIG_64BIT=y
# CONFIG_X86_32 is not set
CONFIG_X86_64=y
CONFIG_X86=y
CONFIG_OUTPUT_FORMAT="elf64-x86-64"
CONFIG_ARCH_DEFCONFIG="arch/x86/configs/x86_64_defconfig"
CONFIG_GENERIC_TIME=y
CONFIG_GENERIC_CMOS_UPDATE=y
CONFIG_CLOCKSOURCE_WATCHDOG=y
CONFIG_GENERIC_CLOCKEVENTS=y
CONFIG_GENERIC_CLOCKEVENTS_BROADCAST=y
CONFIG_LOCKDEP_SUPPORT=y
CONFIG_STACKTRACE_SUPPORT=y
CONFIG_HAVE_LATENCYTOP_SUPPORT=y
CONFIG_MMU=y
CONFIG_ZONE_DMA=y
CONFIG_GENERIC_ISA_DMA=y
CONFIG_GENERIC_IOMAP=y
CONFIG_GENERIC_BUG=y
CONFIG_GENERIC_BUG_RELATIVE_POINTERS=y
CONFIG_GENERIC_HWEIGHT=y
CONFIG_ARCH_MAY_HAVE_PC_FDC=y
# CONFIG_RWSEM_GENERIC_SPINLOCK is not set
CONFIG_RWSEM_XCHGADD_ALGORITHM=y
CONFIG_ARCH_HAS_CPU_IDLE_WAIT=y
CONFIG_GENERIC_CALIBRATE_DELAY=y
CONFIG_GENERIC_TIME_VSYSCALL=y
CONFIG_ARCH_HAS_CPU_RELAX=y
CONFIG_ARCH_HAS_DEFAULT_IDLE=y
CONFIG_ARCH_HAS_CACHE_LINE_SIZE=y
CONFIG_HAVE_SETUP_PER_CPU_AREA=y
CONFIG_NEED_PER_CPU_EMBED_FIRST_CHUNK=y
CONFIG_NEED_PER_CPU_PAGE_FIRST_CHUNK=y
CONFIG_HAVE_CPUMASK_OF_CPU_MAP=y
CONFIG_ARCH_HIBERNATION_POSSIBLE=y
CONFIG_ARCH_SUSPEND_POSSIBLE=y
CONFIG_ZONE_DMA32=y
CONFIG_ARCH_POPULATES_NODE_MAP=y
CONFIG_AUDIT_ARCH=y
CONFIG_ARCH_SUPPORTS_OPTIMIZED_INLINING=y
CONFIG_ARCH_SUPPORTS_DEBUG_PAGEALLOC=y
CONFIG_GENERIC_HARDIRQS=y
CONFIG_GENERIC_HARDIRQS_NO__DO_IRQ=y
CONFIG_GENERIC_IRQ_PROBE=y
CONFIG_GENERIC_PENDING_IRQ=y
CONFIG_USE_GENERIC_SMP_HELPERS=y
CONFIG_X86_64_SMP=y
CONFIG_X86_HT=y
CONFIG_X86_TRAMPOLINE=y
# CONFIG_KTIME_SCALAR is not set
CONFIG_ARCH_CPU_PROBE_RELEASE=y
CONFIG_DEFCONFIG_LIST="/lib/modules/$UNAME_RELEASE/.config"
CONFIG_CONSTRUCTORS=y

#
# General setup
#
CONFIG_EXPERIMENTAL=y
CONFIG_LOCK_KERNEL=y
CONFIG_INIT_ENV_ARG_LIMIT=32
CONFIG_LOCALVERSION=""
# CONFIG_LOCALVERSION_AUTO is not set
CONFIG_HAVE_KERNEL_GZIP=y
CONFIG_HAVE_KERNEL_BZIP2=y
CONFIG_HAVE_KERNEL_LZMA=y
CONFIG_KERNEL_GZIP=y
# CONFIG_KERNEL_BZIP2 is not set
# CONFIG_KERNEL_LZMA is not set
CONFIG_SWAP=y
CONFIG_SYSVIPC=y
CONFIG_SYSVIPC_SYSCTL=y
CONFIG_POSIX_MQUEUE=y
CONFIG_POSIX_MQUEUE_SYSCTL=y
CONFIG_BSD_PROCESS_ACCT=y
# CONFIG_BSD_PROCESS_ACCT_V3 is not set
CONFIG_TASKSTATS=y
CONFIG_TASK_DELAY_ACCT=y
# CONFIG_TASK_XACCT is not set
CONFIG_AUDIT=y
CONFIG_AUDITSYSCALL=y
CONFIG_AUDIT_TREE=y

#
# RCU Subsystem
#
CONFIG_TREE_RCU=y
# CONFIG_TREE_PREEMPT_RCU is not set
# CONFIG_RCU_TRACE is not set
CONFIG_RCU_FANOUT=64
# CONFIG_RCU_FANOUT_EXACT is not set
# CONFIG_TREE_RCU_TRACE is not set
# CONFIG_IKCONFIG is not set
CONFIG_LOG_BUF_SHIFT=17
CONFIG_HAVE_UNSTABLE_SCHED_CLOCK=y
# CONFIG_GROUP_SCHED is not set
# CONFIG_CGROUPS is not set
CONFIG_SYSFS_DEPRECATED=y
CONFIG_SYSFS_DEPRECATED_V2=y
CONFIG_RELAY=y
CONFIG_NAMESPACES=y
CONFIG_UTS_NS=y
CONFIG_IPC_NS=y
# CONFIG_USER_NS is not set
# CONFIG_PID_NS is not set
# CONFIG_NET_NS is not set
CONFIG_BLK_DEV_INITRD=y
CONFIG_INITRAMFS_SOURCE=""
CONFIG_RD_GZIP=y
CONFIG_RD_BZIP2=y
CONFIG_RD_LZMA=y
CONFIG_CC_OPTIMIZE_FOR_SIZE=y
CONFIG_SYSCTL=y
CONFIG_ANON_INODES=y
# CONFIG_EMBEDDED is not set
CONFIG_UID16=y
CONFIG_SYSCTL_SYSCALL=y
CONFIG_KALLSYMS=y
CONFIG_KALLSYMS_ALL=y
CONFIG_KALLSYMS_EXTRA_PASS=y
CONFIG_HOTPLUG=y
CONFIG_PRINTK=y
CONFIG_BUG=y
CONFIG_ELF_CORE=y
CONFIG_PCSPKR_PLATFORM=y
CONFIG_BASE_FULL=y
CONFIG_FUTEX=y
CONFIG_EPOLL=y
CONFIG_SIGNALFD=y
CONFIG_TIMERFD=y
CONFIG_EVENTFD=y
CONFIG_SHMEM=y
CONFIG_AIO=y
CONFIG_HAVE_PERF_EVENTS=y

#
# Kernel Performance Events And Counters
#
CONFIG_PERF_EVENTS=y
CONFIG_PERF_COUNTERS=y
# CONFIG_DEBUG_PERF_USE_VMALLOC is not set
CONFIG_VM_EVENT_COUNTERS=y
CONFIG_PCI_QUIRKS=y
# CONFIG_COMPAT_BRK is not set
CONFIG_SLAB=y
# CONFIG_SLUB is not set
# CONFIG_SLOB is not set
CONFIG_PROFILING=y
CONFIG_OPROFILE=m
# CONFIG_OPROFILE_IBS is not set
# CONFIG_OPROFILE_EVENT_MULTIPLEX is not set
CONFIG_HAVE_OPROFILE=y
CONFIG_KPROBES=y
CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS=y
CONFIG_KRETPROBES=y
CONFIG_HAVE_IOREMAP_PROT=y
CONFIG_HAVE_KPROBES=y
CONFIG_HAVE_KRETPROBES=y
CONFIG_HAVE_ARCH_TRACEHOOK=y
CONFIG_HAVE_DMA_ATTRS=y
CONFIG_HAVE_DMA_API_DEBUG=y

#
# GCOV-based kernel profiling
#
# CONFIG_GCOV_KERNEL is not set
CONFIG_SLOW_WORK=y
# CONFIG_SLOW_WORK_DEBUG is not set
# CONFIG_HAVE_GENERIC_DMA_COHERENT is not set
CONFIG_SLABINFO=y
CONFIG_RT_MUTEXES=y
CONFIG_BASE_SMALL=0
CONFIG_MODULES=y
# CONFIG_MODULE_FORCE_LOAD is not set
CONFIG_MODULE_UNLOAD=y
# CONFIG_MODULE_FORCE_UNLOAD is not set
CONFIG_MODVERSIONS=y
CONFIG_MODULE_SRCVERSION_ALL=y
CONFIG_STOP_MACHINE=y
CONFIG_BLOCK=y
# CONFIG_BLK_DEV_BSG is not set
# CONFIG_BLK_DEV_INTEGRITY is not set
CONFIG_BLOCK_COMPAT=y

#
# IO Schedulers
#
CONFIG_IOSCHED_NOOP=y
CONFIG_IOSCHED_AS=y
CONFIG_IOSCHED_DEADLINE=y
CONFIG_IOSCHED_CFQ=y
# CONFIG_DEFAULT_AS is not set
# CONFIG_DEFAULT_DEADLINE is not set
CONFIG_DEFAULT_CFQ=y
# CONFIG_DEFAULT_NOOP is not set
CONFIG_DEFAULT_IOSCHED="cfq"
CONFIG_FREEZER=y

#
# Processor type and features
#
CONFIG_TICK_ONESHOT=y
CONFIG_NO_HZ=y
CONFIG_HIGH_RES_TIMERS=y
CONFIG_GENERIC_CLOCKEVENTS_BUILD=y
CONFIG_SMP=y
# CONFIG_SPARSE_IRQ is not set
CONFIG_X86_MPPARSE=y
CONFIG_X86_EXTENDED_PLATFORM=y
# CONFIG_X86_VSMP is not set
CONFIG_X86_SUPPORTS_MEMORY_FAILURE=y
CONFIG_SCHED_OMIT_FRAME_POINTER=y
CONFIG_PARAVIRT_GUEST=y
CONFIG_XEN=y
CONFIG_XEN_PVHVM=y
CONFIG_XEN_MAX_DOMAIN_MEMORY=128
CONFIG_XEN_SAVE_RESTORE=y
# CONFIG_XEN_DEBUG_FS is not set
CONFIG_SWIOTLB_XEN=y
CONFIG_MICROCODE_XEN=y
CONFIG_XEN_DOM0=y
CONFIG_XEN_PRIVILEGED_GUEST=y
CONFIG_XEN_DOM0_PCI=y
CONFIG_XEN_PCI_PASSTHROUGH=y
# CONFIG_KVM_CLOCK is not set
# CONFIG_KVM_GUEST is not set
CONFIG_PARAVIRT=y
# CONFIG_PARAVIRT_SPINLOCKS is not set
CONFIG_PARAVIRT_CLOCK=y
# CONFIG_PARAVIRT_DEBUG is not set
CONFIG_MEMTEST=y
# CONFIG_M386 is not set
# CONFIG_M486 is not set
# CONFIG_M586 is not set
# CONFIG_M586TSC is not set
# CONFIG_M586MMX is not set
# CONFIG_M686 is not set
# CONFIG_MPENTIUMII is not set
# CONFIG_MPENTIUMIII is not set
# CONFIG_MPENTIUMM is not set
# CONFIG_MPENTIUM4 is not set
# CONFIG_MK6 is not set
# CONFIG_MK7 is not set
# CONFIG_MK8 is not set
# CONFIG_MCRUSOE is not set
# CONFIG_MEFFICEON is not set
# CONFIG_MWINCHIPC6 is not set
# CONFIG_MWINCHIP3D is not set
# CONFIG_MGEODEGX1 is not set
# CONFIG_MGEODE_LX is not set
# CONFIG_MCYRIXIII is not set
# CONFIG_MVIAC3_2 is not set
# CONFIG_MVIAC7 is not set
# CONFIG_MPSC is not set
# CONFIG_MCORE2 is not set
# CONFIG_MATOM is not set
CONFIG_GENERIC_CPU=y
CONFIG_X86_CPU=y
CONFIG_X86_L1_CACHE_BYTES=64
CONFIG_X86_INTERNODE_CACHE_BYTES=64
CONFIG_X86_CMPXCHG=y
CONFIG_X86_L1_CACHE_SHIFT=6
CONFIG_X86_XADD=y
CONFIG_X86_WP_WORKS_OK=y
CONFIG_X86_TSC=y
CONFIG_X86_CMPXCHG64=y
CONFIG_X86_CMOV=y
CONFIG_X86_MINIMUM_CPU_FAMILY=64
CONFIG_X86_DEBUGCTLMSR=y
CONFIG_CPU_SUP_INTEL=y
CONFIG_CPU_SUP_AMD=y
CONFIG_CPU_SUP_CENTAUR=y
# CONFIG_X86_DS is not set
CONFIG_HPET_TIMER=y
CONFIG_HPET_EMULATE_RTC=y
CONFIG_DMI=y
CONFIG_GART_IOMMU=y
CONFIG_CALGARY_IOMMU=y
# CONFIG_CALGARY_IOMMU_ENABLED_BY_DEFAULT is not set
CONFIG_AMD_IOMMU=y
# CONFIG_AMD_IOMMU_STATS is not set
CONFIG_SWIOTLB=y
CONFIG_IOMMU_HELPER=y
CONFIG_IOMMU_API=y
CONFIG_MAXSMP=y
CONFIG_NR_CPUS=4096
CONFIG_SCHED_SMT=y
CONFIG_SCHED_MC=y
# CONFIG_PREEMPT_NONE is not set
CONFIG_PREEMPT_VOLUNTARY=y
# CONFIG_PREEMPT is not set
CONFIG_X86_LOCAL_APIC=y
CONFIG_X86_IO_APIC=y
CONFIG_X86_REROUTE_FOR_BROKEN_BOOT_IRQS=y
CONFIG_X86_MCE=y
CONFIG_X86_MCE_INTEL=y
CONFIG_X86_MCE_AMD=y
CONFIG_X86_MCE_THRESHOLD=y
# CONFIG_X86_MCE_INJECT is not set
CONFIG_X86_THERMAL_VECTOR=y
# CONFIG_I8K is not set
CONFIG_MICROCODE=m
CONFIG_MICROCODE_INTEL=y
CONFIG_MICROCODE_AMD=y
CONFIG_MICROCODE_OLD_INTERFACE=y
CONFIG_X86_MSR=y
CONFIG_X86_CPUID=y
CONFIG_ARCH_PHYS_ADDR_T_64BIT=y
CONFIG_DIRECT_GBPAGES=y
CONFIG_NUMA=y
CONFIG_K8_NUMA=y
CONFIG_X86_64_ACPI_NUMA=y
CONFIG_NODES_SPAN_OTHER_NODES=y
# CONFIG_NUMA_EMU is not set
CONFIG_NODES_SHIFT=9
CONFIG_ARCH_PROC_KCORE_TEXT=y
CONFIG_ARCH_SPARSEMEM_DEFAULT=y
CONFIG_ARCH_SPARSEMEM_ENABLE=y
CONFIG_ARCH_SELECT_MEMORY_MODEL=y
CONFIG_ARCH_MEMORY_PROBE=y
CONFIG_ILLEGAL_POINTER_VALUE=0xdead000000000000
CONFIG_SELECT_MEMORY_MODEL=y
# CONFIG_FLATMEM_MANUAL is not set
# CONFIG_DISCONTIGMEM_MANUAL is not set
CONFIG_SPARSEMEM_MANUAL=y
CONFIG_SPARSEMEM=y
CONFIG_NEED_MULTIPLE_NODES=y
CONFIG_HAVE_MEMORY_PRESENT=y
CONFIG_SPARSEMEM_EXTREME=y
CONFIG_SPARSEMEM_VMEMMAP_ENABLE=y
CONFIG_SPARSEMEM_VMEMMAP=y
CONFIG_MEMORY_HOTPLUG=y
CONFIG_MEMORY_HOTPLUG_SPARSE=y
# CONFIG_MEMORY_HOTREMOVE is not set
CONFIG_PAGEFLAGS_EXTENDED=y
CONFIG_SPLIT_PTLOCK_CPUS=4
CONFIG_MIGRATION=y
CONFIG_PHYS_ADDR_T_64BIT=y
CONFIG_ZONE_DMA_FLAG=1
CONFIG_BOUNCE=y
CONFIG_VIRT_TO_BUS=y
CONFIG_HAVE_MLOCK=y
CONFIG_HAVE_MLOCKED_PAGE_BIT=y
CONFIG_MMU_NOTIFIER=y
CONFIG_KSM=y
CONFIG_DEFAULT_MMAP_MIN_ADDR=4096
CONFIG_ARCH_SUPPORTS_MEMORY_FAILURE=y
# CONFIG_MEMORY_FAILURE is not set
CONFIG_X86_CHECK_BIOS_CORRUPTION=y
# CONFIG_X86_BOOTPARAM_MEMORY_CORRUPTION_CHECK is not set
# CONFIG_X86_RESERVE_LOW_64K is not set
CONFIG_MTRR=y
# CONFIG_MTRR_SANITIZER is not set
CONFIG_X86_PAT=y
CONFIG_ARCH_USES_PG_UNCACHED=y
# CONFIG_EFI is not set
# CONFIG_SECCOMP is not set
# CONFIG_CC_STACKPROTECTOR is not set
# CONFIG_HZ_100 is not set
# CONFIG_HZ_250 is not set
# CONFIG_HZ_300 is not set
CONFIG_HZ_1000=y
CONFIG_HZ=1000
CONFIG_SCHED_HRTICK=y
CONFIG_KEXEC=y
CONFIG_CRASH_DUMP=y
CONFIG_PHYSICAL_START=0x200000
CONFIG_RELOCATABLE=y
CONFIG_PHYSICAL_ALIGN=0x1000000
CONFIG_HOTPLUG_CPU=y
# CONFIG_COMPAT_VDSO is not set
# CONFIG_CMDLINE_BOOL is not set
CONFIG_ARCH_ENABLE_MEMORY_HOTPLUG=y
CONFIG_ARCH_ENABLE_MEMORY_HOTREMOVE=y
CONFIG_HAVE_ARCH_EARLY_PFN_TO_NID=y

#
# Power management and ACPI options
#
CONFIG_PM=y
# CONFIG_PM_DEBUG is not set
CONFIG_PM_SLEEP_SMP=y
CONFIG_PM_SLEEP=y
# CONFIG_SUSPEND is not set
# CONFIG_HIBERNATION is not set
# CONFIG_PM_RUNTIME is not set
CONFIG_ACPI=y
CONFIG_ACPI_PROCFS=y
# CONFIG_ACPI_PROCFS_POWER is not set
CONFIG_ACPI_POWER_METER=m
# CONFIG_ACPI_SYSFS_POWER is not set
CONFIG_ACPI_PROC_EVENT=y
CONFIG_ACPI_AC=m
CONFIG_ACPI_BATTERY=m
CONFIG_ACPI_BUTTON=m
CONFIG_ACPI_VIDEO=m
CONFIG_ACPI_FAN=y
CONFIG_ACPI_DOCK=y
CONFIG_ACPI_PROCESSOR=y
CONFIG_ACPI_HOTPLUG_CPU=y
CONFIG_ACPI_PROCESSOR_AGGREGATOR=m
CONFIG_ACPI_THERMAL=y
CONFIG_ACPI_NUMA=y
# CONFIG_ACPI_CUSTOM_DSDT is not set
CONFIG_ACPI_BLACKLIST_YEAR=0
# CONFIG_ACPI_DEBUG is not set
# CONFIG_ACPI_PCI_SLOT is not set
CONFIG_X86_PM_TIMER=y
CONFIG_ACPI_CONTAINER=y
CONFIG_ACPI_HOTPLUG_MEMORY=m
CONFIG_ACPI_SBS=m
# CONFIG_SFI is not set

#
# CPU Frequency scaling
#
CONFIG_CPU_FREQ=y
CONFIG_CPU_FREQ_TABLE=y
CONFIG_CPU_FREQ_DEBUG=y
CONFIG_CPU_FREQ_STAT=m
CONFIG_CPU_FREQ_STAT_DETAILS=y
# CONFIG_CPU_FREQ_DEFAULT_GOV_PERFORMANCE is not set
# CONFIG_CPU_FREQ_DEFAULT_GOV_POWERSAVE is not set
CONFIG_CPU_FREQ_DEFAULT_GOV_USERSPACE=y
# CONFIG_CPU_FREQ_DEFAULT_GOV_ONDEMAND is not set
# CONFIG_CPU_FREQ_DEFAULT_GOV_CONSERVATIVE is not set
CONFIG_CPU_FREQ_GOV_PERFORMANCE=y
CONFIG_CPU_FREQ_GOV_POWERSAVE=m
CONFIG_CPU_FREQ_GOV_USERSPACE=y
CONFIG_CPU_FREQ_GOV_ONDEMAND=m
CONFIG_CPU_FREQ_GOV_CONSERVATIVE=m

#
# CPUFreq processor drivers
#
CONFIG_X86_ACPI_CPUFREQ=m
CONFIG_X86_POWERNOW_K8=y
CONFIG_X86_SPEEDSTEP_CENTRINO=y
CONFIG_X86_P4_CLOCKMOD=m

#
# shared options
#
CONFIG_X86_SPEEDSTEP_LIB=m
CONFIG_CPU_IDLE=y
CONFIG_CPU_IDLE_GOV_LADDER=y
CONFIG_CPU_IDLE_GOV_MENU=y

#
# Memory power savings
#
# CONFIG_I7300_IDLE is not set

#
# Bus options (PCI etc.)
#
CONFIG_PCI=y
CONFIG_PCI_DIRECT=y
CONFIG_PCI_MMCONFIG=y
CONFIG_PCI_XEN=y
CONFIG_PCI_DOMAINS=y
# CONFIG_DMAR is not set
# CONFIG_INTR_REMAP is not set
CONFIG_PCIEPORTBUS=y
CONFIG_HOTPLUG_PCI_PCIE=m
# CONFIG_PCIEAER is not set
# CONFIG_PCIEASPM is not set
CONFIG_ARCH_SUPPORTS_MSI=y
CONFIG_PCI_MSI=y
CONFIG_PCI_LEGACY=y
# CONFIG_PCI_DEBUG is not set
# CONFIG_PCI_STUB is not set
CONFIG_XEN_PCIDEV_FRONTEND=y
# CONFIG_HT_IRQ is not set
# CONFIG_PCI_IOV is not set
CONFIG_ISA_DMA_API=y
CONFIG_K8_NB=y
CONFIG_PCCARD=y
# CONFIG_PCMCIA_DEBUG is not set
CONFIG_PCMCIA=y
CONFIG_PCMCIA_LOAD_CIS=y
CONFIG_PCMCIA_IOCTL=y
CONFIG_CARDBUS=y

#
# PC-card bridges
#
CONFIG_YENTA=y
CONFIG_YENTA_O2=y
CONFIG_YENTA_RICOH=y
CONFIG_YENTA_TI=y
CONFIG_YENTA_ENE_TUNE=y
CONFIG_YENTA_TOSHIBA=y
CONFIG_PD6729=m
# CONFIG_I82092 is not set
CONFIG_PCCARD_NONSTATIC=y
CONFIG_HOTPLUG_PCI=y
CONFIG_HOTPLUG_PCI_FAKE=m
CONFIG_HOTPLUG_PCI_ACPI=m
CONFIG_HOTPLUG_PCI_ACPI_IBM=m
# CONFIG_HOTPLUG_PCI_CPCI is not set
CONFIG_HOTPLUG_PCI_SHPC=m

#
# Executable file formats / Emulations
#
CONFIG_BINFMT_ELF=y
CONFIG_COMPAT_BINFMT_ELF=y
# CONFIG_CORE_DUMP_DEFAULT_ELF_HEADERS is not set
# CONFIG_HAVE_AOUT is not set
CONFIG_BINFMT_MISC=y
CONFIG_IA32_EMULATION=y
# CONFIG_IA32_AOUT is not set
CONFIG_COMPAT=y
CONFIG_COMPAT_FOR_U64_ALIGNMENT=y
CONFIG_SYSVIPC_COMPAT=y
CONFIG_NET=y

#
# Networking options
#
CONFIG_PACKET=y
CONFIG_PACKET_MMAP=y
CONFIG_UNIX=y
CONFIG_XFRM=y
CONFIG_XFRM_USER=y
# CONFIG_XFRM_SUB_POLICY is not set
# CONFIG_XFRM_MIGRATE is not set
# CONFIG_XFRM_STATISTICS is not set
CONFIG_XFRM_IPCOMP=m
CONFIG_NET_KEY=m
# CONFIG_NET_KEY_MIGRATE is not set
CONFIG_INET=y
CONFIG_IP_MULTICAST=y
CONFIG_IP_ADVANCED_ROUTER=y
CONFIG_ASK_IP_FIB_HASH=y
# CONFIG_IP_FIB_TRIE is not set
CONFIG_IP_FIB_HASH=y
CONFIG_IP_MULTIPLE_TABLES=y
CONFIG_IP_ROUTE_MULTIPATH=y
CONFIG_IP_ROUTE_VERBOSE=y
# CONFIG_IP_PNP is not set
CONFIG_NET_IPIP=m
CONFIG_NET_IPGRE=m
CONFIG_NET_IPGRE_BROADCAST=y
CONFIG_IP_MROUTE=y
CONFIG_IP_PIMSM_V1=y
CONFIG_IP_PIMSM_V2=y
# CONFIG_ARPD is not set
CONFIG_SYN_COOKIES=y
CONFIG_INET_AH=m
CONFIG_INET_ESP=m
CONFIG_INET_IPCOMP=m
CONFIG_INET_XFRM_TUNNEL=m
CONFIG_INET_TUNNEL=m
CONFIG_INET_XFRM_MODE_TRANSPORT=m
CONFIG_INET_XFRM_MODE_TUNNEL=m
CONFIG_INET_XFRM_MODE_BEET=m
CONFIG_INET_LRO=y
CONFIG_INET_DIAG=m
CONFIG_INET_TCP_DIAG=m
CONFIG_TCP_CONG_ADVANCED=y
CONFIG_TCP_CONG_BIC=y
CONFIG_TCP_CONG_CUBIC=m
CONFIG_TCP_CONG_WESTWOOD=m
CONFIG_TCP_CONG_HTCP=m
CONFIG_TCP_CONG_HSTCP=m
CONFIG_TCP_CONG_HYBLA=m
CONFIG_TCP_CONG_VEGAS=m
CONFIG_TCP_CONG_SCALABLE=m
CONFIG_TCP_CONG_LP=m
CONFIG_TCP_CONG_VENO=m
CONFIG_TCP_CONG_YEAH=m
CONFIG_TCP_CONG_ILLINOIS=m
CONFIG_DEFAULT_BIC=y
# CONFIG_DEFAULT_CUBIC is not set
# CONFIG_DEFAULT_HTCP is not set
# CONFIG_DEFAULT_VEGAS is not set
# CONFIG_DEFAULT_WESTWOOD is not set
# CONFIG_DEFAULT_RENO is not set
CONFIG_DEFAULT_TCP_CONG="bic"
# CONFIG_TCP_MD5SIG is not set
CONFIG_IPV6=m
CONFIG_IPV6_PRIVACY=y
CONFIG_IPV6_ROUTER_PREF=y
CONFIG_IPV6_ROUTE_INFO=y
CONFIG_IPV6_OPTIMISTIC_DAD=y
CONFIG_INET6_AH=m
CONFIG_INET6_ESP=m
CONFIG_INET6_IPCOMP=m
# CONFIG_IPV6_MIP6 is not set
CONFIG_INET6_XFRM_TUNNEL=m
CONFIG_INET6_TUNNEL=m
CONFIG_INET6_XFRM_MODE_TRANSPORT=m
CONFIG_INET6_XFRM_MODE_TUNNEL=m
CONFIG_INET6_XFRM_MODE_BEET=m
# CONFIG_INET6_XFRM_MODE_ROUTEOPTIMIZATION is not set
CONFIG_IPV6_SIT=m
CONFIG_IPV6_NDISC_NODETYPE=y
CONFIG_IPV6_TUNNEL=m
CONFIG_IPV6_MULTIPLE_TABLES=y
# CONFIG_IPV6_SUBTREES is not set
# CONFIG_IPV6_MROUTE is not set
CONFIG_NETLABEL=y
CONFIG_NETWORK_SECMARK=y
CONFIG_NETFILTER=y
# CONFIG_NETFILTER_DEBUG is not set
CONFIG_NETFILTER_ADVANCED=y
CONFIG_BRIDGE_NETFILTER=y

#
# Core Netfilter Configuration
#
CONFIG_NETFILTER_NETLINK=m
CONFIG_NETFILTER_NETLINK_QUEUE=m
CONFIG_NETFILTER_NETLINK_LOG=m
CONFIG_NF_CONNTRACK=m
CONFIG_NF_CT_ACCT=y
CONFIG_NF_CONNTRACK_MARK=y
CONFIG_NF_CONNTRACK_SECMARK=y
# CONFIG_NF_CONNTRACK_EVENTS is not set
# CONFIG_NF_CT_PROTO_DCCP is not set
CONFIG_NF_CT_PROTO_GRE=m
# CONFIG_NF_CT_PROTO_SCTP is not set
# CONFIG_NF_CT_PROTO_UDPLITE is not set
CONFIG_NF_CONNTRACK_AMANDA=m
CONFIG_NF_CONNTRACK_FTP=m
# CONFIG_NF_CONNTRACK_H323 is not set
CONFIG_NF_CONNTRACK_IRC=m
# CONFIG_NF_CONNTRACK_NETBIOS_NS is not set
CONFIG_NF_CONNTRACK_PPTP=m
# CONFIG_NF_CONNTRACK_SANE is not set
# CONFIG_NF_CONNTRACK_SIP is not set
CONFIG_NF_CONNTRACK_TFTP=m
# CONFIG_NF_CT_NETLINK is not set
# CONFIG_NETFILTER_TPROXY is not set
CONFIG_NETFILTER_XTABLES=m
CONFIG_NETFILTER_XT_TARGET_CLASSIFY=m
CONFIG_NETFILTER_XT_TARGET_CONNMARK=m
CONFIG_NETFILTER_XT_TARGET_CONNSECMARK=m
CONFIG_NETFILTER_XT_TARGET_DSCP=m
CONFIG_NETFILTER_XT_TARGET_HL=m
CONFIG_NETFILTER_XT_TARGET_LED=m
CONFIG_NETFILTER_XT_TARGET_MARK=m
CONFIG_NETFILTER_XT_TARGET_NFLOG=m
CONFIG_NETFILTER_XT_TARGET_NFQUEUE=m
CONFIG_NETFILTER_XT_TARGET_NOTRACK=m
CONFIG_NETFILTER_XT_TARGET_RATEEST=m
CONFIG_NETFILTER_XT_TARGET_TRACE=m
CONFIG_NETFILTER_XT_TARGET_SECMARK=m
CONFIG_NETFILTER_XT_TARGET_TCPMSS=m
# CONFIG_NETFILTER_XT_TARGET_TCPOPTSTRIP is not set
CONFIG_NETFILTER_XT_MATCH_CLUSTER=m
CONFIG_NETFILTER_XT_MATCH_COMMENT=m
CONFIG_NETFILTER_XT_MATCH_CONNBYTES=m
CONFIG_NETFILTER_XT_MATCH_CONNLIMIT=m
CONFIG_NETFILTER_XT_MATCH_CONNMARK=m
CONFIG_NETFILTER_XT_MATCH_CONNTRACK=m
CONFIG_NETFILTER_XT_MATCH_DCCP=m
CONFIG_NETFILTER_XT_MATCH_DSCP=m
CONFIG_NETFILTER_XT_MATCH_ESP=m
CONFIG_NETFILTER_XT_MATCH_HASHLIMIT=m
CONFIG_NETFILTER_XT_MATCH_HELPER=m
CONFIG_NETFILTER_XT_MATCH_HL=m
CONFIG_NETFILTER_XT_MATCH_IPRANGE=m
CONFIG_NETFILTER_XT_MATCH_LENGTH=m
CONFIG_NETFILTER_XT_MATCH_LIMIT=m
CONFIG_NETFILTER_XT_MATCH_MAC=m
CONFIG_NETFILTER_XT_MATCH_MARK=m
CONFIG_NETFILTER_XT_MATCH_MULTIPORT=m
CONFIG_NETFILTER_XT_MATCH_OWNER=m
CONFIG_NETFILTER_XT_MATCH_POLICY=m
CONFIG_NETFILTER_XT_MATCH_PHYSDEV=m
CONFIG_NETFILTER_XT_MATCH_PKTTYPE=m
CONFIG_NETFILTER_XT_MATCH_QUOTA=m
CONFIG_NETFILTER_XT_MATCH_RATEEST=m
CONFIG_NETFILTER_XT_MATCH_REALM=m
CONFIG_NETFILTER_XT_MATCH_RECENT=m
CONFIG_NETFILTER_XT_MATCH_RECENT_PROC_COMPAT=y
CONFIG_NETFILTER_XT_MATCH_SCTP=m
CONFIG_NETFILTER_XT_MATCH_STATE=m
CONFIG_NETFILTER_XT_MATCH_STATISTIC=m
CONFIG_NETFILTER_XT_MATCH_STRING=m
CONFIG_NETFILTER_XT_MATCH_TCPMSS=m
CONFIG_NETFILTER_XT_MATCH_TIME=m
CONFIG_NETFILTER_XT_MATCH_U32=m
CONFIG_NETFILTER_XT_MATCH_OSF=m
CONFIG_IP_VS=m
# CONFIG_IP_VS_IPV6 is not set
# CONFIG_IP_VS_DEBUG is not set
CONFIG_IP_VS_TAB_BITS=12

#
# IPVS transport protocol load balancing support
#
CONFIG_IP_VS_PROTO_TCP=y
CONFIG_IP_VS_PROTO_UDP=y
CONFIG_IP_VS_PROTO_AH_ESP=y
CONFIG_IP_VS_PROTO_ESP=y
CONFIG_IP_VS_PROTO_AH=y

#
# IPVS scheduler
#
CONFIG_IP_VS_RR=m
CONFIG_IP_VS_WRR=m
CONFIG_IP_VS_LC=m
CONFIG_IP_VS_WLC=m
CONFIG_IP_VS_LBLC=m
CONFIG_IP_VS_LBLCR=m
CONFIG_IP_VS_DH=m
CONFIG_IP_VS_SH=m
CONFIG_IP_VS_SED=m
CONFIG_IP_VS_NQ=m

#
# IPVS application helper
#
CONFIG_IP_VS_FTP=m

#
# IP: Netfilter Configuration
#
CONFIG_NF_DEFRAG_IPV4=m
CONFIG_NF_CONNTRACK_IPV4=m
CONFIG_NF_CONNTRACK_PROC_COMPAT=y
CONFIG_IP_NF_QUEUE=m
CONFIG_IP_NF_IPTABLES=m
CONFIG_IP_NF_MATCH_ADDRTYPE=m
CONFIG_IP_NF_MATCH_AH=m
CONFIG_IP_NF_MATCH_ECN=m
CONFIG_IP_NF_MATCH_TTL=m
CONFIG_IP_NF_FILTER=m
CONFIG_IP_NF_TARGET_REJECT=m
CONFIG_IP_NF_TARGET_LOG=m
CONFIG_IP_NF_TARGET_ULOG=m
CONFIG_NF_NAT=m
CONFIG_NF_NAT_NEEDED=y
CONFIG_IP_NF_TARGET_MASQUERADE=m
CONFIG_IP_NF_TARGET_NETMAP=m
CONFIG_IP_NF_TARGET_REDIRECT=m
CONFIG_NF_NAT_SNMP_BASIC=m
CONFIG_NF_NAT_PROTO_GRE=m
CONFIG_NF_NAT_FTP=m
CONFIG_NF_NAT_IRC=m
CONFIG_NF_NAT_TFTP=m
CONFIG_NF_NAT_AMANDA=m
CONFIG_NF_NAT_PPTP=m
# CONFIG_NF_NAT_H323 is not set
# CONFIG_NF_NAT_SIP is not set
CONFIG_IP_NF_MANGLE=m
CONFIG_IP_NF_TARGET_CLUSTERIP=m
CONFIG_IP_NF_TARGET_ECN=m
CONFIG_IP_NF_TARGET_TTL=m
CONFIG_IP_NF_RAW=m
# CONFIG_IP_NF_SECURITY is not set
CONFIG_IP_NF_ARPTABLES=m
CONFIG_IP_NF_ARPFILTER=m
CONFIG_IP_NF_ARP_MANGLE=m

#
# IPv6: Netfilter Configuration
#
CONFIG_NF_CONNTRACK_IPV6=m
CONFIG_IP6_NF_QUEUE=m
CONFIG_IP6_NF_IPTABLES=m
CONFIG_IP6_NF_MATCH_AH=m
CONFIG_IP6_NF_MATCH_EUI64=m
CONFIG_IP6_NF_MATCH_FRAG=m
CONFIG_IP6_NF_MATCH_OPTS=m
CONFIG_IP6_NF_MATCH_HL=m
CONFIG_IP6_NF_MATCH_IPV6HEADER=m
CONFIG_IP6_NF_MATCH_MH=m
CONFIG_IP6_NF_MATCH_RT=m
CONFIG_IP6_NF_TARGET_HL=m
CONFIG_IP6_NF_TARGET_LOG=m
CONFIG_IP6_NF_FILTER=m
CONFIG_IP6_NF_TARGET_REJECT=m
CONFIG_IP6_NF_MANGLE=m
CONFIG_IP6_NF_RAW=m
# CONFIG_IP6_NF_SECURITY is not set
CONFIG_BRIDGE_NF_EBTABLES=m
CONFIG_BRIDGE_EBT_BROUTE=m
CONFIG_BRIDGE_EBT_T_FILTER=m
CONFIG_BRIDGE_EBT_T_NAT=m
CONFIG_BRIDGE_EBT_802_3=m
CONFIG_BRIDGE_EBT_AMONG=m
CONFIG_BRIDGE_EBT_ARP=m
CONFIG_BRIDGE_EBT_IP=m
CONFIG_BRIDGE_EBT_IP6=m
CONFIG_BRIDGE_EBT_LIMIT=m
CONFIG_BRIDGE_EBT_MARK=m
CONFIG_BRIDGE_EBT_PKTTYPE=m
CONFIG_BRIDGE_EBT_STP=m
CONFIG_BRIDGE_EBT_VLAN=m
CONFIG_BRIDGE_EBT_ARPREPLY=m
CONFIG_BRIDGE_EBT_DNAT=m
CONFIG_BRIDGE_EBT_MARK_T=m
CONFIG_BRIDGE_EBT_REDIRECT=m
CONFIG_BRIDGE_EBT_SNAT=m
CONFIG_BRIDGE_EBT_LOG=m
CONFIG_BRIDGE_EBT_ULOG=m
CONFIG_BRIDGE_EBT_NFLOG=m
CONFIG_IP_DCCP=m
CONFIG_INET_DCCP_DIAG=m

#
# DCCP CCIDs Configuration (EXPERIMENTAL)
#
# CONFIG_IP_DCCP_CCID2_DEBUG is not set
# CONFIG_IP_DCCP_CCID3 is not set

#
# DCCP Kernel Hacking
#
# CONFIG_IP_DCCP_DEBUG is not set
# CONFIG_NET_DCCPPROBE is not set
CONFIG_IP_SCTP=m
# CONFIG_SCTP_DBG_MSG is not set
# CONFIG_SCTP_DBG_OBJCNT is not set
# CONFIG_SCTP_HMAC_NONE is not set
# CONFIG_SCTP_HMAC_SHA1 is not set
CONFIG_SCTP_HMAC_MD5=y
# CONFIG_RDS is not set
CONFIG_TIPC=m
# CONFIG_TIPC_ADVANCED is not set
# CONFIG_TIPC_DEBUG is not set
CONFIG_ATM=m
CONFIG_ATM_CLIP=m
# CONFIG_ATM_CLIP_NO_ICMP is not set
CONFIG_ATM_LANE=m
# CONFIG_ATM_MPOA is not set
CONFIG_ATM_BR2684=m
# CONFIG_ATM_BR2684_IPFILTER is not set
CONFIG_STP=m
CONFIG_BRIDGE=m
# CONFIG_NET_DSA is not set
CONFIG_VLAN_8021Q=m
# CONFIG_VLAN_8021Q_GVRP is not set
# CONFIG_DECNET is not set
CONFIG_LLC=y
# CONFIG_LLC2 is not set
# CONFIG_IPX is not set
# CONFIG_ATALK is not set
# CONFIG_X25 is not set
# CONFIG_LAPB is not set
# CONFIG_ECONET is not set
# CONFIG_WAN_ROUTER is not set
# CONFIG_PHONET is not set
# CONFIG_IEEE802154 is not set
CONFIG_NET_SCHED=y

#
# Queueing/Scheduling
#
CONFIG_NET_SCH_CBQ=m
CONFIG_NET_SCH_HTB=m
CONFIG_NET_SCH_HFSC=m
CONFIG_NET_SCH_ATM=m
CONFIG_NET_SCH_PRIO=m
CONFIG_NET_SCH_MULTIQ=m
CONFIG_NET_SCH_RED=m
CONFIG_NET_SCH_SFQ=m
CONFIG_NET_SCH_TEQL=m
CONFIG_NET_SCH_TBF=m
CONFIG_NET_SCH_GRED=m
CONFIG_NET_SCH_DSMARK=m
CONFIG_NET_SCH_NETEM=m
# CONFIG_NET_SCH_DRR is not set
CONFIG_NET_SCH_INGRESS=m
# CONFIG_NET_SCH_PLUG is not set

#
# Classification
#
CONFIG_NET_CLS=y
CONFIG_NET_CLS_BASIC=m
CONFIG_NET_CLS_TCINDEX=m
CONFIG_NET_CLS_ROUTE4=m
CONFIG_NET_CLS_ROUTE=y
CONFIG_NET_CLS_FW=m
CONFIG_NET_CLS_U32=m
CONFIG_CLS_U32_PERF=y
CONFIG_CLS_U32_MARK=y
CONFIG_NET_CLS_RSVP=m
CONFIG_NET_CLS_RSVP6=m
CONFIG_NET_CLS_FLOW=m
CONFIG_NET_EMATCH=y
CONFIG_NET_EMATCH_STACK=32
CONFIG_NET_EMATCH_CMP=m
CONFIG_NET_EMATCH_NBYTE=m
CONFIG_NET_EMATCH_U32=m
CONFIG_NET_EMATCH_META=m
CONFIG_NET_EMATCH_TEXT=m
CONFIG_NET_CLS_ACT=y
CONFIG_NET_ACT_POLICE=m
CONFIG_NET_ACT_GACT=m
CONFIG_GACT_PROB=y
CONFIG_NET_ACT_MIRRED=m
CONFIG_NET_ACT_IPT=m
CONFIG_NET_ACT_NAT=m
CONFIG_NET_ACT_PEDIT=m
CONFIG_NET_ACT_SIMP=m
# CONFIG_NET_ACT_SKBEDIT is not set
CONFIG_NET_CLS_IND=y
CONFIG_NET_SCH_FIFO=y
# CONFIG_DCB is not set

#
# Network testing
#
CONFIG_NET_PKTGEN=m
# CONFIG_NET_TCPPROBE is not set
# CONFIG_HAMRADIO is not set
# CONFIG_CAN is not set
# CONFIG_IRDA is not set
CONFIG_BT=m
CONFIG_BT_L2CAP=m
CONFIG_BT_SCO=m
CONFIG_BT_RFCOMM=m
CONFIG_BT_RFCOMM_TTY=y
CONFIG_BT_BNEP=m
CONFIG_BT_BNEP_MC_FILTER=y
CONFIG_BT_BNEP_PROTO_FILTER=y
CONFIG_BT_HIDP=m

#
# Bluetooth device drivers
#
# CONFIG_BT_HCIBTUSB is not set
CONFIG_BT_HCIUART=m
CONFIG_BT_HCIUART_H4=y
CONFIG_BT_HCIUART_BCSP=y
# CONFIG_BT_HCIUART_LL is not set
CONFIG_BT_HCIBCM203X=m
CONFIG_BT_HCIBPA10X=m
CONFIG_BT_HCIBFUSB=m
CONFIG_BT_HCIDTL1=m
CONFIG_BT_HCIBT3C=m
CONFIG_BT_HCIBLUECARD=m
CONFIG_BT_HCIBTUART=m
CONFIG_BT_HCIVHCI=m
# CONFIG_BT_MRVL is not set
# CONFIG_AF_RXRPC is not set
CONFIG_FIB_RULES=y
# CONFIG_WIRELESS is not set
# CONFIG_WIMAX is not set
CONFIG_RFKILL=m
CONFIG_RFKILL_LEDS=y
CONFIG_RFKILL_INPUT=y
# CONFIG_NET_9P is not set

#
# Device Drivers
#

#
# Generic Driver Options
#
CONFIG_UEVENT_HELPER_PATH="/sbin/hotplug"
# CONFIG_DEVTMPFS is not set
CONFIG_STANDALONE=y
CONFIG_PREVENT_FIRMWARE_BUILD=y
CONFIG_FW_LOADER=y
CONFIG_FIRMWARE_IN_KERNEL=y
CONFIG_EXTRA_FIRMWARE=""
# CONFIG_DEBUG_DRIVER is not set
# CONFIG_DEBUG_DEVRES is not set
CONFIG_SYS_HYPERVISOR=y
CONFIG_CONNECTOR=y
CONFIG_PROC_EVENTS=y
CONFIG_MTD=m
# CONFIG_MTD_DEBUG is not set
# CONFIG_MTD_TESTS is not set
CONFIG_MTD_CONCAT=m
CONFIG_MTD_PARTITIONS=y
CONFIG_MTD_REDBOOT_PARTS=m
CONFIG_MTD_REDBOOT_DIRECTORY_BLOCK=-1
# CONFIG_MTD_REDBOOT_PARTS_UNALLOCATED is not set
# CONFIG_MTD_REDBOOT_PARTS_READONLY is not set
# CONFIG_MTD_AR7_PARTS is not set

#
# User Modules And Translation Layers
#
CONFIG_MTD_CHAR=m
CONFIG_MTD_BLKDEVS=m
CONFIG_MTD_BLOCK=m
CONFIG_MTD_BLOCK_RO=m
CONFIG_FTL=m
CONFIG_NFTL=m
CONFIG_NFTL_RW=y
# CONFIG_INFTL is not set
CONFIG_RFD_FTL=m
# CONFIG_SSFDC is not set
# CONFIG_MTD_OOPS is not set

#
# RAM/ROM/Flash chip drivers
#
CONFIG_MTD_CFI=m
CONFIG_MTD_JEDECPROBE=m
CONFIG_MTD_GEN_PROBE=m
# CONFIG_MTD_CFI_ADV_OPTIONS is not set
CONFIG_MTD_MAP_BANK_WIDTH_1=y
CONFIG_MTD_MAP_BANK_WIDTH_2=y
CONFIG_MTD_MAP_BANK_WIDTH_4=y
# CONFIG_MTD_MAP_BANK_WIDTH_8 is not set
# CONFIG_MTD_MAP_BANK_WIDTH_16 is not set
# CONFIG_MTD_MAP_BANK_WIDTH_32 is not set
CONFIG_MTD_CFI_I1=y
CONFIG_MTD_CFI_I2=y
# CONFIG_MTD_CFI_I4 is not set
# CONFIG_MTD_CFI_I8 is not set
CONFIG_MTD_CFI_INTELEXT=m
CONFIG_MTD_CFI_AMDSTD=m
CONFIG_MTD_CFI_STAA=m
CONFIG_MTD_CFI_UTIL=m
CONFIG_MTD_RAM=m
CONFIG_MTD_ROM=m
CONFIG_MTD_ABSENT=m

#
# Mapping drivers for chip access
#
# CONFIG_MTD_COMPLEX_MAPPINGS is not set
# CONFIG_MTD_PHYSMAP is not set
CONFIG_MTD_SC520CDP=m
CONFIG_MTD_NETSC520=m
CONFIG_MTD_TS5500=m
# CONFIG_MTD_AMD76XROM is not set
# CONFIG_MTD_ICHXROM is not set
# CONFIG_MTD_ESB2ROM is not set
# CONFIG_MTD_CK804XROM is not set
CONFIG_MTD_SCB2_FLASH=m
# CONFIG_MTD_NETtel is not set
# CONFIG_MTD_L440GX is not set
# CONFIG_MTD_INTEL_VR_NOR is not set
# CONFIG_MTD_PLATRAM is not set

#
# Self-contained MTD device drivers
#
# CONFIG_MTD_PMC551 is not set
# CONFIG_MTD_SLRAM is not set
# CONFIG_MTD_PHRAM is not set
CONFIG_MTD_MTDRAM=m
CONFIG_MTDRAM_TOTAL_SIZE=4096
CONFIG_MTDRAM_ERASE_SIZE=128
CONFIG_MTD_BLOCK2MTD=m

#
# Disk-On-Chip Device Drivers
#
# CONFIG_MTD_DOC2000 is not set
# CONFIG_MTD_DOC2001 is not set
# CONFIG_MTD_DOC2001PLUS is not set
CONFIG_MTD_NAND=m
# CONFIG_MTD_NAND_VERIFY_WRITE is not set
CONFIG_MTD_NAND_ECC_SMC=y
# CONFIG_MTD_NAND_MUSEUM_IDS is not set
CONFIG_MTD_NAND_IDS=m
CONFIG_MTD_NAND_DISKONCHIP=m
# CONFIG_MTD_NAND_DISKONCHIP_PROBE_ADVANCED is not set
CONFIG_MTD_NAND_DISKONCHIP_PROBE_ADDRESS=0
# CONFIG_MTD_NAND_DISKONCHIP_BBTWRITE is not set
# CONFIG_MTD_NAND_CAFE is not set
CONFIG_MTD_NAND_NANDSIM=m
# CONFIG_MTD_NAND_PLATFORM is not set
# CONFIG_MTD_ALAUDA is not set
# CONFIG_MTD_ONENAND is not set

#
# LPDDR flash memory drivers
#
# CONFIG_MTD_LPDDR is not set

#
# UBI - Unsorted block images
#
# CONFIG_MTD_UBI is not set
CONFIG_PARPORT=m
CONFIG_PARPORT_PC=m
CONFIG_PARPORT_SERIAL=m
# CONFIG_PARPORT_PC_FIFO is not set
# CONFIG_PARPORT_PC_SUPERIO is not set
CONFIG_PARPORT_PC_PCMCIA=m
# CONFIG_PARPORT_GSC is not set
# CONFIG_PARPORT_AX88796 is not set
CONFIG_PARPORT_1284=y
CONFIG_PARPORT_NOT_PC=y
CONFIG_PNP=y
# CONFIG_PNP_DEBUG_MESSAGES is not set

#
# Protocols
#
CONFIG_PNPACPI=y
CONFIG_BLK_DEV=y
CONFIG_BLK_DEV_FD=m
CONFIG_PARIDE=m

#
# Parallel IDE high-level drivers
#
CONFIG_PARIDE_PD=m
CONFIG_PARIDE_PCD=m
CONFIG_PARIDE_PF=m
CONFIG_PARIDE_PT=m
CONFIG_PARIDE_PG=m

#
# Parallel IDE protocol modules
#
CONFIG_PARIDE_ATEN=m
CONFIG_PARIDE_BPCK=m
CONFIG_PARIDE_COMM=m
CONFIG_PARIDE_DSTR=m
CONFIG_PARIDE_FIT2=m
CONFIG_PARIDE_FIT3=m
CONFIG_PARIDE_EPAT=m
CONFIG_PARIDE_EPATC8=y
CONFIG_PARIDE_EPIA=m
CONFIG_PARIDE_FRIQ=m
CONFIG_PARIDE_FRPW=m
CONFIG_PARIDE_KBIC=m
CONFIG_PARIDE_KTTI=m
CONFIG_PARIDE_ON20=m
CONFIG_PARIDE_ON26=m
CONFIG_BLK_CPQ_DA=m
CONFIG_BLK_CPQ_CISS_DA=m
CONFIG_CISS_SCSI_TAPE=y
CONFIG_BLK_DEV_DAC960=m
# CONFIG_BLK_DEV_UMEM is not set
# CONFIG_BLK_DEV_COW_COMMON is not set
CONFIG_BLK_DEV_LOOP=m
CONFIG_BLK_DEV_CRYPTOLOOP=m
CONFIG_BLK_DEV_NBD=m
CONFIG_BLK_DEV_OSD=m
CONFIG_BLK_DEV_SX8=m
# CONFIG_BLK_DEV_UB is not set
CONFIG_BLK_DEV_RAM=y
CONFIG_BLK_DEV_RAM_COUNT=16
CONFIG_BLK_DEV_RAM_SIZE=16384
# CONFIG_BLK_DEV_XIP is not set
CONFIG_CDROM_PKTCDVD=m
CONFIG_CDROM_PKTCDVD_BUFFERS=8
# CONFIG_CDROM_PKTCDVD_WCACHE is not set
CONFIG_ATA_OVER_ETH=m
CONFIG_XEN_BLKDEV_FRONTEND=y
# CONFIG_BLK_DEV_HD is not set
# CONFIG_MISC_DEVICES is not set
CONFIG_HAVE_IDE=y
CONFIG_IDE=y

#
# Please see Documentation/ide/ide.txt for help/info on IDE drives
#
CONFIG_IDE_XFER_MODE=y
CONFIG_IDE_TIMINGS=y
CONFIG_IDE_ATAPI=y
# CONFIG_BLK_DEV_IDE_SATA is not set
CONFIG_IDE_GD=y
CONFIG_IDE_GD_ATA=y
CONFIG_IDE_GD_ATAPI=y
CONFIG_BLK_DEV_IDECS=m
# CONFIG_BLK_DEV_DELKIN is not set
CONFIG_BLK_DEV_IDECD=m
CONFIG_BLK_DEV_IDECD_VERBOSE_ERRORS=y
# CONFIG_BLK_DEV_IDETAPE is not set
CONFIG_BLK_DEV_IDEACPI=y
CONFIG_IDE_TASK_IOCTL=y
CONFIG_IDE_PROC_FS=y

#
# IDE chipset support/bugfixes
#
CONFIG_IDE_GENERIC=y
# CONFIG_BLK_DEV_PLATFORM is not set
# CONFIG_BLK_DEV_CMD640 is not set
CONFIG_BLK_DEV_IDEPNP=y
CONFIG_BLK_DEV_IDEDMA_SFF=y

#
# PCI IDE chipsets support
#
CONFIG_BLK_DEV_IDEPCI=y
CONFIG_IDEPCI_PCIBUS_ORDER=y
# CONFIG_BLK_DEV_OFFBOARD is not set
CONFIG_BLK_DEV_GENERIC=y
# CONFIG_BLK_DEV_OPTI621 is not set
# CONFIG_BLK_DEV_RZ1000 is not set
CONFIG_BLK_DEV_IDEDMA_PCI=y
CONFIG_BLK_DEV_AEC62XX=y
CONFIG_BLK_DEV_ALI15X3=y
CONFIG_BLK_DEV_AMD74XX=y
CONFIG_BLK_DEV_ATIIXP=y
CONFIG_BLK_DEV_CMD64X=y
# CONFIG_BLK_DEV_TRIFLEX is not set
# CONFIG_BLK_DEV_CS5520 is not set
# CONFIG_BLK_DEV_CS5530 is not set
CONFIG_BLK_DEV_HPT366=y
# CONFIG_BLK_DEV_JMICRON is not set
# CONFIG_BLK_DEV_SC1200 is not set
CONFIG_BLK_DEV_PIIX=y
CONFIG_BLK_DEV_IT8172=m
CONFIG_BLK_DEV_IT8213=m
CONFIG_BLK_DEV_IT821X=y
# CONFIG_BLK_DEV_NS87415 is not set
CONFIG_BLK_DEV_PDC202XX_OLD=y
CONFIG_BLK_DEV_PDC202XX_NEW=y
CONFIG_BLK_DEV_SVWKS=y
CONFIG_BLK_DEV_SIIMAGE=y
CONFIG_BLK_DEV_SIS5513=y
# CONFIG_BLK_DEV_SLC90E66 is not set
# CONFIG_BLK_DEV_TRM290 is not set
CONFIG_BLK_DEV_VIA82CXXX=y
# CONFIG_BLK_DEV_TC86C001 is not set
CONFIG_BLK_DEV_IDEDMA=y

#
# SCSI device support
#
CONFIG_RAID_ATTRS=m
CONFIG_SCSI=m
CONFIG_SCSI_DMA=y
# CONFIG_SCSI_TGT is not set
CONFIG_SCSI_NETLINK=y
CONFIG_SCSI_PROC_FS=y

#
# SCSI support type (disk, tape, CD-ROM)
#
CONFIG_BLK_DEV_SD=m
CONFIG_CHR_DEV_ST=m
CONFIG_CHR_DEV_OSST=m
CONFIG_BLK_DEV_SR=m
CONFIG_BLK_DEV_SR_VENDOR=y
CONFIG_CHR_DEV_SG=m
CONFIG_CHR_DEV_SCH=m
CONFIG_SCSI_MULTI_LUN=y
CONFIG_SCSI_CONSTANTS=y
CONFIG_SCSI_LOGGING=y
# CONFIG_SCSI_SCAN_ASYNC is not set
CONFIG_SCSI_WAIT_SCAN=m

#
# SCSI Transports
#
CONFIG_SCSI_SPI_ATTRS=m
CONFIG_SCSI_FC_ATTRS=m
CONFIG_SCSI_ISCSI_ATTRS=m
CONFIG_SCSI_SAS_ATTRS=m
CONFIG_SCSI_SAS_LIBSAS=m
CONFIG_SCSI_SAS_ATA=y
CONFIG_SCSI_SAS_HOST_SMP=y
# CONFIG_SCSI_SAS_LIBSAS_DEBUG is not set
CONFIG_SCSI_SRP_ATTRS=m
CONFIG_SCSI_LOWLEVEL=y
CONFIG_ISCSI_TCP=m
CONFIG_SCSI_CXGB3_ISCSI=m
CONFIG_SCSI_BNX2_ISCSI=m
CONFIG_BE2ISCSI=m
CONFIG_SCSI_ISCSI_TRGT=m
CONFIG_BLK_DEV_3W_XXXX_RAID=m
CONFIG_SCSI_3W_9XXX=m
CONFIG_SCSI_ACARD=m
CONFIG_SCSI_AACRAID=m
CONFIG_SCSI_AIC7XXX=m
CONFIG_AIC7XXX_CMDS_PER_DEVICE=4
CONFIG_AIC7XXX_RESET_DELAY_MS=15000
# CONFIG_AIC7XXX_DEBUG_ENABLE is not set
CONFIG_AIC7XXX_DEBUG_MASK=0
# CONFIG_AIC7XXX_REG_PRETTY_PRINT is not set
CONFIG_SCSI_AIC7XXX_OLD=m
CONFIG_SCSI_AIC79XX=m
CONFIG_AIC79XX_CMDS_PER_DEVICE=4
CONFIG_AIC79XX_RESET_DELAY_MS=15000
# CONFIG_AIC79XX_DEBUG_ENABLE is not set
CONFIG_AIC79XX_DEBUG_MASK=0
# CONFIG_AIC79XX_REG_PRETTY_PRINT is not set
CONFIG_SCSI_AIC94XX=m
# CONFIG_AIC94XX_DEBUG is not set
CONFIG_SCSI_MVSAS=m
# CONFIG_SCSI_MVSAS_DEBUG is not set
# CONFIG_SCSI_DPT_I2O is not set
# CONFIG_SCSI_ADVANSYS is not set
CONFIG_SCSI_ARCMSR=m
CONFIG_MEGARAID_NEWGEN=y
CONFIG_MEGARAID_MM=m
CONFIG_MEGARAID_MAILBOX=m
CONFIG_MEGARAID_LEGACY=m
CONFIG_MEGARAID_SAS=m
# CONFIG_SCSI_MPT2SAS is not set
CONFIG_SCSI_HPTIOP=m
# CONFIG_SCSI_BUSLOGIC is not set
# CONFIG_LIBFC is not set
# CONFIG_LIBFCOE is not set
# CONFIG_FCOE is not set
# CONFIG_FCOE_FNIC is not set
# CONFIG_SCSI_DMX3191D is not set
# CONFIG_SCSI_EATA is not set
# CONFIG_SCSI_FUTURE_DOMAIN is not set
CONFIG_SCSI_GDTH=m
CONFIG_SCSI_IPS=m
CONFIG_SCSI_INITIO=m
# CONFIG_SCSI_INIA100 is not set
CONFIG_SCSI_PPA=m
CONFIG_SCSI_IMM=m
# CONFIG_SCSI_IZIP_EPP16 is not set
# CONFIG_SCSI_IZIP_SLOW_CTR is not set
CONFIG_SCSI_STEX=m
CONFIG_SCSI_SYM53C8XX_2=m
CONFIG_SCSI_SYM53C8XX_DMA_ADDRESSING_MODE=1
CONFIG_SCSI_SYM53C8XX_DEFAULT_TAGS=16
CONFIG_SCSI_SYM53C8XX_MAX_TAGS=64
CONFIG_SCSI_SYM53C8XX_MMIO=y
# CONFIG_SCSI_IPR is not set
CONFIG_SCSI_QLOGIC_1280=m
CONFIG_SCSI_QLA_FC=m
CONFIG_SCSI_QLA_ISCSI=m
CONFIG_SCSI_LPFC=m
# CONFIG_SCSI_LPFC_DEBUG_FS is not set
CONFIG_SCSI_DC395x=m
# CONFIG_SCSI_DC390T is not set
# CONFIG_SCSI_DEBUG is not set
CONFIG_SCSI_PMCRAID=m
# CONFIG_SCSI_SRP is not set
CONFIG_SCSI_BFA_FC=m
# CONFIG_SCSI_LOWLEVEL_PCMCIA is not set
CONFIG_SCSI_DH=m
# CONFIG_SCSI_DH_RDAC is not set
# CONFIG_SCSI_DH_HP_SW is not set
# CONFIG_SCSI_DH_EMC is not set
# CONFIG_SCSI_DH_ALUA is not set
CONFIG_SCSI_OSD_INITIATOR=m
CONFIG_SCSI_OSD_ULD=m
CONFIG_SCSI_OSD_DPRINT_SENSE=1
# CONFIG_SCSI_OSD_DEBUG is not set
CONFIG_ATA=m
# CONFIG_ATA_NONSTANDARD is not set
CONFIG_ATA_VERBOSE_ERROR=y
CONFIG_ATA_ACPI=y
# CONFIG_SATA_PMP is not set
CONFIG_SATA_AHCI=m
CONFIG_SATA_SIL24=m
CONFIG_ATA_SFF=y
CONFIG_SATA_SVW=m
CONFIG_ATA_PIIX=m
CONFIG_SATA_MV=m
CONFIG_SATA_NV=m
CONFIG_PDC_ADMA=m
CONFIG_SATA_QSTOR=m
CONFIG_SATA_PROMISE=m
CONFIG_SATA_SX4=m
CONFIG_SATA_SIL=m
CONFIG_SATA_SIS=m
CONFIG_SATA_ULI=m
CONFIG_SATA_VIA=m
CONFIG_SATA_VITESSE=m
CONFIG_SATA_INIC162X=m
# CONFIG_PATA_ACPI is not set
# CONFIG_PATA_ALI is not set
# CONFIG_PATA_AMD is not set
# CONFIG_PATA_ARTOP is not set
CONFIG_PATA_ATP867X=m
# CONFIG_PATA_ATIIXP is not set
# CONFIG_PATA_CMD640_PCI is not set
# CONFIG_PATA_CMD64X is not set
# CONFIG_PATA_CS5520 is not set
# CONFIG_PATA_CS5530 is not set
# CONFIG_PATA_CYPRESS is not set
# CONFIG_PATA_EFAR is not set
# CONFIG_ATA_GENERIC is not set
# CONFIG_PATA_HPT366 is not set
# CONFIG_PATA_HPT37X is not set
# CONFIG_PATA_HPT3X2N is not set
# CONFIG_PATA_HPT3X3 is not set
# CONFIG_PATA_IT821X is not set
# CONFIG_PATA_IT8213 is not set
# CONFIG_PATA_JMICRON is not set
# CONFIG_PATA_TRIFLEX is not set
CONFIG_PATA_MARVELL=m
# CONFIG_PATA_MPIIX is not set
# CONFIG_PATA_OLDPIIX is not set
# CONFIG_PATA_NETCELL is not set
# CONFIG_PATA_NINJA32 is not set
# CONFIG_PATA_NS87410 is not set
# CONFIG_PATA_NS87415 is not set
# CONFIG_PATA_OPTI is not set
# CONFIG_PATA_OPTIDMA is not set
# CONFIG_PATA_PCMCIA is not set
# CONFIG_PATA_PDC_OLD is not set
# CONFIG_PATA_RADISYS is not set
CONFIG_PATA_RDC=m
# CONFIG_PATA_RZ1000 is not set
# CONFIG_PATA_SC1200 is not set
# CONFIG_PATA_SERVERWORKS is not set
CONFIG_PATA_PDC2027X=m
# CONFIG_PATA_SIL680 is not set
CONFIG_PATA_SIS=m
# CONFIG_PATA_VIA is not set
# CONFIG_PATA_WINBOND is not set
CONFIG_PATA_SCH=m
CONFIG_MD=y
CONFIG_BLK_DEV_MD=y
CONFIG_MD_AUTODETECT=y
CONFIG_MD_LINEAR=m
CONFIG_MD_RAID0=m
CONFIG_MD_RAID1=m
CONFIG_MD_RAID10=m
CONFIG_MD_RAID456=m
# CONFIG_MULTICORE_RAID456 is not set
CONFIG_MD_RAID6_PQ=m
# CONFIG_ASYNC_RAID6_TEST is not set
CONFIG_MD_MULTIPATH=m
CONFIG_MD_FAULTY=m
CONFIG_BLK_DEV_DM=m
# CONFIG_DM_DEBUG is not set
CONFIG_DM_CRYPT=m
CONFIG_DM_SNAPSHOT=m
CONFIG_DM_MIRROR=m
# CONFIG_DM_LOG_USERSPACE is not set
CONFIG_DM_ZERO=m
CONFIG_DM_MULTIPATH=m
# CONFIG_DM_MULTIPATH_QL is not set
# CONFIG_DM_MULTIPATH_ST is not set
# CONFIG_DM_DELAY is not set
CONFIG_DM_UEVENT=y
CONFIG_FUSION=y
CONFIG_FUSION_SPI=m
CONFIG_FUSION_FC=m
CONFIG_FUSION_SAS=m
CONFIG_FUSION_MAX_SGE=128
CONFIG_FUSION_CTL=m
CONFIG_FUSION_LAN=m
CONFIG_FUSION_LOGGING=y

#
# IEEE 1394 (FireWire) support
#

#
# You can enable one or both FireWire driver stacks.
#

#
# See the help texts for more information.
#
CONFIG_FIREWIRE=m
CONFIG_FIREWIRE_OHCI=m
CONFIG_FIREWIRE_OHCI_DEBUG=y
CONFIG_FIREWIRE_SBP2=m
# CONFIG_FIREWIRE_NET is not set
# CONFIG_IEEE1394 is not set
CONFIG_I2O=m
# CONFIG_I2O_LCT_NOTIFY_ON_CHANGES is not set
CONFIG_I2O_EXT_ADAPTEC=y
CONFIG_I2O_EXT_ADAPTEC_DMA64=y
CONFIG_I2O_CONFIG=m
CONFIG_I2O_CONFIG_OLD_IOCTL=y
CONFIG_I2O_BUS=m
CONFIG_I2O_BLOCK=m
CONFIG_I2O_SCSI=m
CONFIG_I2O_PROC=m
# CONFIG_MACINTOSH_DRIVERS is not set
CONFIG_NETDEVICES=y
CONFIG_IFB=m
CONFIG_DUMMY=m
CONFIG_BONDING=m
# CONFIG_MACVLAN is not set
# CONFIG_EQUALIZER is not set
CONFIG_TUN=m
# CONFIG_VETH is not set
# CONFIG_NET_SB1000 is not set
# CONFIG_ARCNET is not set
CONFIG_PHYLIB=m

#
# MII PHY device drivers
#
CONFIG_MARVELL_PHY=m
CONFIG_DAVICOM_PHY=m
CONFIG_QSEMI_PHY=m
CONFIG_LXT_PHY=m
CONFIG_CICADA_PHY=m
CONFIG_VITESSE_PHY=m
CONFIG_SMSC_PHY=m
CONFIG_BROADCOM_PHY=m
CONFIG_ICPLUS_PHY=m
CONFIG_REALTEK_PHY=m
CONFIG_NATIONAL_PHY=m
CONFIG_STE10XP=m
CONFIG_LSI_ET1011C_PHY=m
# CONFIG_MDIO_BITBANG is not set
CONFIG_NET_ETHERNET=y
CONFIG_MII=m
CONFIG_HAPPYMEAL=m
CONFIG_SUNGEM=m
CONFIG_CASSINI=m
CONFIG_NET_VENDOR_3COM=y
CONFIG_VORTEX=m
CONFIG_TYPHOON=m
CONFIG_ETHOC=m
CONFIG_DNET=m
CONFIG_NET_TULIP=y
CONFIG_DE2104X=m
CONFIG_DE2104X_DSL=0
CONFIG_TULIP=m
# CONFIG_TULIP_MWI is not set
CONFIG_TULIP_MMIO=y
# CONFIG_TULIP_NAPI is not set
CONFIG_DE4X5=m
CONFIG_WINBOND_840=m
CONFIG_DM9102=m
CONFIG_ULI526X=m
CONFIG_PCMCIA_XIRCOM=m
# CONFIG_HP100 is not set
# CONFIG_IBM_NEW_EMAC_ZMII is not set
# CONFIG_IBM_NEW_EMAC_RGMII is not set
# CONFIG_IBM_NEW_EMAC_TAH is not set
# CONFIG_IBM_NEW_EMAC_EMAC4 is not set
# CONFIG_IBM_NEW_EMAC_NO_FLOW_CTRL is not set
# CONFIG_IBM_NEW_EMAC_MAL_CLR_ICINTSTAT is not set
# CONFIG_IBM_NEW_EMAC_MAL_COMMON_ERR is not set
CONFIG_NET_PCI=y
CONFIG_PCNET32=m
CONFIG_AMD8111_ETH=m
CONFIG_ADAPTEC_STARFIRE=m
CONFIG_B44=m
CONFIG_B44_PCI_AUTOSELECT=y
CONFIG_B44_PCICORE_AUTOSELECT=y
CONFIG_B44_PCI=y
CONFIG_FORCEDETH=m
# CONFIG_FORCEDETH_NAPI is not set
CONFIG_E100=m
CONFIG_FEALNX=m
CONFIG_NATSEMI=m
CONFIG_NE2K_PCI=m
CONFIG_8139CP=m
CONFIG_8139TOO=m
# CONFIG_8139TOO_PIO is not set
# CONFIG_8139TOO_TUNE_TWISTER is not set
CONFIG_8139TOO_8129=y
# CONFIG_8139_OLD_RX_RESET is not set
# CONFIG_R6040 is not set
CONFIG_SIS900=m
CONFIG_EPIC100=m
CONFIG_SMSC9420=m
CONFIG_SUNDANCE=m
# CONFIG_SUNDANCE_MMIO is not set
CONFIG_TLAN=m
CONFIG_KS8842=m
CONFIG_KS8851_MLL=m
CONFIG_VIA_RHINE=m
CONFIG_VIA_RHINE_MMIO=y
# CONFIG_SC92031 is not set
CONFIG_NET_POCKET=y
# CONFIG_ATP is not set
# CONFIG_DE600 is not set
# CONFIG_DE620 is not set
CONFIG_ATL2=m
CONFIG_NETDEV_1000=y
CONFIG_ACENIC=m
# CONFIG_ACENIC_OMIT_TIGON_I is not set
CONFIG_DL2K=m
CONFIG_E1000=m
CONFIG_E1000E=m
CONFIG_IP1000=m
CONFIG_IGB=m
CONFIG_IGBVF=m
CONFIG_NS83820=m
# CONFIG_HAMACHI is not set
# CONFIG_YELLOWFIN is not set
CONFIG_R8169=m
CONFIG_R8169_VLAN=y
CONFIG_SIS190=m
CONFIG_SKGE=m
# CONFIG_SKGE_DEBUG is not set
CONFIG_SKY2=m
# CONFIG_SKY2_DEBUG is not set
CONFIG_VIA_VELOCITY=m
CONFIG_TIGON3=m
CONFIG_BNX2=m
CONFIG_CNIC=m
CONFIG_QLA3XXX=m
# CONFIG_ATL1 is not set
# CONFIG_ATL1E is not set
# CONFIG_ATL1C is not set
CONFIG_JME=m
CONFIG_NETDEV_10000=y
CONFIG_MDIO=m
CONFIG_CHELSIO_T1=m
# CONFIG_CHELSIO_T1_1G is not set
CONFIG_CHELSIO_T3_DEPENDS=y
CONFIG_CHELSIO_T3=m
CONFIG_ENIC=m
CONFIG_IXGBE=m
CONFIG_IXGB=m
CONFIG_S2IO=m
CONFIG_VXGE=m
# CONFIG_VXGE_DEBUG_TRACE_ALL is not set
CONFIG_MYRI10GE=m
CONFIG_NETXEN_NIC=m
CONFIG_NIU=m
CONFIG_MLX4_EN=m
CONFIG_MLX4_CORE=m
CONFIG_MLX4_DEBUG=y
CONFIG_TEHUTI=m
CONFIG_BNX2X=m
CONFIG_QLGE=m
# CONFIG_SFC is not set
# CONFIG_BE2NET is not set
CONFIG_TR=y
CONFIG_IBMOL=m
CONFIG_3C359=m
# CONFIG_TMS380TR is not set
CONFIG_WLAN=y
# CONFIG_WLAN_PRE80211 is not set
# CONFIG_WLAN_80211 is not set

#
# Enable WiMAX (Networking options) to see the WiMAX drivers
#

#
# USB Network Adapters
#
CONFIG_USB_CATC=m
CONFIG_USB_KAWETH=m
CONFIG_USB_PEGASUS=m
CONFIG_USB_RTL8150=m
CONFIG_USB_USBNET=m
CONFIG_USB_NET_AX8817X=m
CONFIG_USB_NET_CDCETHER=m
# CONFIG_USB_NET_CDC_EEM is not set
CONFIG_USB_NET_DM9601=m
# CONFIG_USB_NET_SMSC95XX is not set
CONFIG_USB_NET_GL620A=m
CONFIG_USB_NET_NET1080=m
CONFIG_USB_NET_PLUSB=m
# CONFIG_USB_NET_MCS7830 is not set
CONFIG_USB_NET_RNDIS_HOST=m
CONFIG_USB_NET_CDC_SUBSET=m
CONFIG_USB_ALI_M5632=y
CONFIG_USB_AN2720=y
CONFIG_USB_BELKIN=y
CONFIG_USB_ARMLINUX=y
CONFIG_USB_EPSON2888=y
# CONFIG_USB_KC2190 is not set
CONFIG_USB_NET_ZAURUS=m
# CONFIG_USB_HSO is not set
# CONFIG_USB_NET_INT51X1 is not set
CONFIG_NET_PCMCIA=y
CONFIG_PCMCIA_3C589=m
CONFIG_PCMCIA_3C574=m
CONFIG_PCMCIA_FMVJ18X=m
CONFIG_PCMCIA_PCNET=m
CONFIG_PCMCIA_NMCLAN=m
CONFIG_PCMCIA_SMC91C92=m
CONFIG_PCMCIA_XIRC2PS=m
CONFIG_PCMCIA_AXNET=m
CONFIG_PCMCIA_IBMTR=m
# CONFIG_WAN is not set
# CONFIG_ATM_DRIVERS is not set
CONFIG_XEN_NETDEV_FRONTEND=y
CONFIG_FDDI=y
# CONFIG_DEFXX is not set
# CONFIG_SKFP is not set
# CONFIG_HIPPI is not set
# CONFIG_PLIP is not set
CONFIG_PPP=m
CONFIG_PPP_MULTILINK=y
CONFIG_PPP_FILTER=y
CONFIG_PPP_ASYNC=m
CONFIG_PPP_SYNC_TTY=m
CONFIG_PPP_DEFLATE=m
# CONFIG_PPP_BSDCOMP is not set
CONFIG_PPP_MPPE=m
CONFIG_PPPOE=m
CONFIG_PPPOATM=m
# CONFIG_PPPOL2TP is not set
CONFIG_SLIP=m
CONFIG_SLIP_COMPRESSED=y
CONFIG_SLHC=m
CONFIG_SLIP_SMART=y
# CONFIG_SLIP_MODE_SLIP6 is not set
CONFIG_NET_FC=y
CONFIG_NETCONSOLE=m
# CONFIG_NETCONSOLE_DYNAMIC is not set
CONFIG_NETPOLL=y
CONFIG_NETPOLL_TRAP=y
CONFIG_NET_POLL_CONTROLLER=y
# CONFIG_VMXNET3 is not set
# CONFIG_ISDN is not set
# CONFIG_PHONE is not set

#
# Input device support
#
CONFIG_INPUT=y
CONFIG_INPUT_FF_MEMLESS=y
CONFIG_INPUT_POLLDEV=m

#
# Userland interfaces
#
CONFIG_INPUT_MOUSEDEV=y
# CONFIG_INPUT_MOUSEDEV_PSAUX is not set
CONFIG_INPUT_MOUSEDEV_SCREEN_X=1024
CONFIG_INPUT_MOUSEDEV_SCREEN_Y=768
# CONFIG_INPUT_JOYDEV is not set
CONFIG_INPUT_EVDEV=y
# CONFIG_INPUT_EVBUG is not set
CONFIG_XEN_KBDDEV_FRONTEND=y

#
# Input Device Drivers
#
CONFIG_INPUT_KEYBOARD=y
# CONFIG_KEYBOARD_ADP5588 is not set
CONFIG_KEYBOARD_ATKBD=y
# CONFIG_QT2160 is not set
# CONFIG_KEYBOARD_LKKBD is not set
# CONFIG_KEYBOARD_LM8323 is not set
# CONFIG_KEYBOARD_MAX7359 is not set
# CONFIG_KEYBOARD_NEWTON is not set
# CONFIG_KEYBOARD_OPENCORES is not set
# CONFIG_KEYBOARD_STOWAWAY is not set
# CONFIG_KEYBOARD_SUNKBD is not set
# CONFIG_KEYBOARD_XTKBD is not set
CONFIG_INPUT_MOUSE=y
CONFIG_MOUSE_PS2=y
CONFIG_MOUSE_PS2_ALPS=y
CONFIG_MOUSE_PS2_LOGIPS2PP=y
CONFIG_MOUSE_PS2_SYNAPTICS=y
CONFIG_MOUSE_PS2_LIFEBOOK=y
CONFIG_MOUSE_PS2_TRACKPOINT=y
# CONFIG_MOUSE_PS2_ELANTECH is not set
# CONFIG_MOUSE_PS2_SENTELIC is not set
# CONFIG_MOUSE_PS2_TOUCHKIT is not set
CONFIG_MOUSE_SERIAL=m
# CONFIG_MOUSE_APPLETOUCH is not set
# CONFIG_MOUSE_BCM5974 is not set
CONFIG_MOUSE_VSXXXAA=m
# CONFIG_MOUSE_SYNAPTICS_I2C is not set
# CONFIG_INPUT_JOYSTICK is not set
# CONFIG_INPUT_TABLET is not set
# CONFIG_INPUT_TOUCHSCREEN is not set
CONFIG_INPUT_MISC=y
CONFIG_INPUT_PCSPKR=m
# CONFIG_INPUT_APANEL is not set
# CONFIG_INPUT_ATLAS_BTNS is not set
# CONFIG_INPUT_ATI_REMOTE is not set
# CONFIG_INPUT_ATI_REMOTE2 is not set
# CONFIG_INPUT_KEYSPAN_REMOTE is not set
# CONFIG_INPUT_POWERMATE is not set
# CONFIG_INPUT_YEALINK is not set
# CONFIG_INPUT_CM109 is not set
CONFIG_INPUT_UINPUT=m
# CONFIG_INPUT_WINBOND_CIR is not set

#
# Hardware I/O ports
#
CONFIG_SERIO=y
CONFIG_SERIO_I8042=y
CONFIG_SERIO_SERPORT=y
# CONFIG_SERIO_CT82C710 is not set
# CONFIG_SERIO_PARKBD is not set
# CONFIG_SERIO_PCIPS2 is not set
CONFIG_SERIO_LIBPS2=y
CONFIG_SERIO_RAW=m
CONFIG_GAMEPORT=m
CONFIG_GAMEPORT_NS558=m
CONFIG_GAMEPORT_L4=m
CONFIG_GAMEPORT_EMU10K1=m
CONFIG_GAMEPORT_FM801=m

#
# Character devices
#
CONFIG_VT=y
CONFIG_CONSOLE_TRANSLATIONS=y
CONFIG_VT_CONSOLE=y
CONFIG_HW_CONSOLE=y
CONFIG_VT_HW_CONSOLE_BINDING=y
# CONFIG_DEVKMEM is not set
CONFIG_SERIAL_NONSTANDARD=y
# CONFIG_COMPUTONE is not set
# CONFIG_ROCKETPORT is not set
CONFIG_CYCLADES=m
# CONFIG_CYZ_INTR is not set
# CONFIG_DIGIEPCA is not set
# CONFIG_MOXA_INTELLIO is not set
# CONFIG_MOXA_SMARTIO is not set
# CONFIG_ISI is not set
CONFIG_SYNCLINK=m
CONFIG_SYNCLINKMP=m
CONFIG_SYNCLINK_GT=m
CONFIG_N_HDLC=m
# CONFIG_RISCOM8 is not set
# CONFIG_SPECIALIX is not set
# CONFIG_STALDRV is not set
# CONFIG_NOZOMI is not set

#
# Serial drivers
#
CONFIG_SERIAL_8250=y
CONFIG_SERIAL_8250_CONSOLE=y
CONFIG_FIX_EARLYCON_MEM=y
CONFIG_SERIAL_8250_PCI=y
CONFIG_SERIAL_8250_PNP=y
CONFIG_SERIAL_8250_CS=m
CONFIG_SERIAL_8250_NR_UARTS=32
CONFIG_SERIAL_8250_RUNTIME_UARTS=4
CONFIG_SERIAL_8250_EXTENDED=y
CONFIG_SERIAL_8250_MANY_PORTS=y
CONFIG_SERIAL_8250_SHARE_IRQ=y
CONFIG_SERIAL_8250_DETECT_IRQ=y
CONFIG_SERIAL_8250_RSA=y

#
# Non-8250 serial port support
#
CONFIG_SERIAL_CORE=y
CONFIG_SERIAL_CORE_CONSOLE=y
CONFIG_SERIAL_JSM=m
CONFIG_UNIX98_PTYS=y
# CONFIG_DEVPTS_MULTIPLE_INSTANCES is not set
# CONFIG_LEGACY_PTYS is not set
CONFIG_PRINTER=m
CONFIG_LP_CONSOLE=y
CONFIG_PPDEV=m
CONFIG_HVC_DRIVER=y
CONFIG_HVC_IRQ=y
CONFIG_HVC_XEN=y
CONFIG_IPMI_HANDLER=m
CONFIG_IPMI_PANIC_EVENT=y
CONFIG_IPMI_PANIC_STRING=y
CONFIG_IPMI_DEVICE_INTERFACE=m
CONFIG_IPMI_SI=m
CONFIG_IPMI_WATCHDOG=m
CONFIG_IPMI_POWEROFF=m
CONFIG_HW_RANDOM=y
CONFIG_HW_RANDOM_TIMERIOMEM=m
CONFIG_HW_RANDOM_INTEL=m
CONFIG_HW_RANDOM_AMD=m
CONFIG_HW_RANDOM_VIA=m
CONFIG_NVRAM=y
CONFIG_RTC=y
# CONFIG_R3964 is not set
# CONFIG_APPLICOM is not set

#
# PCMCIA character devices
#
# CONFIG_SYNCLINK_CS is not set
CONFIG_CARDMAN_4000=m
CONFIG_CARDMAN_4040=m
# CONFIG_IPWIRELESS is not set
# CONFIG_MWAVE is not set
CONFIG_PC8736x_GPIO=m
CONFIG_NSC_GPIO=m
CONFIG_RAW_DRIVER=y
CONFIG_MAX_RAW_DEVS=8192
CONFIG_HPET=y
# CONFIG_HPET_MMAP is not set
CONFIG_HANGCHECK_TIMER=m
CONFIG_TCG_TPM=m
CONFIG_TCG_TIS=m
CONFIG_TCG_NSC=m
CONFIG_TCG_ATMEL=m
CONFIG_TCG_INFINEON=m
CONFIG_TELCLOCK=m
CONFIG_DEVPORT=y
CONFIG_I2C=m
CONFIG_I2C_BOARDINFO=y
CONFIG_I2C_COMPAT=y
CONFIG_I2C_CHARDEV=m
CONFIG_I2C_HELPER_AUTO=y
CONFIG_I2C_ALGOBIT=m

#
# I2C Hardware Bus support
#

#
# PC SMBus host controller drivers
#
# CONFIG_I2C_ALI1535 is not set
# CONFIG_I2C_ALI1563 is not set
# CONFIG_I2C_ALI15X3 is not set
CONFIG_I2C_AMD756=m
CONFIG_I2C_AMD756_S4882=m
CONFIG_I2C_AMD8111=m
CONFIG_I2C_I801=m
CONFIG_I2C_ISCH=m
CONFIG_I2C_PIIX4=m
CONFIG_I2C_NFORCE2=m
CONFIG_I2C_NFORCE2_S4985=m
# CONFIG_I2C_SIS5595 is not set
# CONFIG_I2C_SIS630 is not set
CONFIG_I2C_SIS96X=m
CONFIG_I2C_VIA=m
CONFIG_I2C_VIAPRO=m

#
# ACPI drivers
#
# CONFIG_I2C_SCMI is not set

#
# I2C system bus drivers (mostly embedded / system-on-chip)
#
# CONFIG_I2C_OCORES is not set
# CONFIG_I2C_SIMTEC is not set

#
# External I2C/SMBus adapter drivers
#
CONFIG_I2C_PARPORT=m
CONFIG_I2C_PARPORT_LIGHT=m
# CONFIG_I2C_TAOS_EVM is not set
# CONFIG_I2C_TINY_USB is not set

#
# Graphics adapter I2C/DDC channel drivers
#
CONFIG_I2C_VOODOO3=m

#
# Other I2C/SMBus bus drivers
#
# CONFIG_I2C_PCA_PLATFORM is not set
CONFIG_I2C_STUB=m

#
# Miscellaneous I2C Chip support
#
# CONFIG_DS1682 is not set
# CONFIG_SENSORS_TSL2550 is not set
# CONFIG_I2C_DEBUG_CORE is not set
# CONFIG_I2C_DEBUG_ALGO is not set
# CONFIG_I2C_DEBUG_BUS is not set
# CONFIG_I2C_DEBUG_CHIP is not set
# CONFIG_SPI is not set

#
# PPS support
#
# CONFIG_PPS is not set
CONFIG_ARCH_WANT_OPTIONAL_GPIOLIB=y
# CONFIG_GPIOLIB is not set
# CONFIG_W1 is not set
# CONFIG_POWER_SUPPLY is not set
CONFIG_HWMON=m
CONFIG_HWMON_VID=m
# CONFIG_HWMON_DEBUG_CHIP is not set

#
# Native drivers
#
CONFIG_SENSORS_ABITUGURU=m
CONFIG_SENSORS_ABITUGURU3=m
CONFIG_SENSORS_AD7414=m
CONFIG_SENSORS_AD7418=m
CONFIG_SENSORS_ADM1021=m
CONFIG_SENSORS_ADM1025=m
CONFIG_SENSORS_ADM1026=m
CONFIG_SENSORS_ADM1029=m
CONFIG_SENSORS_ADM1031=m
CONFIG_SENSORS_ADM9240=m
CONFIG_SENSORS_ADT7462=m
CONFIG_SENSORS_ADT7470=m
CONFIG_SENSORS_ADT7473=m
CONFIG_SENSORS_ADT7475=m
CONFIG_SENSORS_K8TEMP=m
CONFIG_SENSORS_ASB100=m
CONFIG_SENSORS_ATXP1=m
CONFIG_SENSORS_DS1621=m
CONFIG_SENSORS_I5K_AMB=m
CONFIG_SENSORS_F71805F=m
CONFIG_SENSORS_F71882FG=m
CONFIG_SENSORS_F75375S=m
CONFIG_SENSORS_FSCHMD=m
CONFIG_SENSORS_G760A=m
CONFIG_SENSORS_GL518SM=m
CONFIG_SENSORS_GL520SM=m
CONFIG_SENSORS_CORETEMP=m
CONFIG_SENSORS_IBMAEM=m
CONFIG_SENSORS_IBMPEX=m
CONFIG_SENSORS_IT87=m
CONFIG_SENSORS_LM63=m
CONFIG_SENSORS_LM75=m
CONFIG_SENSORS_LM77=m
CONFIG_SENSORS_LM78=m
CONFIG_SENSORS_LM80=m
CONFIG_SENSORS_LM83=m
CONFIG_SENSORS_LM85=m
CONFIG_SENSORS_LM87=m
CONFIG_SENSORS_LM90=m
CONFIG_SENSORS_LM92=m
CONFIG_SENSORS_LM93=m
CONFIG_SENSORS_LTC4215=m
CONFIG_SENSORS_LTC4245=m
CONFIG_SENSORS_LM95241=m
CONFIG_SENSORS_MAX1619=m
CONFIG_SENSORS_MAX6650=m
CONFIG_SENSORS_PC87360=m
CONFIG_SENSORS_PC87427=m
CONFIG_SENSORS_PCF8591=m
CONFIG_SENSORS_SIS5595=m
CONFIG_SENSORS_DME1737=m
CONFIG_SENSORS_SMSC47M1=m
CONFIG_SENSORS_SMSC47M192=m
CONFIG_SENSORS_SMSC47B397=m
CONFIG_SENSORS_ADS7828=m
CONFIG_SENSORS_THMC50=m
CONFIG_SENSORS_TMP401=m
CONFIG_SENSORS_TMP421=m
CONFIG_SENSORS_VIA686A=m
CONFIG_SENSORS_VT1211=m
CONFIG_SENSORS_VT8231=m
CONFIG_SENSORS_W83781D=m
CONFIG_SENSORS_W83791D=m
CONFIG_SENSORS_W83792D=m
CONFIG_SENSORS_W83793=m
CONFIG_SENSORS_W83L785TS=m
CONFIG_SENSORS_W83L786NG=m
CONFIG_SENSORS_W83627HF=m
CONFIG_SENSORS_W83627EHF=m
CONFIG_SENSORS_HDAPS=m
CONFIG_SENSORS_APPLESMC=m

#
# ACPI drivers
#
CONFIG_SENSORS_ATK0110=m
CONFIG_SENSORS_LIS3LV02D=m
CONFIG_THERMAL=y
CONFIG_WATCHDOG=y
# CONFIG_WATCHDOG_NOWAYOUT is not set

#
# Watchdog Device Drivers
#
CONFIG_SOFT_WATCHDOG=m
# CONFIG_ACQUIRE_WDT is not set
# CONFIG_ADVANTECH_WDT is not set
CONFIG_ALIM1535_WDT=m
CONFIG_ALIM7101_WDT=m
# CONFIG_SC520_WDT is not set
# CONFIG_SBC_FITPC2_WATCHDOG is not set
# CONFIG_EUROTECH_WDT is not set
# CONFIG_IB700_WDT is not set
CONFIG_IBMASR=m
# CONFIG_WAFER_WDT is not set
CONFIG_I6300ESB_WDT=m
CONFIG_ITCO_WDT=m
# CONFIG_ITCO_VENDOR_SUPPORT is not set
CONFIG_IT8712F_WDT=m
CONFIG_IT87_WDT=m
# CONFIG_HP_WATCHDOG is not set
# CONFIG_SC1200_WDT is not set
# CONFIG_PC87413_WDT is not set
# CONFIG_60XX_WDT is not set
# CONFIG_SBC8360_WDT is not set
# CONFIG_CPU5_WDT is not set
CONFIG_SMSC_SCH311X_WDT=m
CONFIG_SMSC37B787_WDT=m
CONFIG_W83627HF_WDT=m
CONFIG_W83697HF_WDT=m
CONFIG_W83697UG_WDT=m
CONFIG_W83877F_WDT=m
CONFIG_W83977F_WDT=m
CONFIG_MACHZ_WDT=m
# CONFIG_SBC_EPX_C3_WATCHDOG is not set
# CONFIG_XEN_WDT is not set

#
# PCI-based Watchdog Cards
#
CONFIG_PCIPCWATCHDOG=m
CONFIG_WDTPCI=m

#
# USB-based Watchdog Cards
#
CONFIG_USBPCWATCHDOG=m
CONFIG_SSB_POSSIBLE=y

#
# Sonics Silicon Backplane
#
CONFIG_SSB=m
CONFIG_SSB_SPROM=y
CONFIG_SSB_PCIHOST_POSSIBLE=y
CONFIG_SSB_PCIHOST=y
# CONFIG_SSB_B43_PCI_BRIDGE is not set
CONFIG_SSB_PCMCIAHOST_POSSIBLE=y
# CONFIG_SSB_PCMCIAHOST is not set
# CONFIG_SSB_DEBUG is not set
CONFIG_SSB_DRIVER_PCICORE_POSSIBLE=y
CONFIG_SSB_DRIVER_PCICORE=y

#
# Multifunction device drivers
#
# CONFIG_MFD_CORE is not set
# CONFIG_MFD_SM501 is not set
# CONFIG_HTC_PASIC3 is not set
# CONFIG_MFD_TMIO is not set
# CONFIG_MFD_WM8400 is not set
# CONFIG_MFD_WM831X is not set
# CONFIG_MFD_WM8350_I2C is not set
# CONFIG_MFD_PCF50633 is not set
# CONFIG_AB3100_CORE is not set
# CONFIG_REGULATOR is not set
# CONFIG_MEDIA_SUPPORT is not set

#
# Graphics support
#
CONFIG_AGP=y
CONFIG_AGP_AMD64=y
CONFIG_AGP_INTEL=y
CONFIG_AGP_SIS=y
CONFIG_AGP_VIA=y
CONFIG_VGA_ARB=y
CONFIG_DRM=m
CONFIG_DRM_KMS_HELPER=m
CONFIG_DRM_TTM=m
# CONFIG_DRM_TDFX is not set
CONFIG_DRM_R128=m
CONFIG_DRM_RADEON=m
CONFIG_DRM_I810=m
CONFIG_DRM_I830=m
CONFIG_DRM_I915=m
# CONFIG_DRM_I915_KMS is not set
CONFIG_DRM_MGA=m
# CONFIG_DRM_SIS is not set
CONFIG_DRM_VIA=m
CONFIG_DRM_SAVAGE=m
CONFIG_VGASTATE=m
CONFIG_VIDEO_OUTPUT_CONTROL=m
CONFIG_FB=y
# CONFIG_FIRMWARE_EDID is not set
CONFIG_FB_DDC=m
CONFIG_FB_BOOT_VESA_SUPPORT=y
CONFIG_FB_CFB_FILLRECT=y
CONFIG_FB_CFB_COPYAREA=y
CONFIG_FB_CFB_IMAGEBLIT=y
# CONFIG_FB_CFB_REV_PIXELS_IN_BYTE is not set
CONFIG_FB_SYS_FILLRECT=y
CONFIG_FB_SYS_COPYAREA=y
CONFIG_FB_SYS_IMAGEBLIT=y
# CONFIG_FB_FOREIGN_ENDIAN is not set
CONFIG_FB_SYS_FOPS=y
CONFIG_FB_DEFERRED_IO=y
CONFIG_FB_SVGALIB=m
# CONFIG_FB_MACMODES is not set
# CONFIG_FB_BACKLIGHT is not set
CONFIG_FB_MODE_HELPERS=y
CONFIG_FB_TILEBLITTING=y

#
# Frame buffer hardware drivers
#
CONFIG_FB_CIRRUS=m
# CONFIG_FB_PM2 is not set
# CONFIG_FB_CYBER2000 is not set
# CONFIG_FB_ARC is not set
# CONFIG_FB_ASILIANT is not set
# CONFIG_FB_IMSTT is not set
CONFIG_FB_VGA16=m
# CONFIG_FB_UVESA is not set
CONFIG_FB_VESA=y
# CONFIG_FB_N411 is not set
# CONFIG_FB_HGA is not set
# CONFIG_FB_S1D13XXX is not set
CONFIG_FB_NVIDIA=m
CONFIG_FB_NVIDIA_I2C=y
# CONFIG_FB_NVIDIA_DEBUG is not set
# CONFIG_FB_NVIDIA_BACKLIGHT is not set
CONFIG_FB_RIVA=m
# CONFIG_FB_RIVA_I2C is not set
# CONFIG_FB_RIVA_DEBUG is not set
# CONFIG_FB_RIVA_BACKLIGHT is not set
CONFIG_FB_LE80578=m
CONFIG_FB_CARILLO_RANCH=m
# CONFIG_FB_MATROX is not set
# CONFIG_FB_RADEON is not set
# CONFIG_FB_ATY128 is not set
# CONFIG_FB_ATY is not set
# CONFIG_FB_S3 is not set
CONFIG_FB_SAVAGE=m
CONFIG_FB_SAVAGE_I2C=y
CONFIG_FB_SAVAGE_ACCEL=y
# CONFIG_FB_SIS is not set
# CONFIG_FB_VIA is not set
# CONFIG_FB_NEOMAGIC is not set
CONFIG_FB_KYRO=m
# CONFIG_FB_3DFX is not set
# CONFIG_FB_VOODOO1 is not set
CONFIG_FB_VT8623=m
# CONFIG_FB_TRIDENT is not set
# CONFIG_FB_ARK is not set
# CONFIG_FB_PM3 is not set
CONFIG_FB_CARMINE=m
CONFIG_FB_CARMINE_DRAM_EVAL=y
# CONFIG_CARMINE_DRAM_CUSTOM is not set
# CONFIG_FB_GEODE is not set
# CONFIG_FB_VIRTUAL is not set
CONFIG_XEN_FBDEV_FRONTEND=y
# CONFIG_FB_METRONOME is not set
# CONFIG_FB_MB862XX is not set
# CONFIG_FB_BROADSHEET is not set
CONFIG_BACKLIGHT_LCD_SUPPORT=y
CONFIG_LCD_CLASS_DEVICE=m
# CONFIG_LCD_ILI9320 is not set
# CONFIG_LCD_PLATFORM is not set
CONFIG_BACKLIGHT_CLASS_DEVICE=m
# CONFIG_BACKLIGHT_GENERIC is not set
# CONFIG_BACKLIGHT_PROGEAR is not set
# CONFIG_BACKLIGHT_CARILLO_RANCH is not set
# CONFIG_BACKLIGHT_MBP_NVIDIA is not set
# CONFIG_BACKLIGHT_SAHARA is not set

#
# Display device support
#
CONFIG_DISPLAY_SUPPORT=m

#
# Display hardware drivers
#

#
# Console display driver support
#
CONFIG_VGA_CONSOLE=y
CONFIG_VGACON_SOFT_SCROLLBACK=y
CONFIG_VGACON_SOFT_SCROLLBACK_SIZE=64
CONFIG_DUMMY_CONSOLE=y
CONFIG_FRAMEBUFFER_CONSOLE=y
# CONFIG_FRAMEBUFFER_CONSOLE_DETECT_PRIMARY is not set
CONFIG_FRAMEBUFFER_CONSOLE_ROTATION=y
# CONFIG_FONTS is not set
CONFIG_FONT_8x8=y
CONFIG_FONT_8x16=y
CONFIG_LOGO=y
# CONFIG_LOGO_LINUX_MONO is not set
# CONFIG_LOGO_LINUX_VGA16 is not set
CONFIG_LOGO_LINUX_CLUT224=y
# CONFIG_SOUND is not set
CONFIG_HID_SUPPORT=y
CONFIG_HID=y
# CONFIG_HIDRAW is not set

#
# USB Input Devices
#
CONFIG_USB_HID=y
CONFIG_HID_PID=y
CONFIG_USB_HIDDEV=y

#
# Special HID drivers
#
CONFIG_HID_A4TECH=y
CONFIG_HID_APPLE=y
CONFIG_HID_BELKIN=y
CONFIG_HID_CHERRY=y
CONFIG_HID_CHICONY=y
CONFIG_HID_CYPRESS=y
CONFIG_HID_DRAGONRISE=y
# CONFIG_DRAGONRISE_FF is not set
CONFIG_HID_EZKEY=y
CONFIG_HID_KYE=y
CONFIG_HID_GYRATION=y
CONFIG_HID_TWINHAN=y
CONFIG_HID_KENSINGTON=y
CONFIG_HID_LOGITECH=y
CONFIG_LOGITECH_FF=y
# CONFIG_LOGIRUMBLEPAD2_FF is not set
CONFIG_HID_MICROSOFT=y
CONFIG_HID_MONTEREY=y
CONFIG_HID_NTRIG=y
CONFIG_HID_PANTHERLORD=y
# CONFIG_PANTHERLORD_FF is not set
CONFIG_HID_PETALYNX=y
CONFIG_HID_SAMSUNG=y
CONFIG_HID_SONY=y
CONFIG_HID_SUNPLUS=y
CONFIG_HID_GREENASIA=y
# CONFIG_GREENASIA_FF is not set
CONFIG_HID_SMARTJOYPLUS=y
# CONFIG_SMARTJOYPLUS_FF is not set
CONFIG_HID_TOPSEED=y
CONFIG_HID_THRUSTMASTER=y
CONFIG_THRUSTMASTER_FF=y
CONFIG_HID_WACOM=m
CONFIG_HID_ZEROPLUS=y
# CONFIG_ZEROPLUS_FF is not set
CONFIG_USB_SUPPORT=y
CONFIG_USB_ARCH_HAS_HCD=y
CONFIG_USB_ARCH_HAS_OHCI=y
CONFIG_USB_ARCH_HAS_EHCI=y
CONFIG_USB=y
# CONFIG_USB_DEBUG is not set
# CONFIG_USB_ANNOUNCE_NEW_DEVICES is not set

#
# Miscellaneous USB options
#
CONFIG_USB_DEVICEFS=y
CONFIG_USB_DEVICE_CLASS=y
# CONFIG_USB_DYNAMIC_MINORS is not set
# CONFIG_USB_SUSPEND is not set
# CONFIG_USB_OTG is not set
CONFIG_USB_MON=y
# CONFIG_USB_WUSB is not set
# CONFIG_USB_WUSB_CBAF is not set

#
# USB Host Controller Drivers
#
# CONFIG_USB_C67X00_HCD is not set
# CONFIG_USB_XHCI_HCD is not set
CONFIG_USB_EHCI_HCD=m
CONFIG_USB_EHCI_ROOT_HUB_TT=y
CONFIG_USB_EHCI_TT_NEWSCHED=y
# CONFIG_USB_OXU210HP_HCD is not set
CONFIG_USB_ISP116X_HCD=m
# CONFIG_USB_ISP1760_HCD is not set
# CONFIG_USB_ISP1362_HCD is not set
CONFIG_USB_OHCI_HCD=m
# CONFIG_USB_OHCI_HCD_SSB is not set
# CONFIG_USB_OHCI_BIG_ENDIAN_DESC is not set
# CONFIG_USB_OHCI_BIG_ENDIAN_MMIO is not set
CONFIG_USB_OHCI_LITTLE_ENDIAN=y
CONFIG_USB_UHCI_HCD=m
CONFIG_USB_SL811_HCD=m
CONFIG_USB_SL811_CS=m
# CONFIG_USB_R8A66597_HCD is not set
# CONFIG_USB_WHCI_HCD is not set
# CONFIG_USB_HWA_HCD is not set

#
# USB Device Class drivers
#
CONFIG_USB_ACM=m
CONFIG_USB_PRINTER=m
# CONFIG_USB_WDM is not set
# CONFIG_USB_TMC is not set

#
# NOTE: USB_STORAGE depends on SCSI but BLK_DEV_SD may
#

#
# also be needed; see USB_STORAGE Help for more info
#
CONFIG_USB_STORAGE=m
# CONFIG_USB_STORAGE_DEBUG is not set
CONFIG_USB_STORAGE_DATAFAB=m
CONFIG_USB_STORAGE_FREECOM=m
CONFIG_USB_STORAGE_ISD200=m
CONFIG_USB_STORAGE_USBAT=m
CONFIG_USB_STORAGE_SDDR09=m
CONFIG_USB_STORAGE_SDDR55=m
CONFIG_USB_STORAGE_JUMPSHOT=m
CONFIG_USB_STORAGE_ALAUDA=m
# CONFIG_USB_STORAGE_ONETOUCH is not set
# CONFIG_USB_STORAGE_KARMA is not set
# CONFIG_USB_STORAGE_CYPRESS_ATACB is not set
# CONFIG_USB_LIBUSUAL is not set

#
# USB Imaging devices
#
CONFIG_USB_MDC800=m
CONFIG_USB_MICROTEK=m

#
# USB port drivers
#
CONFIG_USB_USS720=m
CONFIG_USB_SERIAL=m
CONFIG_USB_EZUSB=y
CONFIG_USB_SERIAL_GENERIC=y
# CONFIG_USB_SERIAL_AIRCABLE is not set
CONFIG_USB_SERIAL_ARK3116=m
CONFIG_USB_SERIAL_BELKIN=m
# CONFIG_USB_SERIAL_CH341 is not set
CONFIG_USB_SERIAL_WHITEHEAT=m
CONFIG_USB_SERIAL_DIGI_ACCELEPORT=m
# CONFIG_USB_SERIAL_CP210X is not set
CONFIG_USB_SERIAL_CYPRESS_M8=m
CONFIG_USB_SERIAL_EMPEG=m
CONFIG_USB_SERIAL_FTDI_SIO=m
CONFIG_USB_SERIAL_FUNSOFT=m
CONFIG_USB_SERIAL_VISOR=m
CONFIG_USB_SERIAL_IPAQ=m
CONFIG_USB_SERIAL_IR=m
CONFIG_USB_SERIAL_EDGEPORT=m
CONFIG_USB_SERIAL_EDGEPORT_TI=m
CONFIG_USB_SERIAL_GARMIN=m
CONFIG_USB_SERIAL_IPW=m
# CONFIG_USB_SERIAL_IUU is not set
CONFIG_USB_SERIAL_KEYSPAN_PDA=m
CONFIG_USB_SERIAL_KEYSPAN=m
# CONFIG_USB_SERIAL_KEYSPAN_MPR is not set
# CONFIG_USB_SERIAL_KEYSPAN_USA28 is not set
# CONFIG_USB_SERIAL_KEYSPAN_USA28X is not set
# CONFIG_USB_SERIAL_KEYSPAN_USA28XA is not set
# CONFIG_USB_SERIAL_KEYSPAN_USA28XB is not set
# CONFIG_USB_SERIAL_KEYSPAN_USA19 is not set
# CONFIG_USB_SERIAL_KEYSPAN_USA18X is not set
# CONFIG_USB_SERIAL_KEYSPAN_USA19W is not set
# CONFIG_USB_SERIAL_KEYSPAN_USA19QW is not set
# CONFIG_USB_SERIAL_KEYSPAN_USA19QI is not set
# CONFIG_USB_SERIAL_KEYSPAN_USA49W is not set
# CONFIG_USB_SERIAL_KEYSPAN_USA49WLC is not set
CONFIG_USB_SERIAL_KLSI=m
CONFIG_USB_SERIAL_KOBIL_SCT=m
CONFIG_USB_SERIAL_MCT_U232=m
# CONFIG_USB_SERIAL_MOS7720 is not set
# CONFIG_USB_SERIAL_MOS7840 is not set
# CONFIG_USB_SERIAL_MOTOROLA is not set
CONFIG_USB_SERIAL_NAVMAN=m
CONFIG_USB_SERIAL_PL2303=m
# CONFIG_USB_SERIAL_OTI6858 is not set
# CONFIG_USB_SERIAL_QUALCOMM is not set
# CONFIG_USB_SERIAL_SPCP8X5 is not set
CONFIG_USB_SERIAL_HP4X=m
CONFIG_USB_SERIAL_SAFE=m
CONFIG_USB_SERIAL_SAFE_PADDED=y
# CONFIG_USB_SERIAL_SIEMENS_MPI is not set
CONFIG_USB_SERIAL_SIERRAWIRELESS=m
# CONFIG_USB_SERIAL_SYMBOL is not set
CONFIG_USB_SERIAL_TI=m
CONFIG_USB_SERIAL_CYBERJACK=m
CONFIG_USB_SERIAL_XIRCOM=m
CONFIG_USB_SERIAL_OPTION=m
CONFIG_USB_SERIAL_OMNINET=m
# CONFIG_USB_SERIAL_OPTICON is not set
# CONFIG_USB_SERIAL_DEBUG is not set

#
# USB Miscellaneous drivers
#
CONFIG_USB_EMI62=m
CONFIG_USB_EMI26=m
# CONFIG_USB_ADUTUX is not set
# CONFIG_USB_SEVSEG is not set
CONFIG_USB_RIO500=m
CONFIG_USB_LEGOTOWER=m
CONFIG_USB_LCD=m
# CONFIG_USB_BERRY_CHARGE is not set
CONFIG_USB_LED=m
# CONFIG_USB_CYPRESS_CY7C63 is not set
# CONFIG_USB_CYTHERM is not set
CONFIG_USB_IDMOUSE=m
# CONFIG_USB_FTDI_ELAN is not set
CONFIG_USB_APPLEDISPLAY=m
CONFIG_USB_SISUSBVGA=m
CONFIG_USB_SISUSBVGA_CON=y
CONFIG_USB_LD=m
# CONFIG_USB_TRANCEVIBRATOR is not set
# CONFIG_USB_IOWARRIOR is not set
CONFIG_USB_TEST=m
# CONFIG_USB_ISIGHTFW is not set
# CONFIG_USB_VST is not set
CONFIG_USB_ATM=m
CONFIG_USB_SPEEDTOUCH=m
CONFIG_USB_CXACRU=m
CONFIG_USB_UEAGLEATM=m
CONFIG_USB_XUSBATM=m
# CONFIG_USB_GADGET is not set

#
# OTG and related infrastructure
#
CONFIG_USB_OTG_UTILS=y
CONFIG_NOP_USB_XCEIV=m
# CONFIG_UWB is not set
# CONFIG_MMC is not set
# CONFIG_MEMSTICK is not set
CONFIG_NEW_LEDS=y
CONFIG_LEDS_CLASS=y

#
# LED drivers
#
# CONFIG_LEDS_ALIX2 is not set
CONFIG_LEDS_PCA9532=m
# CONFIG_LEDS_LP3944 is not set
# CONFIG_LEDS_CLEVO_MAIL is not set
CONFIG_LEDS_PCA955X=m
CONFIG_LEDS_BD2802=m

#
# LED Triggers
#
CONFIG_LEDS_TRIGGERS=y
CONFIG_LEDS_TRIGGER_TIMER=m
CONFIG_LEDS_TRIGGER_IDE_DISK=y
CONFIG_LEDS_TRIGGER_HEARTBEAT=m
# CONFIG_LEDS_TRIGGER_BACKLIGHT is not set
CONFIG_LEDS_TRIGGER_DEFAULT_ON=m

#
# iptables trigger is under Netfilter config (LED target)
#
# CONFIG_ACCESSIBILITY is not set
# CONFIG_INFINIBAND is not set
# CONFIG_EDAC is not set
# CONFIG_RTC_CLASS is not set
# CONFIG_DMADEVICES is not set
# CONFIG_AUXDISPLAY is not set
CONFIG_UIO=m
# CONFIG_UIO_CIF is not set
# CONFIG_UIO_PDRV is not set
# CONFIG_UIO_PDRV_GENIRQ is not set
# CONFIG_UIO_SMX is not set
# CONFIG_UIO_AEC is not set
# CONFIG_UIO_SERCOS3 is not set
# CONFIG_UIO_PCI_GENERIC is not set

#
# TI VLYNQ
#
CONFIG_XEN_BALLOON=y
CONFIG_XEN_SCRUB_PAGES=y
CONFIG_XEN_DEV_EVTCHN=y
CONFIG_XEN_BACKEND=y
CONFIG_XEN_NETDEV_BACKEND=y
CONFIG_XEN_BLKDEV_BACKEND=y
CONFIG_XEN_BLKDEV_TAP=y
CONFIG_XEN_BLKBACK_PAGEMAP=y
CONFIG_XEN_PCIDEV_BACKEND=y
CONFIG_XEN_PCIDEV_BACKEND_VPCI=y
# CONFIG_XEN_PCIDEV_BACKEND_PASS is not set
# CONFIG_XEN_PCIDEV_BACKEND_SLOT is not set
# CONFIG_XEN_PCIDEV_BACKEND_CONTROLLER is not set
# CONFIG_XEN_PCIDEV_BE_DEBUG is not set
CONFIG_XENFS=y
CONFIG_XEN_COMPAT_XENFS=y
CONFIG_XEN_SYS_HYPERVISOR=y
CONFIG_XEN_MCE=y
CONFIG_XEN_XENBUS_FRONTEND=y
CONFIG_XEN_GNTDEV=y
CONFIG_XEN_S3=y
CONFIG_ACPI_PROCESSOR_XEN=y
CONFIG_XEN_PLATFORM_PCI=m
# CONFIG_STAGING is not set
# CONFIG_X86_PLATFORM_DEVICES is not set

#
# Firmware Drivers
#
CONFIG_EDD=m
CONFIG_EDD_OFF=y
CONFIG_FIRMWARE_MEMMAP=y
CONFIG_DELL_RBU=m
CONFIG_DCDBAS=m
# CONFIG_DMIID is not set
CONFIG_ISCSI_IBFT_FIND=y
CONFIG_ISCSI_IBFT=m

#
# File systems
#
CONFIG_EXT2_FS=y
CONFIG_EXT2_FS_XATTR=y
CONFIG_EXT2_FS_POSIX_ACL=y
CONFIG_EXT2_FS_SECURITY=y
CONFIG_EXT2_FS_XIP=y
CONFIG_EXT3_FS=m
CONFIG_EXT3_DEFAULTS_TO_ORDERED=y
CONFIG_EXT3_FS_XATTR=y
CONFIG_EXT3_FS_POSIX_ACL=y
CONFIG_EXT3_FS_SECURITY=y
CONFIG_EXT4_FS=m
CONFIG_EXT4_FS_XATTR=y
CONFIG_EXT4_FS_POSIX_ACL=y
CONFIG_EXT4_FS_SECURITY=y
# CONFIG_EXT4_DEBUG is not set
CONFIG_FS_XIP=y
CONFIG_JBD=m
# CONFIG_JBD_DEBUG is not set
CONFIG_JBD2=m
# CONFIG_JBD2_DEBUG is not set
CONFIG_FS_MBCACHE=y
# CONFIG_REISERFS_FS is not set
# CONFIG_JFS_FS is not set
CONFIG_FS_POSIX_ACL=y
# CONFIG_XFS_FS is not set
CONFIG_GFS2_FS=m
CONFIG_GFS2_FS_LOCKING_DLM=y
CONFIG_OCFS2_FS=m
CONFIG_OCFS2_FS_O2CB=m
CONFIG_OCFS2_FS_USERSPACE_CLUSTER=m
CONFIG_OCFS2_FS_STATS=y
CONFIG_OCFS2_DEBUG_MASKLOG=y
# CONFIG_OCFS2_DEBUG_FS is not set
# CONFIG_OCFS2_FS_POSIX_ACL is not set
# CONFIG_BTRFS_FS is not set
# CONFIG_NILFS2_FS is not set
CONFIG_FILE_LOCKING=y
CONFIG_FSNOTIFY=y
CONFIG_DNOTIFY=y
CONFIG_INOTIFY=y
CONFIG_INOTIFY_USER=y
CONFIG_QUOTA=y
CONFIG_QUOTA_NETLINK_INTERFACE=y
# CONFIG_PRINT_QUOTA_WARNING is not set
CONFIG_QUOTA_TREE=y
# CONFIG_QFMT_V1 is not set
CONFIG_QFMT_V2=y
CONFIG_QUOTACTL=y
# CONFIG_AUTOFS_FS is not set
CONFIG_AUTOFS4_FS=m
CONFIG_FUSE_FS=m
CONFIG_CUSE=m

#
# Caches
#
CONFIG_FSCACHE=m
CONFIG_FSCACHE_STATS=y
# CONFIG_FSCACHE_HISTOGRAM is not set
# CONFIG_FSCACHE_DEBUG is not set
# CONFIG_FSCACHE_OBJECT_LIST is not set
CONFIG_CACHEFILES=m
CONFIG_CACHEFILES_DEBUG=y
# CONFIG_CACHEFILES_HISTOGRAM is not set

#
# CD-ROM/DVD Filesystems
#
CONFIG_ISO9660_FS=y
CONFIG_JOLIET=y
CONFIG_ZISOFS=y
CONFIG_UDF_FS=m
CONFIG_UDF_NLS=y

#
# DOS/FAT/NT Filesystems
#
CONFIG_FAT_FS=m
CONFIG_MSDOS_FS=m
CONFIG_VFAT_FS=m
CONFIG_FAT_DEFAULT_CODEPAGE=437
CONFIG_FAT_DEFAULT_IOCHARSET="ascii"
CONFIG_NTFS_FS=m
# CONFIG_NTFS_DEBUG is not set
CONFIG_NTFS_RW=y

#
# Pseudo filesystems
#
CONFIG_PROC_FS=y
CONFIG_PROC_KCORE=y
CONFIG_PROC_VMCORE=y
CONFIG_PROC_SYSCTL=y
CONFIG_PROC_PAGE_MONITOR=y
CONFIG_SYSFS=y
CONFIG_TMPFS=y
# CONFIG_TMPFS_POSIX_ACL is not set
CONFIG_HUGETLBFS=y
CONFIG_HUGETLB_PAGE=y
CONFIG_CONFIGFS_FS=m
CONFIG_MISC_FILESYSTEMS=y
# CONFIG_ADFS_FS is not set
# CONFIG_AFFS_FS is not set
# CONFIG_ECRYPT_FS is not set
# CONFIG_HFS_FS is not set
# CONFIG_HFSPLUS_FS is not set
# CONFIG_BEFS_FS is not set
# CONFIG_BFS_FS is not set
# CONFIG_EFS_FS is not set
# CONFIG_JFFS2_FS is not set
# CONFIG_CRAMFS is not set
# CONFIG_SQUASHFS is not set
# CONFIG_VXFS_FS is not set
# CONFIG_MINIX_FS is not set
# CONFIG_OMFS_FS is not set
# CONFIG_HPFS_FS is not set
# CONFIG_QNX4FS_FS is not set
# CONFIG_ROMFS_FS is not set
# CONFIG_SYSV_FS is not set
CONFIG_UFS_FS=m
CONFIG_UFS_FS_WRITE=y
# CONFIG_UFS_DEBUG is not set
# CONFIG_EXOFS_FS is not set
CONFIG_NETWORK_FILESYSTEMS=y
CONFIG_NFS_FS=m
CONFIG_NFS_V3=y
CONFIG_NFS_V3_ACL=y
CONFIG_NFS_V4=y
CONFIG_NFS_V4_1=y
# CONFIG_NFS_FSCACHE is not set
CONFIG_NFSD=m
CONFIG_NFSD_V2_ACL=y
CONFIG_NFSD_V3=y
CONFIG_NFSD_V3_ACL=y
CONFIG_NFSD_V4=y
CONFIG_LOCKD=m
CONFIG_LOCKD_V4=y
CONFIG_EXPORTFS=m
CONFIG_NFS_ACL_SUPPORT=m
CONFIG_NFS_COMMON=y
CONFIG_SUNRPC=m
CONFIG_SUNRPC_GSS=m
CONFIG_RPCSEC_GSS_KRB5=m
CONFIG_RPCSEC_GSS_SPKM3=m
# CONFIG_SMB_FS is not set
CONFIG_CIFS=m
# CONFIG_CIFS_STATS is not set
CONFIG_CIFS_WEAK_PW_HASH=y
# CONFIG_CIFS_UPCALL is not set
CONFIG_CIFS_XATTR=y
CONFIG_CIFS_POSIX=y
# CONFIG_CIFS_DEBUG2 is not set
# CONFIG_CIFS_DFS_UPCALL is not set
# CONFIG_CIFS_EXPERIMENTAL is not set
# CONFIG_NCP_FS is not set
# CONFIG_CODA_FS is not set
# CONFIG_AFS_FS is not set

#
# Partition Types
#
CONFIG_PARTITION_ADVANCED=y
# CONFIG_ACORN_PARTITION is not set
CONFIG_OSF_PARTITION=y
CONFIG_AMIGA_PARTITION=y
# CONFIG_ATARI_PARTITION is not set
CONFIG_MAC_PARTITION=y
CONFIG_MSDOS_PARTITION=y
CONFIG_BSD_DISKLABEL=y
CONFIG_MINIX_SUBPARTITION=y
CONFIG_SOLARIS_X86_PARTITION=y
CONFIG_UNIXWARE_DISKLABEL=y
# CONFIG_LDM_PARTITION is not set
CONFIG_SGI_PARTITION=y
# CONFIG_ULTRIX_PARTITION is not set
CONFIG_SUN_PARTITION=y
CONFIG_KARMA_PARTITION=y
CONFIG_EFI_PARTITION=y
# CONFIG_SYSV68_PARTITION is not set
CONFIG_NLS=y
CONFIG_NLS_DEFAULT="utf8"
CONFIG_NLS_CODEPAGE_437=y
CONFIG_NLS_CODEPAGE_737=m
CONFIG_NLS_CODEPAGE_775=m
CONFIG_NLS_CODEPAGE_850=m
CONFIG_NLS_CODEPAGE_852=m
CONFIG_NLS_CODEPAGE_855=m
CONFIG_NLS_CODEPAGE_857=m
CONFIG_NLS_CODEPAGE_860=m
CONFIG_NLS_CODEPAGE_861=m
CONFIG_NLS_CODEPAGE_862=m
CONFIG_NLS_CODEPAGE_863=m
CONFIG_NLS_CODEPAGE_864=m
CONFIG_NLS_CODEPAGE_865=m
CONFIG_NLS_CODEPAGE_866=m
CONFIG_NLS_CODEPAGE_869=m
CONFIG_NLS_CODEPAGE_936=m
CONFIG_NLS_CODEPAGE_950=m
CONFIG_NLS_CODEPAGE_932=m
CONFIG_NLS_CODEPAGE_949=m
CONFIG_NLS_CODEPAGE_874=m
CONFIG_NLS_ISO8859_8=m
CONFIG_NLS_CODEPAGE_1250=m
CONFIG_NLS_CODEPAGE_1251=m
CONFIG_NLS_ASCII=y
CONFIG_NLS_ISO8859_1=m
CONFIG_NLS_ISO8859_2=m
CONFIG_NLS_ISO8859_3=m
CONFIG_NLS_ISO8859_4=m
CONFIG_NLS_ISO8859_5=m
CONFIG_NLS_ISO8859_6=m
CONFIG_NLS_ISO8859_7=m
CONFIG_NLS_ISO8859_9=m
CONFIG_NLS_ISO8859_13=m
CONFIG_NLS_ISO8859_14=m
CONFIG_NLS_ISO8859_15=m
CONFIG_NLS_KOI8_R=m
CONFIG_NLS_KOI8_U=m
CONFIG_NLS_UTF8=m
CONFIG_DLM=m
CONFIG_DLM_DEBUG=y

#
# Kernel hacking
#
CONFIG_TRACE_IRQFLAGS_SUPPORT=y
# CONFIG_PRINTK_TIME is not set
# CONFIG_ENABLE_WARN_DEPRECATED is not set
# CONFIG_ENABLE_MUST_CHECK is not set
CONFIG_FRAME_WARN=2048
CONFIG_MAGIC_SYSRQ=y
CONFIG_STRIP_ASM_SYMS=y
# CONFIG_UNUSED_SYMBOLS is not set
CONFIG_DEBUG_FS=y
# CONFIG_HEADERS_CHECK is not set
CONFIG_DEBUG_KERNEL=y
# CONFIG_DEBUG_SHIRQ is not set
CONFIG_DETECT_SOFTLOCKUP=y
CONFIG_BOOTPARAM_SOFTLOCKUP_PANIC=y
CONFIG_BOOTPARAM_SOFTLOCKUP_PANIC_VALUE=1
CONFIG_DETECT_HUNG_TASK=y
CONFIG_BOOTPARAM_HUNG_TASK_PANIC=y
CONFIG_BOOTPARAM_HUNG_TASK_PANIC_VALUE=1
CONFIG_SCHED_DEBUG=y
CONFIG_SCHEDSTATS=y
# CONFIG_TIMER_STATS is not set
# CONFIG_DEBUG_OBJECTS is not set
# CONFIG_DEBUG_SLAB is not set
# CONFIG_DEBUG_RT_MUTEXES is not set
# CONFIG_RT_MUTEX_TESTER is not set
# CONFIG_DEBUG_SPINLOCK is not set
# CONFIG_DEBUG_MUTEXES is not set
# CONFIG_DEBUG_LOCK_ALLOC is not set
# CONFIG_PROVE_LOCKING is not set
# CONFIG_LOCK_STAT is not set
# CONFIG_DEBUG_SPINLOCK_SLEEP is not set
# CONFIG_DEBUG_LOCKING_API_SELFTESTS is not set
CONFIG_STACKTRACE=y
# CONFIG_DEBUG_KOBJECT is not set
CONFIG_DEBUG_BUGVERBOSE=y
CONFIG_DEBUG_INFO=y
# CONFIG_DEBUG_VM is not set
# CONFIG_DEBUG_VIRTUAL is not set
# CONFIG_DEBUG_WRITECOUNT is not set
CONFIG_DEBUG_MEMORY_INIT=y
CONFIG_DEBUG_LIST=y
# CONFIG_DEBUG_SG is not set
# CONFIG_DEBUG_NOTIFIERS is not set
# CONFIG_DEBUG_CREDENTIALS is not set
CONFIG_ARCH_WANT_FRAME_POINTERS=y
CONFIG_FRAME_POINTER=y
# CONFIG_BOOT_PRINTK_DELAY is not set
# CONFIG_RCU_TORTURE_TEST is not set
# CONFIG_RCU_CPU_STALL_DETECTOR is not set
# CONFIG_KPROBES_SANITY_TEST is not set
# CONFIG_BACKTRACE_SELF_TEST is not set
# CONFIG_DEBUG_BLOCK_EXT_DEVT is not set
# CONFIG_DEBUG_FORCE_WEAK_PER_CPU is not set
# CONFIG_LKDTM is not set
# CONFIG_FAULT_INJECTION is not set
CONFIG_LATENCYTOP=y
CONFIG_SYSCTL_SYSCALL_CHECK=y
# CONFIG_DEBUG_PAGEALLOC is not set
CONFIG_USER_STACKTRACE_SUPPORT=y
CONFIG_HAVE_FUNCTION_TRACER=y
CONFIG_HAVE_FUNCTION_GRAPH_TRACER=y
CONFIG_HAVE_FUNCTION_GRAPH_FP_TEST=y
CONFIG_HAVE_FUNCTION_TRACE_MCOUNT_TEST=y
CONFIG_HAVE_DYNAMIC_FTRACE=y
CONFIG_HAVE_FTRACE_MCOUNT_RECORD=y
CONFIG_HAVE_SYSCALL_TRACEPOINTS=y
CONFIG_RING_BUFFER=y
CONFIG_RING_BUFFER_ALLOW_SWAP=y
CONFIG_TRACING_SUPPORT=y
# CONFIG_FTRACE is not set
# CONFIG_PROVIDE_OHCI1394_DMA_INIT is not set
# CONFIG_FIREWIRE_OHCI_REMOTE_DMA is not set
# CONFIG_DYNAMIC_DEBUG is not set
# CONFIG_DMA_API_DEBUG is not set
# CONFIG_SAMPLES is not set
CONFIG_HAVE_ARCH_KGDB=y
# CONFIG_KGDB is not set
CONFIG_HAVE_ARCH_KMEMCHECK=y
CONFIG_STRICT_DEVMEM=y
# CONFIG_X86_VERBOSE_BOOTUP is not set
CONFIG_EARLY_PRINTK=y
# CONFIG_EARLY_PRINTK_DBGP is not set
CONFIG_DEBUG_STACKOVERFLOW=y
# CONFIG_DEBUG_STACK_USAGE is not set
# CONFIG_DEBUG_PER_CPU_MAPS is not set
# CONFIG_X86_PTDUMP is not set
CONFIG_DEBUG_RODATA=y
# CONFIG_DEBUG_RODATA_TEST is not set
# CONFIG_DEBUG_NX_TEST is not set
# CONFIG_IOMMU_DEBUG is not set
# CONFIG_IOMMU_STRESS is not set
CONFIG_HAVE_MMIOTRACE_SUPPORT=y
CONFIG_IO_DELAY_TYPE_0X80=0
CONFIG_IO_DELAY_TYPE_0XED=1
CONFIG_IO_DELAY_TYPE_UDELAY=2
CONFIG_IO_DELAY_TYPE_NONE=3
CONFIG_IO_DELAY_0X80=y
# CONFIG_IO_DELAY_0XED is not set
# CONFIG_IO_DELAY_UDELAY is not set
# CONFIG_IO_DELAY_NONE is not set
CONFIG_DEFAULT_IO_DELAY_TYPE=0
# CONFIG_DEBUG_BOOT_PARAMS is not set
# CONFIG_CPA_DEBUG is not set
# CONFIG_OPTIMIZE_INLINING is not set

#
# Security options
#
CONFIG_KEYS=y
CONFIG_KEYS_DEBUG_PROC_KEYS=y
CONFIG_SECURITY=y
CONFIG_SECURITYFS=y
CONFIG_SECURITY_NETWORK=y
CONFIG_SECURITY_NETWORK_XFRM=y
# CONFIG_SECURITY_PATH is not set
# CONFIG_SECURITY_FILE_CAPABILITIES is not set
# CONFIG_SECURITY_ROOTPLUG is not set
CONFIG_LSM_MMAP_MIN_ADDR=65536
CONFIG_SECURITY_SELINUX=y
CONFIG_SECURITY_SELINUX_BOOTPARAM=y
CONFIG_SECURITY_SELINUX_BOOTPARAM_VALUE=1
CONFIG_SECURITY_SELINUX_DISABLE=y
CONFIG_SECURITY_SELINUX_DEVELOP=y
CONFIG_SECURITY_SELINUX_AVC_STATS=y
CONFIG_SECURITY_SELINUX_CHECKREQPROT_VALUE=1
# CONFIG_SECURITY_SELINUX_POLICYDB_VERSION_MAX is not set
# CONFIG_SECURITY_SMACK is not set
# CONFIG_SECURITY_TOMOYO is not set
# CONFIG_IMA is not set
CONFIG_XOR_BLOCKS=m
CONFIG_ASYNC_CORE=m
CONFIG_ASYNC_MEMCPY=m
CONFIG_ASYNC_XOR=m
CONFIG_ASYNC_PQ=m
CONFIG_ASYNC_RAID6_RECOV=m
CONFIG_CRYPTO=y

#
# Crypto core or helper
#
CONFIG_CRYPTO_FIPS=y
CONFIG_CRYPTO_ALGAPI=y
CONFIG_CRYPTO_ALGAPI2=y
CONFIG_CRYPTO_AEAD=m
CONFIG_CRYPTO_AEAD2=y
CONFIG_CRYPTO_BLKCIPHER=m
CONFIG_CRYPTO_BLKCIPHER2=y
CONFIG_CRYPTO_HASH=y
CONFIG_CRYPTO_HASH2=y
CONFIG_CRYPTO_RNG=m
CONFIG_CRYPTO_RNG2=y
CONFIG_CRYPTO_PCOMP=y
CONFIG_CRYPTO_MANAGER=y
CONFIG_CRYPTO_MANAGER2=y
CONFIG_CRYPTO_GF128MUL=m
CONFIG_CRYPTO_NULL=m
CONFIG_CRYPTO_WORKQUEUE=y
CONFIG_CRYPTO_CRYPTD=m
CONFIG_CRYPTO_AUTHENC=m
# CONFIG_CRYPTO_TEST is not set

#
# Authenticated Encryption with Associated Data
#
CONFIG_CRYPTO_CCM=m
CONFIG_CRYPTO_GCM=m
CONFIG_CRYPTO_SEQIV=m

#
# Block modes
#
CONFIG_CRYPTO_CBC=m
CONFIG_CRYPTO_CTR=m
CONFIG_CRYPTO_CTS=m
CONFIG_CRYPTO_ECB=m
# CONFIG_CRYPTO_LRW is not set
CONFIG_CRYPTO_PCBC=m
# CONFIG_CRYPTO_XTS is not set
CONFIG_CRYPTO_FPU=m

#
# Hash modes
#
CONFIG_CRYPTO_HMAC=y
CONFIG_CRYPTO_XCBC=m
CONFIG_CRYPTO_VMAC=m

#
# Digest
#
CONFIG_CRYPTO_CRC32C=y
CONFIG_CRYPTO_CRC32C_INTEL=m
CONFIG_CRYPTO_GHASH=m
CONFIG_CRYPTO_MD4=m
CONFIG_CRYPTO_MD5=m
CONFIG_CRYPTO_MICHAEL_MIC=m
CONFIG_CRYPTO_RMD128=m
CONFIG_CRYPTO_RMD160=m
CONFIG_CRYPTO_RMD256=m
CONFIG_CRYPTO_RMD320=m
CONFIG_CRYPTO_SHA1=y
CONFIG_CRYPTO_SHA256=m
CONFIG_CRYPTO_SHA512=m
CONFIG_CRYPTO_TGR192=m
CONFIG_CRYPTO_WP512=m

#
# Ciphers
#
CONFIG_CRYPTO_AES=m
CONFIG_CRYPTO_AES_X86_64=m
CONFIG_CRYPTO_AES_NI_INTEL=m
CONFIG_CRYPTO_ANUBIS=m
CONFIG_CRYPTO_ARC4=m
CONFIG_CRYPTO_BLOWFISH=m
CONFIG_CRYPTO_CAMELLIA=m
CONFIG_CRYPTO_CAST5=m
CONFIG_CRYPTO_CAST6=m
CONFIG_CRYPTO_DES=m
CONFIG_CRYPTO_FCRYPT=m
CONFIG_CRYPTO_KHAZAD=m
# CONFIG_CRYPTO_SALSA20 is not set
# CONFIG_CRYPTO_SALSA20_X86_64 is not set
CONFIG_CRYPTO_SEED=m
CONFIG_CRYPTO_SERPENT=m
CONFIG_CRYPTO_TEA=m
CONFIG_CRYPTO_TWOFISH=m
CONFIG_CRYPTO_TWOFISH_COMMON=m
CONFIG_CRYPTO_TWOFISH_X86_64=m

#
# Compression
#
CONFIG_CRYPTO_DEFLATE=m
CONFIG_CRYPTO_ZLIB=m
CONFIG_CRYPTO_LZO=m

#
# Random Number Generation
#
CONFIG_CRYPTO_ANSI_CPRNG=m
CONFIG_CRYPTO_HW=y
CONFIG_CRYPTO_DEV_PADLOCK=m
CONFIG_CRYPTO_DEV_PADLOCK_AES=m
CONFIG_CRYPTO_DEV_PADLOCK_SHA=m
# CONFIG_CRYPTO_DEV_HIFN_795X is not set
CONFIG_HAVE_KVM=y
# CONFIG_VIRTUALIZATION is not set
# CONFIG_BINARY_PRINTF is not set

#
# Library routines
#
CONFIG_BITREVERSE=y
CONFIG_GENERIC_FIND_FIRST_BIT=y
CONFIG_GENERIC_FIND_NEXT_BIT=y
CONFIG_GENERIC_FIND_LAST_BIT=y
CONFIG_CRC_CCITT=m
CONFIG_CRC16=m
CONFIG_CRC_T10DIF=m
CONFIG_CRC_ITU_T=m
CONFIG_CRC32=y
CONFIG_CRC7=m
CONFIG_LIBCRC32C=y
CONFIG_ZLIB_INFLATE=y
CONFIG_ZLIB_DEFLATE=m
CONFIG_LZO_COMPRESS=m
CONFIG_LZO_DECOMPRESS=m
CONFIG_DECOMPRESS_GZIP=y
CONFIG_DECOMPRESS_BZIP2=y
CONFIG_DECOMPRESS_LZMA=y
CONFIG_REED_SOLOMON=m
CONFIG_REED_SOLOMON_DEC16=y
CONFIG_TEXTSEARCH=y
CONFIG_TEXTSEARCH_KMP=m
CONFIG_TEXTSEARCH_BM=m
CONFIG_TEXTSEARCH_FSM=m
CONFIG_HAS_IOMEM=y
CONFIG_HAS_IOPORT=y
CONFIG_HAS_DMA=y
CONFIG_CPUMASK_OFFSTACK=y
CONFIG_NLATTR=y

[-- Attachment #3: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: kernel BUG at arch/x86/xen/mmu.c:1860!

[Teck Choon Giam]

[-- Attachment #1.1: Type: text/plain, Size: 12472 bytes --]

On Fri, Jan 28, 2011 at 6:20 AM, Teck Choon Giam <giamteckchoon@gmail.com>wrote:

>
>
> On Fri, Jan 28, 2011 at 4:32 AM, Konrad Rzeszutek Wilk <
> konrad.wilk@oracle.com> wrote:
>
>> > > What does git log show? What is the latest commit you have there?
>> > >
>> >
>> > commit 75cc13f5aa29b4f3227d269ca165dfa8937c94fe
>> > Merge: 2607c07 a386bf7
>> > Author: Jeremy Fitzhardinge <jeremy.fitzhardinge@citrix.com>
>> > Date:   Thu Dec 9 17:16:16 2010 -0800
>> >
>> >     Merge commit 'v2.6.32.27' into xen/next-2.6.32
>> >
>> >     * commit 'v2.6.32.27': (128 commits)
>> >       Linux 2.6.32.27
>>
>> .. snip..
>> >
>> > Anything that I can test or you need me to test... feel free to ask so
>> that
>> > I can speed up in tracking/hunting this bug down :)
>>
>> You got the latest one so you should have the fixes. But something is not
>> working.. just to make sure I am not missing anything can you send me your
>> .config
>> file?
>>
>
> Ok... please see attached.
>
>
> Thanks.
>
> Kindest regards,
> Giam Teck Choon
>
>
Hi Konrad,

This is just an update from my testing.

I tried with latest xen/next-2.6.32.x and same bug still there:

Feb 23 22:56:56 xen06 kernel: ------------[ cut here ]------------
Feb 23 22:56:56 xen06 kernel: kernel BUG at arch/x86/xen/mmu.c:1872!
Feb 23 22:56:56 xen06 kernel: invalid opcode: 0000 [#1] SMP
Feb 23 22:56:56 xen06 kernel: last sysfs file:
/sys/devices/system/cpu/cpu3/cache/index2/shared_cpu_map
Feb 23 22:56:56 xen06 kernel: CPU 0
Feb 23 22:56:56 xen06 kernel: Modules linked in: dlm configfs xt_physdev
iptable_filter ip_tables x_tables bridge stp be2iscsi iscsi_tcp bnx2i cnic
uio ipv6 cxgb3i cxgb3 mdio libiscsi_tcp libiscsi scsi_transport_iscsi
dm_multipath scsi_dh video backlight output sbs sbshc power_meter hwmon
battery acpi_memhotplug xen_acpi_memhotplug ac parport_pc lp parport sg
ide_cd_mod cdrom serio_raw tg3 libphy button tpm_tis tpm tpm_bios iTCO_wdt
i2c_i801 i2c_core pcspkr shpchp dm_snapshot dm_zero dm_mirror dm_region_hash
dm_log dm_mod ata_piix libata sd_mod scsi_mod raid1 ext3 jbd uhci_hcd
ohci_hcd ehci_hcd [last unloaded: microcode]
Feb 23 22:56:56 xen06 kernel: Pid: 15222, comm: mpath_wait Not tainted
2.6.32.28-2.xen.pvops.choon.centos5 #1 PowerEdge 860
Feb 23 22:56:56 xen06 kernel: RIP: e030:[<ffffffff8100cb5b>]
[<ffffffff8100cb5b>] pin_pagetable_pfn+0x53/0x59
Feb 23 22:56:56 xen06 kernel: RSP: e02b:ffff8800265b9dc8  EFLAGS: 00010282
Feb 23 22:56:56 xen06 kernel: RAX: 00000000ffffffea RBX: 00000000000306d7
RCX: 0000000000000183
Feb 23 22:56:56 xen06 kernel: RDX: 00000000deadbeef RSI: 00000000deadbeef
RDI: 00000000deadbeef
Feb 23 22:56:56 xen06 kernel: RBP: ffff8800265b9de8 R08: 00000000000006b8
R09: ffff880000000000
Feb 23 22:56:56 xen06 kernel: R10: 00000000deadbeef R11: 0000000000000246
R12: 0000000000000003
Feb 23 22:56:56 xen06 kernel: R13: 00000000000306d7 R14: ffff88003b940200
R15: 00007f7655df6258
Feb 23 22:56:56 xen06 kernel: FS:  00007f76560016e0(0000)
GS:ffff88002804f000(0000) knlGS:0000000000000000
Feb 23 22:56:56 xen06 kernel: CS:  e033 DS: 0000 ES: 0000 CR0:
000000008005003b
Feb 23 22:56:56 xen06 kernel: CR2: 00007f7655df6258 CR3: 000000002559b000
CR4: 0000000000002660
Feb 23 22:56:56 xen06 lvm[3984]: Monitoring snapshot
XenGroup-testcrash1--snapshot
Feb 23 22:56:56 xen06 kernel: DR0: 0000000000000000 DR1: 0000000000000000
DR2: 0000000000000000
Feb 23 22:56:56 xen06 kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0
DR7: 0000000000000400
Feb 23 22:56:56 xen06 kernel: Process mpath_wait (pid: 15222, threadinfo
ffff8800265b8000, task ffff88003b940200)
Feb 23 22:56:56 xen06 kernel: Stack:
Feb 23 22:56:56 xen06 kernel:  0000000000000000 0000000000202bff
000000013f009e18 00000000000306d7
Feb 23 22:56:56 xen06 kernel: <0> ffff8800265b9e08 ffffffff8100e07c
ffff8800267bdac0 ffff88002679f570
Feb 23 22:56:56 xen06 kernel: <0> ffff8800265b9e18 ffffffff8100e0af
ffff8800265b9e58 ffffffff810a408b
Feb 23 22:56:56 xen06 kernel: Call Trace:
Feb 23 22:56:56 xen06 kernel:  [<ffffffff8100e07c>]
xen_alloc_ptpage+0x64/0x69
Feb 23 22:56:56 xen06 kernel:  [<ffffffff8100e0af>] xen_alloc_pte+0xe/0x10
Feb 23 22:56:56 xen06 kernel:  [<ffffffff810a408b>] __pte_alloc+0x70/0xce
Feb 23 22:56:56 xen06 kernel:  [<ffffffff810a4229>]
handle_mm_fault+0x140/0x8b9
Feb 23 22:56:56 xen06 kernel:  [<ffffffff8131be2d>]
do_page_fault+0x252/0x2e2
Feb 23 22:56:56 xen06 kernel:  [<ffffffff81319db5>] page_fault+0x25/0x30
Feb 23 22:56:56 xen06 kernel: Code: 48 b8 ff ff ff ff ff ff ff 7f 48 21 c2
48 89 55 e8 48 8d 7d e0 be 01 00 00 00 31 d2 41 ba f0 7f 00 00 e8 e9 c7 ff
ff 85 c0 74 04 <0f> 0b eb fe c9 c3 55 40 f6 c7 01 48 89 e5 53 48 89 fb 74 5b
48
Feb 23 22:56:56 xen06 kernel: RIP  [<ffffffff8100cb5b>]
pin_pagetable_pfn+0x53/0x59
Feb 23 22:56:56 xen06 kernel:  RSP <ffff8800265b9dc8>
Feb 23 22:56:56 xen06 kernel: ---[ end trace fd2f141edfc37649 ]---
Feb 23 22:56:56 xen06 kernel: kjournald starting.  Commit interval 5 seconds
Feb 23 22:56:56 xen06 kernel: EXT3 FS on dm-11, internal journal
Feb 23 22:56:56 xen06 kernel: EXT3-fs: mounted filesystem with ordered data
mode.
Feb 23 23:00:01 xen06 syslogd 1.4.1: restart.
Feb 23 23:00:01 xen06 kernel: klogd 1.4.1, log source = /proc/kmsg started.
Feb 23 23:00:01 xen06 kernel: Linux version
2.6.32.28-2.xen.pvops.choon.centos5 (mockbuild@builder5.choon.net) (gcc
version 4.1.2 20080704 (Red Hat 4.1.2-48)) #1 SMP Wed Feb 23 22:29:39 SGT
2011
Feb 23 23:00:01 xen06 kernel: Command line: ro root=/dev/md1 panic=5
panic_timeout=5
Feb 23 23:00:01 xen06 kernel: KERNEL supported cpus:
Feb 23 23:00:01 xen06 kernel:   Intel GenuineIntel
Feb 23 23:00:01 xen06 kernel:   AMD AuthenticAMD
Feb 23 23:00:01 xen06 kernel:   Centaur CentaurHauls
Feb 23 23:00:01 xen06 kernel: released 0 pages of unused memory
Feb 23 23:00:01 xen06 kernel: BIOS-provided physical RAM map:
Feb 23 23:00:01 xen06 kernel:  Xen: 0000000000000000 - 00000000000a0000
(usable)
Feb 23 23:00:01 xen06 kernel:  Xen: 00000000000a0000 - 0000000000100000
(reserved)
Feb 23 23:00:01 xen06 kernel:  Xen: 0000000000100000 - 0000000040000000
(usable)
Feb 23 23:00:01 xen06 kernel:  Xen: 00000000dffc0000 - 00000000dffcfc00
(ACPI data)
Feb 23 23:00:01 xen06 kernel:  Xen: 00000000dffcfc00 - 00000000dffff000
(reserved)
Feb 23 23:00:01 xen06 kernel:  Xen: 00000000f0000000 - 00000000f4000000
(reserved)
Feb 23 23:00:01 xen06 kernel:  Xen: 00000000fec00000 - 00000000fed00400
(reserved)
Feb 23 23:00:01 xen06 kernel:  Xen: 00000000fed13000 - 00000000feda0000
(reserved)
Feb 23 23:00:01 xen06 kernel:  Xen: 00000000fee00000 - 00000000fee10000
(reserved)
Feb 23 23:00:01 xen06 kernel:  Xen: 00000000ffb00000 - 0000000100000000
(reserved)
Feb 23 23:00:01 xen06 kernel:  Xen: 00000001ffffe000 - 0000000200000000
(reserved)
Feb 23 23:00:01 xen06 kernel:  Xen: 0000000200000000 - 00000003bffbe000
(usable)

Even I patch with latest 2.6.32.29:

Feb 24 01:03:17 xen06 kernel: ------------[ cut here ]------------
Feb 24 01:03:17 xen06 kernel: kernel BUG at arch/x86/xen/mmu.c:1872!
Feb 24 01:03:17 xen06 kernel: invalid opcode: 0000 [#2] SMP
Feb 24 01:03:17 xen06 kernel: last sysfs file: /sys/block/dm-13/dev
Feb 24 01:03:17 xen06 kernel: CPU 2
Feb 24 01:03:17 xen06 kernel: Modules linked in: dlm configfs xt_physdev
iptable_filter ip_tables x_tables bridge stp be2iscsi iscsi_tcp bnx2i cnic
uio ipv6 cxgb3i cxgb3 mdio libiscsi_tcp libiscsi scsi_transport_iscsi
dm_multipath scsi_dh video backlight output sbs sbshc power_meter hwmon
battery acpi_memhotplug xen_acpi_memhotplug ac parport_pc lp parport sg
ide_cd_mod cdrom tg3 serio_raw libphy button tpm_tis tpm tpm_bios pcspkr
shpchp i2c_i801 i2c_core iTCO_wdt dm_snapshot dm_zero dm_mirror
dm_region_hash dm_log dm_mod ata_piix libata sd_mod scsi_mod raid1 ext3 jbd
uhci_hcd ohci_hcd ehci_hcd [last unloaded: microcode]
Feb 24 01:03:17 xen06 kernel: Pid: 509, comm: dmsetup Tainted: G      D
2.6.32.29-0.xen.pvops.choon.centos5 #1 PowerEdge 860
Feb 24 01:03:17 xen06 kernel: RIP: e030:[<ffffffff8100cb5b>]
[<ffffffff8100cb5b>] pin_pagetable_pfn+0x53/0x59
Feb 24 01:03:17 xen06 kernel: RSP: e02b:ffff88003ced5dc8  EFLAGS: 00010282
Feb 24 01:03:17 xen06 kernel: RAX: 00000000ffffffea RBX: 0000000000030395
RCX: 0000000000000181
Feb 24 01:03:17 xen06 kernel: RDX: 00000000deadbeef RSI: 00000000deadbeef
RDI: 00000000deadbeef
Feb 24 01:03:17 xen06 kernel: RBP: ffff88003ced5de8 R08: 0000000000000ca8
R09: ffff880000000000
Feb 24 01:03:17 xen06 kernel: R10: 00000000deadbeef R11: 0000000000000246
R12: 0000000000000003
Feb 24 01:03:17 xen06 kernel: R13: 0000000000030395 R14: ffff88003025c4c0
R15: 00000033e1e00258
Feb 24 01:03:17 xen06 kernel: FS:  00007ff20eb366e0(0000)
GS:ffff880028089000(0000) knlGS:0000000000000000
Feb 24 01:03:17 xen06 kernel: CS:  e033 DS: 0000 ES: 0000 CR0:
000000008005003b
Feb 24 01:03:17 xen06 kernel: CR2: 00000033e1e00258 CR3: 000000003b57b000
CR4: 0000000000002660
Feb 24 01:03:17 xen06 kernel: DR0: 0000000000000000 DR1: 0000000000000000
DR2: 0000000000000000
Feb 24 01:03:17 xen06 kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0
DR7: 0000000000000400
Feb 24 01:03:17 xen06 kernel: Process dmsetup (pid: 509, threadinfo
ffff88003ced4000, task ffff88003025c4c0)
Feb 24 01:03:17 xen06 kernel: Stack:
Feb 24 01:03:17 xen06 kernel:  0000000000000000 0000000000202741
000000013e6e5a18 0000000000030395
Feb 24 01:03:17 xen06 kernel: <0> ffff88003ced5e08 ffffffff8100e07c
ffff88003000aac0 ffff880036bb1878
Feb 24 01:03:17 xen06 kernel: <0> ffff88003ced5e18 ffffffff8100e0af
ffff88003ced5e58 ffffffff810a4309
Feb 24 01:03:17 xen06 kernel: Call Trace:
Feb 24 01:03:17 xen06 kernel:  [<ffffffff8100e07c>]
xen_alloc_ptpage+0x64/0x69
Feb 24 01:03:17 xen06 kernel:  [<ffffffff8100e0af>] xen_alloc_pte+0xe/0x10
Feb 24 01:03:17 xen06 kernel:  [<ffffffff810a4309>] __pte_alloc+0x70/0xce
Feb 24 01:03:17 xen06 kernel:  [<ffffffff810a44a7>]
handle_mm_fault+0x140/0x8b9
Feb 24 01:03:17 xen06 kernel:  [<ffffffff8131c1fd>]
do_page_fault+0x252/0x2e2
Feb 24 01:03:17 xen06 kernel:  [<ffffffff8131a185>] page_fault+0x25/0x30
Feb 24 01:03:18 xen06 kernel: Code: 48 b8 ff ff ff ff ff ff ff 7f 48 21 c2
48 89 55 e8 48 8d 7d e0 be 01 00 00 00 31 d2 41 ba f0 7f 00 00 e8 e9 c7 ff
ff 85 c0 74 04 <0f> 0b eb fe c9 c3 55 40 f6 c7 01 48 89 e5 53 48 89 fb 74 5b
48
Feb 24 01:03:18 xen06 kernel: RIP  [<ffffffff8100cb5b>]
pin_pagetable_pfn+0x53/0x59
Feb 24 01:03:18 xen06 kernel:  RSP <ffff88003ced5dc8>
Feb 24 01:03:18 xen06 kernel: ---[ end trace 5ea31e622470b519 ]---
Feb 24 01:06:21 xen06 syslogd 1.4.1: restart.
Feb 24 01:06:21 xen06 kernel: klogd 1.4.1, log source = /proc/kmsg started.
Feb 24 01:06:21 xen06 kernel: Linux version
2.6.32.29-0.xen.pvops.choon.centos5 (mockbuild@builder5.choon.net) (gcc
version 4.1.2 20080704 (Red Hat 4.1.2-48)) #1 SMP Thu Feb 24 00:21:24 SGT
2011
Feb 24 01:06:21 xen06 kernel: Command line: ro root=/dev/md1 panic=5
panic_timeout=5
Feb 24 01:06:21 xen06 kernel: KERNEL supported cpus:
Feb 24 01:06:21 xen06 kernel:   Intel GenuineIntel
Feb 24 01:06:21 xen06 kernel:   AMD AuthenticAMD
Feb 24 01:06:21 xen06 kernel:   Centaur CentaurHauls
Feb 24 01:06:21 xen06 kernel: released 0 pages of unused memory
Feb 24 01:06:21 xen06 kernel: BIOS-provided physical RAM map:
Feb 24 01:06:21 xen06 kernel:  Xen: 0000000000000000 - 00000000000a0000
(usable)
Feb 24 01:06:21 xen06 kernel:  Xen: 00000000000a0000 - 0000000000100000
(reserved)
Feb 24 01:06:21 xen06 kernel:  Xen: 0000000000100000 - 0000000040000000
(usable)
Feb 24 01:06:21 xen06 kernel:  Xen: 00000000dffc0000 - 00000000dffcfc00
(ACPI data)
Feb 24 01:06:21 xen06 kernel:  Xen: 00000000dffcfc00 - 00000000dffff000
(reserved)
Feb 24 01:06:21 xen06 kernel:  Xen: 00000000f0000000 - 00000000f4000000
(reserved)
Feb 24 01:06:21 xen06 kernel:  Xen: 00000000fec00000 - 00000000fed00400
(reserved)
Feb 24 01:06:21 xen06 kernel:  Xen: 00000000fed13000 - 00000000feda0000
(reserved)
Feb 24 01:06:21 xen06 kernel:  Xen: 00000000fee00000 - 00000000fee10000
(reserved)
Feb 24 01:06:21 xen06 kernel:  Xen: 00000000ffb00000 - 0000000100000000
(reserved)
Feb 24 01:06:21 xen06 kernel:  Xen: 00000001ffffe000 - 0000000200000000
(reserved)
Feb 24 01:06:21 xen06 kernel:  Xen: 0000000200000000 - 00000003bffbe000
(usable)
Feb 24 01:06:21 xen06 kernel: DMI 2.4 present.
Feb 24 01:06:21 xen06 kernel: last_pfn = 0x3bffbe max_arch_pfn = 0x400000000
Feb 24 01:06:21 xen06 kernel: x86 PAT enabled: cpu 0, old 0x50100070406, new
0x7010600070106
Feb 24 01:06:21 xen06 kernel: last_pfn = 0x40000 max_arch_pfn = 0x400000000

Thanks.

Kindest regards,
Giam Teck Choon

[-- Attachment #1.2: Type: text/html, Size: 14086 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: kernel BUG at arch/x86/xen/mmu.c:1860!

[Konrad Rzeszutek Wilk]

> This is just an update from my testing.
> 
> I tried with latest xen/next-2.6.32.x and same bug still there:

Grrrrrrrr..

Let me setup a machine to reproduce this. Is the distro you are using
still CentOS 5?

> 
> Feb 23 22:56:56 xen06 kernel: ------------[ cut here ]------------
> Feb 23 22:56:56 xen06 kernel: kernel BUG at arch/x86/xen/mmu.c:1872!
> Feb 23 22:56:56 xen06 kernel: invalid opcode: 0000 [#1] SMP
> Feb 23 22:56:56 xen06 kernel: last sysfs file:
> /sys/devices/system/cpu/cpu3/cache/index2/shared_cpu_map
> Feb 23 22:56:56 xen06 kernel: CPU 0
> Feb 23 22:56:56 xen06 kernel: Modules linked in: dlm configfs xt_physdev
> iptable_filter ip_tables x_tables bridge stp be2iscsi iscsi_tcp bnx2i cnic
> uio ipv6 cxgb3i cxgb3 mdio libiscsi_tcp libiscsi scsi_transport_iscsi
> dm_multipath scsi_dh video backlight output sbs sbshc power_meter hwmon
> battery acpi_memhotplug xen_acpi_memhotplug ac parport_pc lp parport sg
> ide_cd_mod cdrom serio_raw tg3 libphy button tpm_tis tpm tpm_bios iTCO_wdt
> i2c_i801 i2c_core pcspkr shpchp dm_snapshot dm_zero dm_mirror dm_region_hash
> dm_log dm_mod ata_piix libata sd_mod scsi_mod raid1 ext3 jbd uhci_hcd
> ohci_hcd ehci_hcd [last unloaded: microcode]
> Feb 23 22:56:56 xen06 kernel: Pid: 15222, comm: mpath_wait Not tainted
> 2.6.32.28-2.xen.pvops.choon.centos5 #1 PowerEdge 860
> Feb 23 22:56:56 xen06 kernel: RIP: e030:[<ffffffff8100cb5b>]
> [<ffffffff8100cb5b>] pin_pagetable_pfn+0x53/0x59
> Feb 23 22:56:56 xen06 kernel: RSP: e02b:ffff8800265b9dc8  EFLAGS: 00010282
> Feb 23 22:56:56 xen06 kernel: RAX: 00000000ffffffea RBX: 00000000000306d7
> RCX: 0000000000000183
> Feb 23 22:56:56 xen06 kernel: RDX: 00000000deadbeef RSI: 00000000deadbeef
> RDI: 00000000deadbeef
> Feb 23 22:56:56 xen06 kernel: RBP: ffff8800265b9de8 R08: 00000000000006b8
> R09: ffff880000000000
> Feb 23 22:56:56 xen06 kernel: R10: 00000000deadbeef R11: 0000000000000246
> R12: 0000000000000003
> Feb 23 22:56:56 xen06 kernel: R13: 00000000000306d7 R14: ffff88003b940200
> R15: 00007f7655df6258
> Feb 23 22:56:56 xen06 kernel: FS:  00007f76560016e0(0000)
> GS:ffff88002804f000(0000) knlGS:0000000000000000
> Feb 23 22:56:56 xen06 kernel: CS:  e033 DS: 0000 ES: 0000 CR0:
> 000000008005003b
> Feb 23 22:56:56 xen06 kernel: CR2: 00007f7655df6258 CR3: 000000002559b000
> CR4: 0000000000002660
> Feb 23 22:56:56 xen06 lvm[3984]: Monitoring snapshot
> XenGroup-testcrash1--snapshot
> Feb 23 22:56:56 xen06 kernel: DR0: 0000000000000000 DR1: 0000000000000000
> DR2: 0000000000000000
> Feb 23 22:56:56 xen06 kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0
> DR7: 0000000000000400
> Feb 23 22:56:56 xen06 kernel: Process mpath_wait (pid: 15222, threadinfo
> ffff8800265b8000, task ffff88003b940200)
> Feb 23 22:56:56 xen06 kernel: Stack:
> Feb 23 22:56:56 xen06 kernel:  0000000000000000 0000000000202bff
> 000000013f009e18 00000000000306d7
> Feb 23 22:56:56 xen06 kernel: <0> ffff8800265b9e08 ffffffff8100e07c
> ffff8800267bdac0 ffff88002679f570
> Feb 23 22:56:56 xen06 kernel: <0> ffff8800265b9e18 ffffffff8100e0af
> ffff8800265b9e58 ffffffff810a408b
> Feb 23 22:56:56 xen06 kernel: Call Trace:
> Feb 23 22:56:56 xen06 kernel:  [<ffffffff8100e07c>]
> xen_alloc_ptpage+0x64/0x69
> Feb 23 22:56:56 xen06 kernel:  [<ffffffff8100e0af>] xen_alloc_pte+0xe/0x10
> Feb 23 22:56:56 xen06 kernel:  [<ffffffff810a408b>] __pte_alloc+0x70/0xce
> Feb 23 22:56:56 xen06 kernel:  [<ffffffff810a4229>]
> handle_mm_fault+0x140/0x8b9
> Feb 23 22:56:56 xen06 kernel:  [<ffffffff8131be2d>]
> do_page_fault+0x252/0x2e2
> Feb 23 22:56:56 xen06 kernel:  [<ffffffff81319db5>] page_fault+0x25/0x30
> Feb 23 22:56:56 xen06 kernel: Code: 48 b8 ff ff ff ff ff ff ff 7f 48 21 c2
> 48 89 55 e8 48 8d 7d e0 be 01 00 00 00 31 d2 41 ba f0 7f 00 00 e8 e9 c7 ff
> ff 85 c0 74 04 <0f> 0b eb fe c9 c3 55 40 f6 c7 01 48 89 e5 53 48 89 fb 74 5b
> 48
> Feb 23 22:56:56 xen06 kernel: RIP  [<ffffffff8100cb5b>]
> pin_pagetable_pfn+0x53/0x59
> Feb 23 22:56:56 xen06 kernel:  RSP <ffff8800265b9dc8>
> Feb 23 22:56:56 xen06 kernel: ---[ end trace fd2f141edfc37649 ]---
> Feb 23 22:56:56 xen06 kernel: kjournald starting.  Commit interval 5 seconds
> Feb 23 22:56:56 xen06 kernel: EXT3 FS on dm-11, internal journal
> Feb 23 22:56:56 xen06 kernel: EXT3-fs: mounted filesystem with ordered data
> mode.
> Feb 23 23:00:01 xen06 syslogd 1.4.1: restart.
> Feb 23 23:00:01 xen06 kernel: klogd 1.4.1, log source = /proc/kmsg started.
> Feb 23 23:00:01 xen06 kernel: Linux version
> 2.6.32.28-2.xen.pvops.choon.centos5 (mockbuild@builder5.choon.net) (gcc
> version 4.1.2 20080704 (Red Hat 4.1.2-48)) #1 SMP Wed Feb 23 22:29:39 SGT
> 2011
> Feb 23 23:00:01 xen06 kernel: Command line: ro root=/dev/md1 panic=5
> panic_timeout=5
> Feb 23 23:00:01 xen06 kernel: KERNEL supported cpus:
> Feb 23 23:00:01 xen06 kernel:   Intel GenuineIntel
> Feb 23 23:00:01 xen06 kernel:   AMD AuthenticAMD
> Feb 23 23:00:01 xen06 kernel:   Centaur CentaurHauls
> Feb 23 23:00:01 xen06 kernel: released 0 pages of unused memory
> Feb 23 23:00:01 xen06 kernel: BIOS-provided physical RAM map:
> Feb 23 23:00:01 xen06 kernel:  Xen: 0000000000000000 - 00000000000a0000
> (usable)
> Feb 23 23:00:01 xen06 kernel:  Xen: 00000000000a0000 - 0000000000100000
> (reserved)
> Feb 23 23:00:01 xen06 kernel:  Xen: 0000000000100000 - 0000000040000000
> (usable)
> Feb 23 23:00:01 xen06 kernel:  Xen: 00000000dffc0000 - 00000000dffcfc00
> (ACPI data)
> Feb 23 23:00:01 xen06 kernel:  Xen: 00000000dffcfc00 - 00000000dffff000
> (reserved)
> Feb 23 23:00:01 xen06 kernel:  Xen: 00000000f0000000 - 00000000f4000000
> (reserved)
> Feb 23 23:00:01 xen06 kernel:  Xen: 00000000fec00000 - 00000000fed00400
> (reserved)
> Feb 23 23:00:01 xen06 kernel:  Xen: 00000000fed13000 - 00000000feda0000
> (reserved)
> Feb 23 23:00:01 xen06 kernel:  Xen: 00000000fee00000 - 00000000fee10000
> (reserved)
> Feb 23 23:00:01 xen06 kernel:  Xen: 00000000ffb00000 - 0000000100000000
> (reserved)
> Feb 23 23:00:01 xen06 kernel:  Xen: 00000001ffffe000 - 0000000200000000
> (reserved)
> Feb 23 23:00:01 xen06 kernel:  Xen: 0000000200000000 - 00000003bffbe000
> (usable)
> 
> Even I patch with latest 2.6.32.29:
> 
> Feb 24 01:03:17 xen06 kernel: ------------[ cut here ]------------
> Feb 24 01:03:17 xen06 kernel: kernel BUG at arch/x86/xen/mmu.c:1872!
> Feb 24 01:03:17 xen06 kernel: invalid opcode: 0000 [#2] SMP
> Feb 24 01:03:17 xen06 kernel: last sysfs file: /sys/block/dm-13/dev
> Feb 24 01:03:17 xen06 kernel: CPU 2
> Feb 24 01:03:17 xen06 kernel: Modules linked in: dlm configfs xt_physdev
> iptable_filter ip_tables x_tables bridge stp be2iscsi iscsi_tcp bnx2i cnic
> uio ipv6 cxgb3i cxgb3 mdio libiscsi_tcp libiscsi scsi_transport_iscsi
> dm_multipath scsi_dh video backlight output sbs sbshc power_meter hwmon
> battery acpi_memhotplug xen_acpi_memhotplug ac parport_pc lp parport sg
> ide_cd_mod cdrom tg3 serio_raw libphy button tpm_tis tpm tpm_bios pcspkr
> shpchp i2c_i801 i2c_core iTCO_wdt dm_snapshot dm_zero dm_mirror
> dm_region_hash dm_log dm_mod ata_piix libata sd_mod scsi_mod raid1 ext3 jbd
> uhci_hcd ohci_hcd ehci_hcd [last unloaded: microcode]
> Feb 24 01:03:17 xen06 kernel: Pid: 509, comm: dmsetup Tainted: G      D
> 2.6.32.29-0.xen.pvops.choon.centos5 #1 PowerEdge 860
> Feb 24 01:03:17 xen06 kernel: RIP: e030:[<ffffffff8100cb5b>]
> [<ffffffff8100cb5b>] pin_pagetable_pfn+0x53/0x59
> Feb 24 01:03:17 xen06 kernel: RSP: e02b:ffff88003ced5dc8  EFLAGS: 00010282
> Feb 24 01:03:17 xen06 kernel: RAX: 00000000ffffffea RBX: 0000000000030395
> RCX: 0000000000000181
> Feb 24 01:03:17 xen06 kernel: RDX: 00000000deadbeef RSI: 00000000deadbeef
> RDI: 00000000deadbeef
> Feb 24 01:03:17 xen06 kernel: RBP: ffff88003ced5de8 R08: 0000000000000ca8
> R09: ffff880000000000
> Feb 24 01:03:17 xen06 kernel: R10: 00000000deadbeef R11: 0000000000000246
> R12: 0000000000000003
> Feb 24 01:03:17 xen06 kernel: R13: 0000000000030395 R14: ffff88003025c4c0
> R15: 00000033e1e00258
> Feb 24 01:03:17 xen06 kernel: FS:  00007ff20eb366e0(0000)
> GS:ffff880028089000(0000) knlGS:0000000000000000
> Feb 24 01:03:17 xen06 kernel: CS:  e033 DS: 0000 ES: 0000 CR0:
> 000000008005003b
> Feb 24 01:03:17 xen06 kernel: CR2: 00000033e1e00258 CR3: 000000003b57b000
> CR4: 0000000000002660
> Feb 24 01:03:17 xen06 kernel: DR0: 0000000000000000 DR1: 0000000000000000
> DR2: 0000000000000000
> Feb 24 01:03:17 xen06 kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0
> DR7: 0000000000000400
> Feb 24 01:03:17 xen06 kernel: Process dmsetup (pid: 509, threadinfo
> ffff88003ced4000, task ffff88003025c4c0)
> Feb 24 01:03:17 xen06 kernel: Stack:
> Feb 24 01:03:17 xen06 kernel:  0000000000000000 0000000000202741
> 000000013e6e5a18 0000000000030395
> Feb 24 01:03:17 xen06 kernel: <0> ffff88003ced5e08 ffffffff8100e07c
> ffff88003000aac0 ffff880036bb1878
> Feb 24 01:03:17 xen06 kernel: <0> ffff88003ced5e18 ffffffff8100e0af
> ffff88003ced5e58 ffffffff810a4309
> Feb 24 01:03:17 xen06 kernel: Call Trace:
> Feb 24 01:03:17 xen06 kernel:  [<ffffffff8100e07c>]
> xen_alloc_ptpage+0x64/0x69
> Feb 24 01:03:17 xen06 kernel:  [<ffffffff8100e0af>] xen_alloc_pte+0xe/0x10
> Feb 24 01:03:17 xen06 kernel:  [<ffffffff810a4309>] __pte_alloc+0x70/0xce
> Feb 24 01:03:17 xen06 kernel:  [<ffffffff810a44a7>]
> handle_mm_fault+0x140/0x8b9
> Feb 24 01:03:17 xen06 kernel:  [<ffffffff8131c1fd>]
> do_page_fault+0x252/0x2e2
> Feb 24 01:03:17 xen06 kernel:  [<ffffffff8131a185>] page_fault+0x25/0x30
> Feb 24 01:03:18 xen06 kernel: Code: 48 b8 ff ff ff ff ff ff ff 7f 48 21 c2
> 48 89 55 e8 48 8d 7d e0 be 01 00 00 00 31 d2 41 ba f0 7f 00 00 e8 e9 c7 ff
> ff 85 c0 74 04 <0f> 0b eb fe c9 c3 55 40 f6 c7 01 48 89 e5 53 48 89 fb 74 5b
> 48
> Feb 24 01:03:18 xen06 kernel: RIP  [<ffffffff8100cb5b>]
> pin_pagetable_pfn+0x53/0x59
> Feb 24 01:03:18 xen06 kernel:  RSP <ffff88003ced5dc8>
> Feb 24 01:03:18 xen06 kernel: ---[ end trace 5ea31e622470b519 ]---
> Feb 24 01:06:21 xen06 syslogd 1.4.1: restart.
> Feb 24 01:06:21 xen06 kernel: klogd 1.4.1, log source = /proc/kmsg started.
> Feb 24 01:06:21 xen06 kernel: Linux version
> 2.6.32.29-0.xen.pvops.choon.centos5 (mockbuild@builder5.choon.net) (gcc
> version 4.1.2 20080704 (Red Hat 4.1.2-48)) #1 SMP Thu Feb 24 00:21:24 SGT
> 2011
> Feb 24 01:06:21 xen06 kernel: Command line: ro root=/dev/md1 panic=5
> panic_timeout=5
> Feb 24 01:06:21 xen06 kernel: KERNEL supported cpus:
> Feb 24 01:06:21 xen06 kernel:   Intel GenuineIntel
> Feb 24 01:06:21 xen06 kernel:   AMD AuthenticAMD
> Feb 24 01:06:21 xen06 kernel:   Centaur CentaurHauls
> Feb 24 01:06:21 xen06 kernel: released 0 pages of unused memory
> Feb 24 01:06:21 xen06 kernel: BIOS-provided physical RAM map:
> Feb 24 01:06:21 xen06 kernel:  Xen: 0000000000000000 - 00000000000a0000
> (usable)
> Feb 24 01:06:21 xen06 kernel:  Xen: 00000000000a0000 - 0000000000100000
> (reserved)
> Feb 24 01:06:21 xen06 kernel:  Xen: 0000000000100000 - 0000000040000000
> (usable)
> Feb 24 01:06:21 xen06 kernel:  Xen: 00000000dffc0000 - 00000000dffcfc00
> (ACPI data)
> Feb 24 01:06:21 xen06 kernel:  Xen: 00000000dffcfc00 - 00000000dffff000
> (reserved)
> Feb 24 01:06:21 xen06 kernel:  Xen: 00000000f0000000 - 00000000f4000000
> (reserved)
> Feb 24 01:06:21 xen06 kernel:  Xen: 00000000fec00000 - 00000000fed00400
> (reserved)
> Feb 24 01:06:21 xen06 kernel:  Xen: 00000000fed13000 - 00000000feda0000
> (reserved)
> Feb 24 01:06:21 xen06 kernel:  Xen: 00000000fee00000 - 00000000fee10000
> (reserved)
> Feb 24 01:06:21 xen06 kernel:  Xen: 00000000ffb00000 - 0000000100000000
> (reserved)
> Feb 24 01:06:21 xen06 kernel:  Xen: 00000001ffffe000 - 0000000200000000
> (reserved)
> Feb 24 01:06:21 xen06 kernel:  Xen: 0000000200000000 - 00000003bffbe000
> (usable)
> Feb 24 01:06:21 xen06 kernel: DMI 2.4 present.
> Feb 24 01:06:21 xen06 kernel: last_pfn = 0x3bffbe max_arch_pfn = 0x400000000
> Feb 24 01:06:21 xen06 kernel: x86 PAT enabled: cpu 0, old 0x50100070406, new
> 0x7010600070106
> Feb 24 01:06:21 xen06 kernel: last_pfn = 0x40000 max_arch_pfn = 0x400000000
> 
> Thanks.
> 
> Kindest regards,
> Giam Teck Choon

> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: kernel BUG at arch/x86/xen/mmu.c:1860!

[Teck Choon Giam]

[-- Attachment #1.1: Type: text/plain, Size: 364 bytes --]

On Tue, Mar 1, 2011 at 12:20 AM, Konrad Rzeszutek Wilk <
konrad.wilk@oracle.com> wrote:

> > This is just an update from my testing.
> >
> > I tried with latest xen/next-2.6.32.x and same bug still there:
>
> Grrrrrrrr..
>
> Let me setup a machine to reproduce this. Is the distro you are using
> still CentOS 5?
>

Yes.

Thanks.

Kindest regards,
Giam Teck Choon

[-- Attachment #1.2: Type: text/html, Size: 708 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: kernel BUG at arch/x86/xen/mmu.c:1860!

[Konrad Rzeszutek Wilk]

On Tue, Mar 01, 2011 at 05:59:54PM +0800, Teck Choon Giam wrote:
> On Tue, Mar 1, 2011 at 12:20 AM, Konrad Rzeszutek Wilk <
> konrad.wilk@oracle.com> wrote:
> 
> > > This is just an update from my testing.
> > >
> > > I tried with latest xen/next-2.6.32.x and same bug still there:
> >
> > Grrrrrrrr..
> >
> > Let me setup a machine to reproduce this. Is the distro you are using
> > still CentOS 5?
> >
> 
> Yes.

OK, I got a machine with CentOS 5.5 installed. Had some trouble getting
the kernel to boot - a different issue that the one you are seeing however.

What arguments are you using for your dom0 and Xen hypervisor? 

Re: kernel BUG at arch/x86/xen/mmu.c:1860!

[Teck Choon Giam]

[-- Attachment #1.1: Type: text/plain, Size: 1142 bytes --]

On Fri, Mar 4, 2011 at 6:16 AM, Konrad Rzeszutek Wilk <
konrad.wilk@oracle.com> wrote:

> On Tue, Mar 01, 2011 at 05:59:54PM +0800, Teck Choon Giam wrote:
> > On Tue, Mar 1, 2011 at 12:20 AM, Konrad Rzeszutek Wilk <
> > konrad.wilk@oracle.com> wrote:
> >
> > > > This is just an update from my testing.
> > > >
> > > > I tried with latest xen/next-2.6.32.x and same bug still there:
> > >
> > > Grrrrrrrr..
> > >
> > > Let me setup a machine to reproduce this. Is the distro you are using
> > > still CentOS 5?
> > >
> >
> > Yes.
>
> OK, I got a machine with CentOS 5.5 installed. Had some trouble getting
> the kernel to boot - a different issue that the one you are seeing however.
>
> What arguments are you using for your dom0 and Xen hypervisor?
>


Example in one of my test server:

title CentOS (2.6.32.29-0.xen.pvops.choon.centos5)
    root (hd0,0)
    kernel /xen.gz dom0_mem=1024M loglvl=all guest_loglvl=all cpuidle=0
cpufreq=none
    module /vmlinuz-2.6.32.29-0.xen.pvops.choon.centos5 ro root=/dev/md1
panic=5 panic_timeout=5
    module /initrd-2.6.32.29-0.xen.pvops.choon.centos5.img

Thanks.

Kindest regards,
Giam Teck Choon

[-- Attachment #1.2: Type: text/html, Size: 1661 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: kernel BUG at arch/x86/xen/mmu.c:1860!

[Fajar A. Nugraha]

[-- Attachment #1.1: Type: text/plain, Size: 1385 bytes --]

On Fri, Mar 4, 2011 at 12:30 PM, Teck Choon Giam <giamteckchoon@gmail.com>wrote:

>
>
> On Fri, Mar 4, 2011 at 6:16 AM, Konrad Rzeszutek Wilk <
> konrad.wilk@oracle.com> wrote:
>
>> On Tue, Mar 01, 2011 at 05:59:54PM +0800, Teck Choon Giam wrote:
>> > On Tue, Mar 1, 2011 at 12:20 AM, Konrad Rzeszutek Wilk <
>> > konrad.wilk@oracle.com> wrote:
>> >
>> > > > This is just an update from my testing.
>> > > >
>> > > > I tried with latest xen/next-2.6.32.x and same bug still there:
>> > >
>> > > Grrrrrrrr..
>> > >
>> > > Let me setup a machine to reproduce this. Is the distro you are using
>> > > still CentOS 5?
>> > >
>> >
>> > Yes.
>>
>> OK, I got a machine with CentOS 5.5 installed. Had some trouble getting
>> the kernel to boot - a different issue that the one you are seeing
>> however.
>>
>> What arguments are you using for your dom0 and Xen hypervisor?
>>
>
>
> Example in one of my test server:
>
> title CentOS (2.6.32.29-0.xen.pvops.choon.centos5)
>     root (hd0,0)
>     kernel /xen.gz dom0_mem=1024M loglvl=all guest_loglvl=all cpuidle=0
> cpufreq=none
>     module /vmlinuz-2.6.32.29-0.xen.pvops.choon.centos5 ro root=/dev/md1
> panic=5 panic_timeout=5
>     module /initrd-2.6.32.29-0.xen.pvops.choon.centos5.img
>


RHEL/Centos5 also
needs CONFIG_SYSFS_DEPRECATED=y, CONFIG_SYSFS_DEPRECATED_V2=y for 2.6.32
kernel to work correctly with userland tools.

-- 
Fajar

[-- Attachment #1.2: Type: text/html, Size: 2179 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: kernel BUG at arch/x86/xen/mmu.c:1860!

[Teck Choon Giam]

[-- Attachment #1.1: Type: text/plain, Size: 1710 bytes --]

On Fri, Mar 4, 2011 at 2:15 PM, Fajar A. Nugraha <list@fajar.net> wrote:

> On Fri, Mar 4, 2011 at 12:30 PM, Teck Choon Giam <giamteckchoon@gmail.com>wrote:
>
>>
>>
>> On Fri, Mar 4, 2011 at 6:16 AM, Konrad Rzeszutek Wilk <
>> konrad.wilk@oracle.com> wrote:
>>
>>> On Tue, Mar 01, 2011 at 05:59:54PM +0800, Teck Choon Giam wrote:
>>> > On Tue, Mar 1, 2011 at 12:20 AM, Konrad Rzeszutek Wilk <
>>> > konrad.wilk@oracle.com> wrote:
>>> >
>>> > > > This is just an update from my testing.
>>> > > >
>>> > > > I tried with latest xen/next-2.6.32.x and same bug still there:
>>> > >
>>> > > Grrrrrrrr..
>>> > >
>>> > > Let me setup a machine to reproduce this. Is the distro you are using
>>> > > still CentOS 5?
>>> > >
>>> >
>>> > Yes.
>>>
>>> OK, I got a machine with CentOS 5.5 installed. Had some trouble getting
>>> the kernel to boot - a different issue that the one you are seeing
>>> however.
>>>
>>> What arguments are you using for your dom0 and Xen hypervisor?
>>>
>>
>>
>> Example in one of my test server:
>>
>> title CentOS (2.6.32.29-0.xen.pvops.choon.centos5)
>>     root (hd0,0)
>>     kernel /xen.gz dom0_mem=1024M loglvl=all guest_loglvl=all cpuidle=0
>> cpufreq=none
>>     module /vmlinuz-2.6.32.29-0.xen.pvops.choon.centos5 ro root=/dev/md1
>> panic=5 panic_timeout=5
>>     module /initrd-2.6.32.29-0.xen.pvops.choon.centos5.img
>>
>
>
> RHEL/Centos5 also
> needs CONFIG_SYSFS_DEPRECATED=y, CONFIG_SYSFS_DEPRECATED_V2=y for 2.6.32
> kernel to work correctly with userland tools.
>

Thanks Fajar ;)

Yes.  All are set in my kernel config and I believed Konrad has my kernel
config.  I didn't use nomodeset as my servers don't have those graphic
cards... ...

Kindest regards,
Giam Teck Choon

[-- Attachment #1.2: Type: text/html, Size: 2797 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: kernel BUG at arch/x86/xen/mmu.c:1860!

[Konrad Rzeszutek Wilk]

[-- Attachment #1: Type: text/plain, Size: 2582 bytes --]

> Yes.  All are set in my kernel config and I believed Konrad has my kernel
> config.  I didn't use nomodeset as my servers don't have those graphic
> cards... ...

I am having a hard-time reproducing this. I made six LV's:
[root@tst011 ~]# lvs
  LV                   VG         Attr   LSize  Origin      Snap%  Move Log Copy%  Convert
  LogVol00             VolGroup00 -wi-ao 17.56G                                           
  LogVol01             VolGroup00 -wi-ao  5.81G                                           
  data-1.ext3          XenGroup   -wi-ao 10.00G                                           
  data-2.ext3          XenGroup   owi-ao 10.00G                                           
  data-2.ext3-snapshot XenGroup   swi-a-  1.00G data-2.ext3   2.41                        
  data-3.ext3          XenGroup   -wi-a- 10.00G                                           
  scratch-1.ext3       XenGroup   -wi-ao 10.00G                                           
  scratch-2.ext3       XenGroup   -wi-ao 10.00G                                           
  scratch-3.ext3       XenGroup   -wi-a- 10.00G                                           
[root@tst011 ~]# 

Where each scratch-X/data-X was prepared with 'mkfs.ext3'

And ran two guests, where each guest configuration looks as so:

kernel="/mnt/lab/latest/vmlinuz"
ramdisk="/mnt/lab/latest/initramfs.cpio.gz"
extra="console=hvc0 debug"
memory=768
vcpus=4
on_crash="preserve"
#vif = [ 'mac=00:0f:4b:00:00:68, bridge=switch' ]
vfb = [ 'vnc=1, vnclisten=0.0.0.0,vncunused=1']
disk = [ 'phy:/dev/XenGroup/scratch-1.ext3,xvda,w', 'phy:/dev/XenGroup/data-1.ext3,xvdb,w']

(the other is using -2 obviously).

And each guest is running 'mount /dev/xvda /mnt-1;(cd /mnt-1;fio iometer-file-access-server)'
to produce I/Os on the xvdb/xvda disks.

As those guests are chugging along, I ran your script with:

root      6250  4438  0 10:12 pts/4    00:00:00 /bin/sh ./lvm-test.sh loop 100 1G 5

(I tried 100 1G 0)

and so far it is running.... how long should I wait until I hit this problem?

This is on CentOS 5.5 and also on Fedora Core 13. Dom0 and DomU are all x86_64.

Dom0 is:

commit 892d2f052e979cf1916647c752b94cf62ec1c6dc
Merge: 35e2e28... 376faec...
Author: Jeremy Fitzhardinge <jeremy.fitzhardinge@citrix.com>
Date:   Fri Feb 11 13:31:03 2011 -0800

    Merge commit 'v2.6.32.28' into xen/next-2.6.32

(plus one patch I just posted - but that is to fix the serial console, so
it is not relevant to this problem).

DomU is 2.6.38 kernel, but I can swap over to the same as Dom0..

Attached is the script I am using.

[-- Attachment #2: iometer-file-access-server --]
[-- Type: text/plain, Size: 439 bytes --]

# This job file tries to mimic the Intel IOMeter File Server Access Pattern
[global]
description=Emulation of Intel IOmeter File Server Access Pattern

[iometer]
bssplit=512/10:1k/5:2k/5:4k/60:8k/2:16k/4:32k/4:64k/10
rw=randrw
rwmixread=80
direct=1
size=4g
ioengine=libaio
# IOMeter defines the server loads as the following:
# iodepth=1	Linear
# iodepth=4	Very Light
# iodepth=8	Light
# iodepth=64	Moderate
# iodepth=256	Heavy
iodepth=64

[-- Attachment #3: lvm-test.sh --]
[-- Type: application/x-sh, Size: 3237 bytes --]

[-- Attachment #4: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: kernel BUG at arch/x86/xen/mmu.c:1860!

[Konrad Rzeszutek Wilk]

On Tue, Mar 08, 2011 at 02:29:50PM -0500, Konrad Rzeszutek Wilk wrote:
> > Yes.  All are set in my kernel config and I believed Konrad has my kernel
> > config.  I didn't use nomodeset as my servers don't have those graphic
> > cards... ...
> 
> I am having a hard-time reproducing this. I made six LV's:

Aha!

respawning too fast: disabled for 5 minutes
INIT: Id "co" respawning too fast: disabled for 5 minutes
INIT: Id "co" respawning too fast: disabled for 5 minutes
INIT: Id "co" respawning too fast: disabled for 5 minutes
(XEN) mm.c:2364:d0 Bad type (saw 7400000000000001 != exp 3000000000000000) for mfn 354e0 (pfn c76d7)
(XEN) mm.c:942:d0 Attempt : entry 255
(XEN) mm.c:2117:d0 Error while validating mfn 35620 (pfn c7417) for type 4000000000000000: caf=8000000000000003 taf=4000000000000001
(XEN) mm.c:2733:d0 Error while pinning mfn 35620
(XEN) mm.c:2364:d0 Bad type (saw 7400000000000001 != exp 3000000000000000) for mfn 354e0 (pfn c76d7)
(XEN) mm.c:942:d0 Attempt to create linear p.t. with write perms
(XEN) mm.c:1487:d0 Failure in alloc_l4_table: entry 255
(XEN) mm.c:2117:d0 Error while validating mfn 3b286 (pfn c187d) for type 4000000000000000: caf=8000000000000003 taf=4000000000000001
(XEN) mm.c:2733:d0 Error while pinning mfn 3b286
(XEN) mm.c:2364:d0 Bad type (saw 7400000000000001 != exp 3000000000000000) for mfn 354e0 (pfn c76d7)
(XEN) mm.c:942:d0 Attempt to create linear p.t. with write perms
(XEN) mm.c:1487:d0 Failure in alloc_l4_table: entry 255
(XEN) mm.c:2117:d0 Error while validating mfn 35620 (pfn c7417) for type 4000000000000000: caf=8000000000000003 taf=4000000000000001
(XEN) mm.c:2500:d0 Error while installing new baseptr 35620
(XEN) mm.c:2364:d0 Bad type (saw 7400000000000001 != exp 3000000000000000) for mfn 354e0 (pfn c76d7)
(XEN) mm.c:942:d0 Attempt to create linear p.t. with write perms
(XEN) mm.c:1487:d0 Failure in alloc_l4_table: entry 255
(XEN) mm.c:2117:d0 Error while validating mfn 3b286 (pfn c187d) for type 4000000000000000: caf=8000000000000003 taf=4000000000000001
(XEN) mm.c:2825:d0 Error while installing new mfn 3b286
(XEN) mm.c:2794:d0 Mfn 35620 not pinned
(XEN) mm.c:2794:d0 Mfn 3b286 not pinned

Takes a bit of time to reproduce it, but I can reproduce it on my CentOS 5.5 OS.

Re: kernel BUG at arch/x86/xen/mmu.c:1860!

[Teck Choon Giam]

[-- Attachment #1.1: Type: text/plain, Size: 2932 bytes --]

On Wed, Mar 9, 2011 at 4:10 AM, Konrad Rzeszutek Wilk <
konrad.wilk@oracle.com> wrote:

> On Tue, Mar 08, 2011 at 02:29:50PM -0500, Konrad Rzeszutek Wilk wrote:
> > > Yes.  All are set in my kernel config and I believed Konrad has my
> kernel
> > > config.  I didn't use nomodeset as my servers don't have those graphic
> > > cards... ...
> >
> > I am having a hard-time reproducing this. I made six LV's:
>
> Aha!
>
> respawning too fast: disabled for 5 minutes
> INIT: Id "co" respawning too fast: disabled for 5 minutes
> INIT: Id "co" respawning too fast: disabled for 5 minutes
> INIT: Id "co" respawning too fast: disabled for 5 minutes
> (XEN) mm.c:2364:d0 Bad type (saw 7400000000000001 != exp 3000000000000000)
> for mfn 354e0 (pfn c76d7)
> (XEN) mm.c:942:d0 Attempt : entry 255
> (XEN) mm.c:2117:d0 Error while validating mfn 35620 (pfn c7417) for type
> 4000000000000000: caf=8000000000000003 taf=4000000000000001
> (XEN) mm.c:2733:d0 Error while pinning mfn 35620
> (XEN) mm.c:2364:d0 Bad type (saw 7400000000000001 != exp 3000000000000000)
> for mfn 354e0 (pfn c76d7)
> (XEN) mm.c:942:d0 Attempt to create linear p.t. with write perms
> (XEN) mm.c:1487:d0 Failure in alloc_l4_table: entry 255
> (XEN) mm.c:2117:d0 Error while validating mfn 3b286 (pfn c187d) for type
> 4000000000000000: caf=8000000000000003 taf=4000000000000001
> (XEN) mm.c:2733:d0 Error while pinning mfn 3b286
> (XEN) mm.c:2364:d0 Bad type (saw 7400000000000001 != exp 3000000000000000)
> for mfn 354e0 (pfn c76d7)
> (XEN) mm.c:942:d0 Attempt to create linear p.t. with write perms
> (XEN) mm.c:1487:d0 Failure in alloc_l4_table: entry 255
> (XEN) mm.c:2117:d0 Error while validating mfn 35620 (pfn c7417) for type
> 4000000000000000: caf=8000000000000003 taf=4000000000000001
> (XEN) mm.c:2500:d0 Error while installing new baseptr 35620
> (XEN) mm.c:2364:d0 Bad type (saw 7400000000000001 != exp 3000000000000000)
> for mfn 354e0 (pfn c76d7)
> (XEN) mm.c:942:d0 Attempt to create linear p.t. with write perms
> (XEN) mm.c:1487:d0 Failure in alloc_l4_table: entry 255
> (XEN) mm.c:2117:d0 Error while validating mfn 3b286 (pfn c187d) for type
> 4000000000000000: caf=8000000000000003 taf=4000000000000001
> (XEN) mm.c:2825:d0 Error while installing new mfn 3b286
> (XEN) mm.c:2794:d0 Mfn 35620 not pinned
> (XEN) mm.c:2794:d0 Mfn 3b286 not pinned
>
> Takes a bit of time to reproduce it, but I can reproduce it on my CentOS
> 5.5 OS.
>

Yep... it takes sometime to hit this bug and if you run my test script
without any guest running will be faster to reproduce it.

May I know what is your normal development platform/distribution for xen?  I
hope this isn't just distribution specific issue for PVOPS
stable/xen-2.6.32.x Dom0... ... mostly not as from this thread alone I see
someone saying they hit this bug as well and I believe they are not running
CentOS 5.x... guess is debian based distribution.

Thanks.

Kindest regards,
Giam Teck Choon

[-- Attachment #1.2: Type: text/html, Size: 3438 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

kernel BUG at arch/x86/xen/mmu.c:1860!

[Guido Hecken]

[-- Attachment #1.1: Type: text/plain, Size: 3500 bytes --]

Hi,

 

maybe I've got the same problem on debian squeeze.

I opened a bug report some time ago but diddn't get any feedback till now.

Perhaps there were not enough informations...

 

http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=614400

 

Guido

 

 

Von: xen-devel-bounces@lists.xensource.com [mailto:xen-devel-bounces@lists.xensource.com] Im Auftrag von Teck Choon Giam
Gesendet: Dienstag, 8. März 2011 21:21
An: Konrad Rzeszutek Wilk
Cc: xen-devel@lists.xensource.com; Fajar A. Nugraha
Betreff: Re: [Xen-devel] kernel BUG at arch/x86/xen/mmu.c:1860!

 

 

On Wed, Mar 9, 2011 at 4:10 AM, Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> wrote:

On Tue, Mar 08, 2011 at 02:29:50PM -0500, Konrad Rzeszutek Wilk wrote:
> > Yes.  All are set in my kernel config and I believed Konrad has my kernel
> > config.  I didn't use nomodeset as my servers don't have those graphic
> > cards... ...
>
> I am having a hard-time reproducing this. I made six LV's:

Aha!


respawning too fast: disabled for 5 minutes

INIT: Id "co" respawning too fast: disabled for 5 minutes
INIT: Id "co" respawning too fast: disabled for 5 minutes
INIT: Id "co" respawning too fast: disabled for 5 minutes
(XEN) mm.c:2364:d0 Bad type (saw 7400000000000001 != exp 3000000000000000) for mfn 354e0 (pfn c76d7)
(XEN) mm.c:942:d0 Attempt : entry 255
(XEN) mm.c:2117:d0 Error while validating mfn 35620 (pfn c7417) for type 4000000000000000: caf=8000000000000003 taf=4000000000000001
(XEN) mm.c:2733:d0 Error while pinning mfn 35620
(XEN) mm.c:2364:d0 Bad type (saw 7400000000000001 != exp 3000000000000000) for mfn 354e0 (pfn c76d7)
(XEN) mm.c:942:d0 Attempt to create linear p.t. with write perms
(XEN) mm.c:1487:d0 Failure in alloc_l4_table: entry 255
(XEN) mm.c:2117:d0 Error while validating mfn 3b286 (pfn c187d) for type 4000000000000000: caf=8000000000000003 taf=4000000000000001
(XEN) mm.c:2733:d0 Error while pinning mfn 3b286
(XEN) mm.c:2364:d0 Bad type (saw 7400000000000001 != exp 3000000000000000) for mfn 354e0 (pfn c76d7)
(XEN) mm.c:942:d0 Attempt to create linear p.t. with write perms
(XEN) mm.c:1487:d0 Failure in alloc_l4_table: entry 255
(XEN) mm.c:2117:d0 Error while validating mfn 35620 (pfn c7417) for type 4000000000000000: caf=8000000000000003 taf=4000000000000001
(XEN) mm.c:2500:d0 Error while installing new baseptr 35620
(XEN) mm.c:2364:d0 Bad type (saw 7400000000000001 != exp 3000000000000000) for mfn 354e0 (pfn c76d7)
(XEN) mm.c:942:d0 Attempt to create linear p.t. with write perms
(XEN) mm.c:1487:d0 Failure in alloc_l4_table: entry 255
(XEN) mm.c:2117:d0 Error while validating mfn 3b286 (pfn c187d) for type 4000000000000000: caf=8000000000000003 taf=4000000000000001
(XEN) mm.c:2825:d0 Error while installing new mfn 3b286
(XEN) mm.c:2794:d0 Mfn 35620 not pinned
(XEN) mm.c:2794:d0 Mfn 3b286 not pinned

Takes a bit of time to reproduce it, but I can reproduce it on my CentOS 5.5 OS.


Yep... it takes sometime to hit this bug and if you run my test script without any guest running will be faster to reproduce it.

May I know what is your normal development platform/distribution for xen?  I hope this isn't just distribution specific issue for PVOPS stable/xen-2.6.32.x Dom0... ... mostly not as from this thread alone I see someone saying they hit this bug as well and I believe they are not running CentOS 5.x... guess is debian based distribution.

Thanks.

Kindest regards,
Giam Teck Choon

 


[-- Attachment #1.2: Type: text/html, Size: 9217 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

[SPAM] Re: kernel BUG at arch/x86/xen/mmu.c:1860!

[tjaouen]

Teck Choon Giam wrote:
> 
> 
> I hope this isn't just distribution specific issue for PVOPS
> stable/xen-2.6.32.x Dom0... ... mostly not as from this thread alone I see
> someone saying they hit this bug as well and I believe they are not
> running
> CentOS 5.x... guess is debian based distribution.
> 
> 

Debian/Squeeze ""Stable""

Debian GNU/Linux 6.0                                                                                                                                                                           
Linux xencave 2.6.32-5-xen-amd64 #1 SMP Wed Jan 12 05:46:49 UTC 2011 x86_64
GNU/Linux                                                                                                          


March 1, I enter this command:

  lvcreate -s -L 5G -n databank_snap /dev/vgraid1/databank

 .... and BUG in /var/log/syslog:

Mar  1 21:23:56 xencave kernel: [1840078.405905] ------------[ cut here
]------------
Mar  1 21:23:56 xencave kernel: [1840078.405937] kernel BUG at
/build/buildd-linux-2.6_2.6.32-30-amd64-d4MbNM/linux-2.6-2.6.32/debian/build/source_amd64_xen/arch/x86/xen/mmu.c:1649!
Mar  1 21:23:56 xencave kernel: [1840078.405981] invalid opcode: 0000 [#1]
SMP 
Mar  1 21:23:56 xencave kernel: [1840078.406010] last sysfs file:
/sys/devices/virtual/bdi/253:35/uevent
Mar  1 21:23:56 xencave kernel: [1840078.406037] CPU 0 
Mar  1 21:23:56 xencave kernel: [1840078.406065] Modules linked in:
dm_snapshot nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack
xt_physdev iptable_filter ip_tables x_tables xen_evtchn xenfs bridge stp
coretemp it87 hwmon_vid fuse loop i915 drm_kms_helper drm snd_pcsp processor
i2c_algo_bit button video acpi_processor i2c_i801 snd_pcm snd_timer rng_core
serio_raw i2c_core evdev snd output soundcore snd_page_alloc usbhid hid ext3
jbd mbcache dm_mod raid1 md_mod sg sr_mod cdrom sd_mod crc_t10dif
ata_generic uhci_hcd ata_piix libata r8169 mii thermal thermal_sys ehci_hcd
scsi_mod usbcore nls_base [last unloaded: scsi_wait_scan]
Mar  1 21:23:56 xencave kernel: [1840078.406505] Pid: 14525, comm: udevd Not
tainted 2.6.32-5-xen-amd64 #1 945GM-S2
Mar  1 21:23:56 xencave kernel: [1840078.406539] RIP: e030:[]  []
pin_pagetable_pfn+0x2d/0x36
Mar  1 21:23:56 xencave kernel: [1840078.406586] RSP: e02b:ffff88000df33e08 
EFLAGS: 00010282
Mar  1 21:23:56 xencave kernel: [1840078.406608] RAX: 00000000ffffffea RBX:
000000000000df31 RCX: 0000000000000001
Mar  1 21:23:56 xencave kernel: [1840078.406642] RDX: 0000000000000000 RSI:
0000000000000001 RDI: ffff88000df33e08
Mar  1 21:23:56 xencave kernel: [1840078.406677] RBP: ffff8800028bb480 R08:
0000000000000988 R09: ffffea000030d2b8
Mar  1 21:23:56 xencave kernel: [1840078.406711] R10: 0000000000007ff0 R11:
ffff880000000838 R12: ffff88004f026a48
Mar  1 21:23:56 xencave kernel: [1840078.406745] R13: ffff880002be3f18 R14:
ffff8800513f2a60 R15: ffff88004f026a48
Mar  1 21:23:56 xencave kernel: [1840078.406784] FS:  00007ff6a99077a0(0000)
GS:ffff88000390b000(0000) knlGS:0000000000000000
Mar  1 21:23:56 xencave kernel: [1840078.406820] CS:  e033 DS: 0000 ES: 0000
CR0: 000000008005003b
Mar  1 21:23:56 xencave kernel: [1840078.406842] CR2: 00007ff6a9215876 CR3:
000000004af1a000 CR4: 0000000000002660
Mar  1 21:23:56 xencave kernel: [1840078.406878] DR0: 0000000000000000 DR1:
0000000000000000 DR2: 0000000000000000
Mar  1 21:23:56 xencave kernel: [1840078.406913] DR3: 0000000000000000 DR6:
00000000ffff0ff0 DR7: 0000000000000400
Mar  1 21:23:56 xencave kernel: [1840078.406948] Process udevd (pid: 14525,
threadinfo ffff88000df32000, task ffff8800513f2a60)
Mar  1 21:23:56 xencave kernel: [1840078.406985] Stack:
Mar  1 21:23:56 xencave kernel: [1840078.407003]  0000000000000000
00000000000af260 ffffea000030d2b8 000000000000df31
Mar  1 21:23:56 xencave kernel: [1840078.407042] <0> ffff8800028bb480
ffffffff810cd4e2 ffff88004af24ed0 00007ff6a9215876
Mar  1 21:23:56 xencave kernel: [1840078.407095] <0> 000000004af24000
ffffffff810cb394 ffff880002be3f18 00007ff6a9215876
Mar  1 21:23:56 xencave kernel: [1840078.407164] Call Trace:
Mar  1 21:23:56 xencave kernel: [1840078.407187]  [] ? __pte_alloc+0x6b/0xc6
Mar  1 21:23:56 xencave kernel: [1840078.407212]  [] ? pmd_alloc+0x28/0x5b
Mar  1 21:23:56 xencave kernel: [1840078.407235]  [] ?
handle_mm_fault+0xce/0x80f
Mar  1 21:23:56 xencave kernel: [1840078.407262]  [] ? page_fault+0x25/0x30
Mar  1 21:23:56 xencave kernel: [1840078.407287]  [] ? error_exit+0x2a/0x60
Mar  1 21:23:56 xencave kernel: [1840078.407311]  [] ?
retint_restore_args+0x5/0x6
Mar  1 21:23:56 xencave kernel: [1840078.407336]  [] ?
do_page_fault+0x2e0/0x2fc
Mar  1 21:23:56 xencave kernel: [1840078.407361]  [] ? page_fault+0x25/0x30
Mar  1 21:23:56 xencave kernel: [1840078.407383] Code: ec 28 89 3c 24 48 89
f7 e8 a2 fd ff ff 48 89 e7 48 89 44 24 08 be 01 00 00 00 31 d2 41 ba f0 7f
00 00 e8 b0 cc ff ff 85 c0 74 04 <0f> 0b eb fe 48 83 c4 28 c3 55 49 89 ca 48
89 d5 40 88 f1 48 89 
Mar  1 21:23:56 xencave kernel: [1840078.407702] RIP  []
pin_pagetable_pfn+0x2d/0x36
Mar  1 21:23:56 xencave kernel: [1840078.407731]  RSP 




--
View this message in context: http://xen.1045712.n5.nabble.com/kernel-BUG-at-arch-x86-xen-mmu-c-1860-tp3318567p3414620.html
Sent from the Xen - Dev mailing list archive at Nabble.com.

Re: kernel BUG at arch/x86/xen/mmu.c:1860!

[Andreas Olsowski]

[-- Attachment #1.1: Type: text/plain, Size: 5206 bytes --]

Well, this is too bad.

I encountered this bug when xen 4.0 was released, around the time 
development on 2.6.31 was halted.

That is why i stuck with 2.6.31 when everyone else went with 2.6.32,
because i determined 2.6.32 was not fit for duty and im guessing it 
still isnt today.

The bug occures on 2.6.32 xen kernels ( maybe even newer ones) and is 
distribution unrelated, i was running debian 5.0 then, i am running 6.0 
testing now and even have tried compiling all the userland stuff myself.

This is error can be encountered during a number of different actions:
1.) any action with lvm (start, stop, create, delete)
2.) while starting multipathd (restarting too, of course)

Sometimes the box only hangs there and no further device mapper 
interactions are possible. This is where i got my syslog entry from.

Back in 2010 i had to serial console the server and stuff like that to 
see the whole error.


my guess is everything one does with the device mapper can and will 
trigger this sooner or later.

Does anybody have any kind of insight on what the problem may be?

------------
Here is my syslog part when i ran "/etc/init.d/multipath-tools restart":

Mar  9 00:24:10 memoryana multipathd: mpatha: stop event checker thread 
(140606587918080)
Mar  9 00:24:10 memoryana multipathd: mpathb: stop event checker thread 
(140606587885312)
Mar  9 00:24:10 memoryana multipathd: mpathc: stop event checker thread 
(140606587852544)
Mar  9 00:24:10 memoryana kernel: ------------[ cut here ]------------
Mar  9 00:24:10 memoryana kernel: kernel BUG at arch/x86/xen/mmu.c:1872!
Mar  9 00:24:10 memoryana kernel: invalid opcode: 0000 [#1] SMP
Mar  9 00:24:10 memoryana kernel: last sysfs file: 
/sys/devices/pci0000:00/0000:00:07.0/0000:04:00.1/host3/rport-3:0-2/target3:0:2/3:0:2:0/state
Mar  9 00:24:10 memoryana kernel: CPU 1
Mar  9 00:24:10 memoryana kernel: Modules linked in: dm_round_robin 
dm_multipath qla2xxx
Mar  9 00:24:10 memoryana kernel: Pid: 10662, comm: multipath-tools Not 
tainted 2.6.32.28-xen0 #4 PowerEdge R610
Mar  9 00:24:10 memoryana kernel: RIP: e030:[<ffffffff8100d471>] 
[<ffffffff8100d471>] pin_pagetable_pfn+0x31/0x60
Mar  9 00:24:10 memoryana kernel: RSP: e02b:ffff8800c3101df8  EFLAGS: 
00010282
Mar  9 00:24:10 memoryana kernel: RAX: 00000000ffffffea RBX: 
ffff8800cc4c3400 RCX: 0000000000000003
Mar  9 00:24:10 memoryana kernel: RDX: 0000000000000000 RSI: 
0000000000000001 RDI: ffff8800c3101df8
Mar  9 00:24:10 memoryana kernel: RBP: ffff8800c3135b60 R08: 
00003ffffffff000 R09: ffff880000000000
Mar  9 00:24:10 memoryana kernel: R10: 0000000000007ff0 R11: 
0000000000000246 R12: 00000000000cc302
Mar  9 00:24:10 memoryana kernel: R13: 0000000000000000 R14: 
ffff8800c374cc60 R15: ffff8800c374cc60
Mar  9 00:24:10 memoryana kernel: FS:  00007f60add15700(0000) 
GS:ffff880028055000(0000) knlGS:0000000000000000
Mar  9 00:24:10 memoryana kernel: CS:  e033 DS: 0000 ES: 0000 CR0: 
000000008005003b
Mar  9 00:24:10 memoryana kernel: CR2: 00007f60ad841876 CR3: 
00000000cef79000 CR4: 0000000000002660
Mar  9 00:24:10 memoryana kernel: DR0: 0000000000000000 DR1: 
0000000000000000 DR2: 0000000000000000
Mar  9 00:24:10 memoryana kernel: DR3: 0000000000000000 DR6: 
00000000ffff0ff0 DR7: 0000000000000400
Mar  9 00:24:10 memoryana kernel: Process multipath-tools (pid: 10662, 
threadinfo ffff8800c3100000, task ffff8800cc01cbc0)
Mar  9 00:24:10 memoryana kernel: Stack:
Mar  9 00:24:10 memoryana kernel: 0000000000000000 00000000008e8302 
ffff8800cc4c3400 ffff8800c3135b60
Mar  9 00:24:10 memoryana kernel: <0> 00000000000cc302 ffffffff810b0382 
00007f60ad841876 ffff8800c30b4c10
Mar  9 00:24:10 memoryana kernel: <0> 00000000000100e0 0000000000000000 
ffff8800c374cc60 ffffffff810b3595
Mar  9 00:24:10 memoryana kernel: Call Trace:
Mar  9 00:24:10 memoryana kernel: [<ffffffff810b0382>] ? 
__pte_alloc+0xf2/0x120
Mar  9 00:24:10 memoryana kernel: [<ffffffff810b3595>] ? 
handle_mm_fault+0xa45/0xab0
Mar  9 00:24:10 memoryana kernel: [<ffffffff8153cfe5>] ? 
page_fault+0x25/0x30
Mar  9 00:24:10 memoryana kernel: [<ffffffff8153d21a>] ? 
error_exit+0x2a/0x60
Mar  9 00:24:10 memoryana kernel: [<ffffffff8101481d>] ? 
retint_restore_args+0x5/0x6
Mar  9 00:24:10 memoryana kernel: [<ffffffff81038631>] ? 
do_page_fault+0x121/0x3c0
Mar  9 00:24:10 memoryana kernel: [<ffffffff812a2e0d>] ? 
__put_user_4+0x1d/0x30
Mar  9 00:24:10 memoryana kernel: [<ffffffff8153cfe5>] ? 
page_fault+0x25/0x30
Mar  9 00:24:10 memoryana kernel: Code: 57 c7 75 00 00 48 89 f0 89 3c 24 
74 27 48 89 44 24 08 48 89 e7 be 01 00 00 00 31 d2 41 ba f0 7f 00 00 e8 
d3 be ff ff 85 c0 74 04 <0f> 0b eb fe 48 83 c4 28 c3 48 89 f7 e8 6e f7 
ff ff 48 83 f8 ff
Mar  9 00:24:10 memoryana kernel: RIP  [<ffffffff8100d471>] 
pin_pagetable_pfn+0x31/0x60
Mar  9 00:24:10 memoryana kernel: RSP <ffff8800c3101df8>
Mar  9 00:24:10 memoryana kernel: ---[ end trace f4eae184c1a9f532 ]---
Mar  9 00:24:11 memoryana multipathd: --------shut down-------

-- 
Andreas Olsowski
Leuphana Universität Lüneburg
Rechen- und Medienzentrum
Scharnhorststraße 1, C7.015
21335 Lüneburg

Tel: ++49 4131 677 1309



[-- Attachment #1.2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 6595 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: kernel BUG at arch/x86/xen/mmu.c:1860!

[benco]

Hello,

I can confirm this bug, I'm using very similar configuration - Debian
Lenny/Squeeze on several servers connected to FC storage with enabled
multipathing.

One more thing - it was quite long ago, but I did not see this bug with Xen
4.0 pre-release versions with 2.6.31 kernel from Jeremy's tree. I'm
regularly updating my system to up-to-date version of Xen/PV-OPS kernel and 
this bug is here across whole development line from Xen 4.0/2.6.32 kernel 
until now:(

Roman

On Wed, Mar 09, 2011 at 01:06:39AM +0100, Andreas Olsowski wrote:
> Well, this is too bad.
> 
> I encountered this bug when xen 4.0 was released, around the time 
> development on 2.6.31 was halted.
> 
> That is why i stuck with 2.6.31 when everyone else went with 2.6.32,
> because i determined 2.6.32 was not fit for duty and im guessing it 
> still isnt today.
> 
> The bug occures on 2.6.32 xen kernels ( maybe even newer ones) and is 
> distribution unrelated, i was running debian 5.0 then, i am running 6.0 
> testing now and even have tried compiling all the userland stuff myself.
> 
> This is error can be encountered during a number of different actions:
> 1.) any action with lvm (start, stop, create, delete)
> 2.) while starting multipathd (restarting too, of course)
> 
> Sometimes the box only hangs there and no further device mapper 
> interactions are possible. This is where i got my syslog entry from.
> 
> Back in 2010 i had to serial console the server and stuff like that to 
> see the whole error.
> 
> 
> my guess is everything one does with the device mapper can and will 
> trigger this sooner or later.
> 
> Does anybody have any kind of insight on what the problem may be?
> 
> ------------
> Here is my syslog part when i ran "/etc/init.d/multipath-tools restart":
> 
> Mar  9 00:24:10 memoryana multipathd: mpatha: stop event checker thread 
> (140606587918080)
> Mar  9 00:24:10 memoryana multipathd: mpathb: stop event checker thread 
> (140606587885312)
> Mar  9 00:24:10 memoryana multipathd: mpathc: stop event checker thread 
> (140606587852544)
> Mar  9 00:24:10 memoryana kernel: ------------[ cut here ]------------
> Mar  9 00:24:10 memoryana kernel: kernel BUG at arch/x86/xen/mmu.c:1872!
> Mar  9 00:24:10 memoryana kernel: invalid opcode: 0000 [#1] SMP
> Mar  9 00:24:10 memoryana kernel: last sysfs file: 
> /sys/devices/pci0000:00/0000:00:07.0/0000:04:00.1/host3/rport-3:0-2/target3:0:2/3:0:2:0/state
> Mar  9 00:24:10 memoryana kernel: CPU 1
> Mar  9 00:24:10 memoryana kernel: Modules linked in: dm_round_robin 
> dm_multipath qla2xxx
> Mar  9 00:24:10 memoryana kernel: Pid: 10662, comm: multipath-tools Not 
> tainted 2.6.32.28-xen0 #4 PowerEdge R610
> Mar  9 00:24:10 memoryana kernel: RIP: e030:[<ffffffff8100d471>] 
> [<ffffffff8100d471>] pin_pagetable_pfn+0x31/0x60
> Mar  9 00:24:10 memoryana kernel: RSP: e02b:ffff8800c3101df8  EFLAGS: 
> 00010282
> Mar  9 00:24:10 memoryana kernel: RAX: 00000000ffffffea RBX: 
> ffff8800cc4c3400 RCX: 0000000000000003
> Mar  9 00:24:10 memoryana kernel: RDX: 0000000000000000 RSI: 
> 0000000000000001 RDI: ffff8800c3101df8
> Mar  9 00:24:10 memoryana kernel: RBP: ffff8800c3135b60 R08: 
> 00003ffffffff000 R09: ffff880000000000
> Mar  9 00:24:10 memoryana kernel: R10: 0000000000007ff0 R11: 
> 0000000000000246 R12: 00000000000cc302
> Mar  9 00:24:10 memoryana kernel: R13: 0000000000000000 R14: 
> ffff8800c374cc60 R15: ffff8800c374cc60
> Mar  9 00:24:10 memoryana kernel: FS:  00007f60add15700(0000) 
> GS:ffff880028055000(0000) knlGS:0000000000000000
> Mar  9 00:24:10 memoryana kernel: CS:  e033 DS: 0000 ES: 0000 CR0: 
> 000000008005003b
> Mar  9 00:24:10 memoryana kernel: CR2: 00007f60ad841876 CR3: 
> 00000000cef79000 CR4: 0000000000002660
> Mar  9 00:24:10 memoryana kernel: DR0: 0000000000000000 DR1: 
> 0000000000000000 DR2: 0000000000000000
> Mar  9 00:24:10 memoryana kernel: DR3: 0000000000000000 DR6: 
> 00000000ffff0ff0 DR7: 0000000000000400
> Mar  9 00:24:10 memoryana kernel: Process multipath-tools (pid: 10662, 
> threadinfo ffff8800c3100000, task ffff8800cc01cbc0)
> Mar  9 00:24:10 memoryana kernel: Stack:
> Mar  9 00:24:10 memoryana kernel: 0000000000000000 00000000008e8302 
> ffff8800cc4c3400 ffff8800c3135b60
> Mar  9 00:24:10 memoryana kernel: <0> 00000000000cc302 ffffffff810b0382 
> 00007f60ad841876 ffff8800c30b4c10
> Mar  9 00:24:10 memoryana kernel: <0> 00000000000100e0 0000000000000000 
> ffff8800c374cc60 ffffffff810b3595
> Mar  9 00:24:10 memoryana kernel: Call Trace:
> Mar  9 00:24:10 memoryana kernel: [<ffffffff810b0382>] ? 
> __pte_alloc+0xf2/0x120
> Mar  9 00:24:10 memoryana kernel: [<ffffffff810b3595>] ? 
> handle_mm_fault+0xa45/0xab0
> Mar  9 00:24:10 memoryana kernel: [<ffffffff8153cfe5>] ? 
> page_fault+0x25/0x30
> Mar  9 00:24:10 memoryana kernel: [<ffffffff8153d21a>] ? 
> error_exit+0x2a/0x60
> Mar  9 00:24:10 memoryana kernel: [<ffffffff8101481d>] ? 
> retint_restore_args+0x5/0x6
> Mar  9 00:24:10 memoryana kernel: [<ffffffff81038631>] ? 
> do_page_fault+0x121/0x3c0
> Mar  9 00:24:10 memoryana kernel: [<ffffffff812a2e0d>] ? 
> __put_user_4+0x1d/0x30
> Mar  9 00:24:10 memoryana kernel: [<ffffffff8153cfe5>] ? 
> page_fault+0x25/0x30
> Mar  9 00:24:10 memoryana kernel: Code: 57 c7 75 00 00 48 89 f0 89 3c 24 
> 74 27 48 89 44 24 08 48 89 e7 be 01 00 00 00 31 d2 41 ba f0 7f 00 00 e8 
> d3 be ff ff 85 c0 74 04 <0f> 0b eb fe 48 83 c4 28 c3 48 89 f7 e8 6e f7 
> ff ff 48 83 f8 ff
> Mar  9 00:24:10 memoryana kernel: RIP  [<ffffffff8100d471>] 
> pin_pagetable_pfn+0x31/0x60
> Mar  9 00:24:10 memoryana kernel: RSP <ffff8800c3101df8>
> Mar  9 00:24:10 memoryana kernel: ---[ end trace f4eae184c1a9f532 ]---
> Mar  9 00:24:11 memoryana multipathd: --------shut down-------
> 
> -- 
> Andreas Olsowski
> Leuphana Universität Lüneburg
> Rechen- und Medienzentrum
> Scharnhorststraße 1, C7.015
> 21335 Lüneburg
> 
> Tel: ++49 4131 677 1309
> 
> 



> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel


-- 
----------------------------------------------------------------------
  ,''`.       [benco] | mailto: benco@acid.sk | silc: /msg benco
 : :' :  -------------------------------------------------------------
 `. `'           GPG publickey: http://www.acid.sk/pubkey.asc
   `-      KF  =  0DF6 0592 74D2 F17A DACF  A5C3 1720 CB7C F54C F429

Re: kernel BUG at arch/x86/xen/mmu.c:1860!

[Sander Eikelenboom]

Haven't seen this one on my systen. I do use LVM, but i'm only using LVM as device mapper, without software raid etc.
Also using debian lenny/squeeze together with custom compiled xen and xen-2.6.32.x pvops kernel

So I would suspect the multipathing / raid part.

--
Sander


Friday, March 11, 2011, 7:38:00 PM, you wrote:

> Hello,

> I can confirm this bug, I'm using very similar configuration - Debian
> Lenny/Squeeze on several servers connected to FC storage with enabled
> multipathing.

> One more thing - it was quite long ago, but I did not see this bug with Xen
> 4.0 pre-release versions with 2.6.31 kernel from Jeremy's tree. I'm
> regularly updating my system to up-to-date version of Xen/PV-OPS kernel and 
> this bug is here across whole development line from Xen 4.0/2.6.32 kernel 
> until now:(

> Roman

> On Wed, Mar 09, 2011 at 01:06:39AM +0100, Andreas Olsowski wrote:
>> Well, this is too bad.
>> 
>> I encountered this bug when xen 4.0 was released, around the time 
>> development on 2.6.31 was halted.
>> 
>> That is why i stuck with 2.6.31 when everyone else went with 2.6.32,
>> because i determined 2.6.32 was not fit for duty and im guessing it 
>> still isnt today.
>> 
>> The bug occures on 2.6.32 xen kernels ( maybe even newer ones) and is 
>> distribution unrelated, i was running debian 5.0 then, i am running 6.0 
>> testing now and even have tried compiling all the userland stuff myself.
>> 
>> This is error can be encountered during a number of different actions:
>> 1.) any action with lvm (start, stop, create, delete)
>> 2.) while starting multipathd (restarting too, of course)
>> 
>> Sometimes the box only hangs there and no further device mapper 
>> interactions are possible. This is where i got my syslog entry from.
>> 
>> Back in 2010 i had to serial console the server and stuff like that to 
>> see the whole error.
>> 
>> 
>> my guess is everything one does with the device mapper can and will 
>> trigger this sooner or later.
>> 
>> Does anybody have any kind of insight on what the problem may be?
>> 
>> ------------
>> Here is my syslog part when i ran "/etc/init.d/multipath-tools restart":
>> 
>> Mar  9 00:24:10 memoryana multipathd: mpatha: stop event checker thread 
>> (140606587918080)
>> Mar  9 00:24:10 memoryana multipathd: mpathb: stop event checker thread 
>> (140606587885312)
>> Mar  9 00:24:10 memoryana multipathd: mpathc: stop event checker thread 
>> (140606587852544)
>> Mar  9 00:24:10 memoryana kernel: ------------[ cut here ]------------
>> Mar  9 00:24:10 memoryana kernel: kernel BUG at arch/x86/xen/mmu.c:1872!
>> Mar  9 00:24:10 memoryana kernel: invalid opcode: 0000 [#1] SMP
>> Mar  9 00:24:10 memoryana kernel: last sysfs file: 
>> /sys/devices/pci0000:00/0000:00:07.0/0000:04:00.1/host3/rport-3:0-2/target3:0:2/3:0:2:0/state
>> Mar  9 00:24:10 memoryana kernel: CPU 1
>> Mar  9 00:24:10 memoryana kernel: Modules linked in: dm_round_robin 
>> dm_multipath qla2xxx
>> Mar  9 00:24:10 memoryana kernel: Pid: 10662, comm: multipath-tools Not 
>> tainted 2.6.32.28-xen0 #4 PowerEdge R610
>> Mar  9 00:24:10 memoryana kernel: RIP: e030:[<ffffffff8100d471>] 
>> [<ffffffff8100d471>] pin_pagetable_pfn+0x31/0x60
>> Mar  9 00:24:10 memoryana kernel: RSP: e02b:ffff8800c3101df8  EFLAGS: 
>> 00010282
>> Mar  9 00:24:10 memoryana kernel: RAX: 00000000ffffffea RBX: 
>> ffff8800cc4c3400 RCX: 0000000000000003
>> Mar  9 00:24:10 memoryana kernel: RDX: 0000000000000000 RSI: 
>> 0000000000000001 RDI: ffff8800c3101df8
>> Mar  9 00:24:10 memoryana kernel: RBP: ffff8800c3135b60 R08: 
>> 00003ffffffff000 R09: ffff880000000000
>> Mar  9 00:24:10 memoryana kernel: R10: 0000000000007ff0 R11: 
>> 0000000000000246 R12: 00000000000cc302
>> Mar  9 00:24:10 memoryana kernel: R13: 0000000000000000 R14: 
>> ffff8800c374cc60 R15: ffff8800c374cc60
>> Mar  9 00:24:10 memoryana kernel: FS:  00007f60add15700(0000) 
>> GS:ffff880028055000(0000) knlGS:0000000000000000
>> Mar  9 00:24:10 memoryana kernel: CS:  e033 DS: 0000 ES: 0000 CR0: 
>> 000000008005003b
>> Mar  9 00:24:10 memoryana kernel: CR2: 00007f60ad841876 CR3: 
>> 00000000cef79000 CR4: 0000000000002660
>> Mar  9 00:24:10 memoryana kernel: DR0: 0000000000000000 DR1: 
>> 0000000000000000 DR2: 0000000000000000
>> Mar  9 00:24:10 memoryana kernel: DR3: 0000000000000000 DR6: 
>> 00000000ffff0ff0 DR7: 0000000000000400
>> Mar  9 00:24:10 memoryana kernel: Process multipath-tools (pid: 10662, 
>> threadinfo ffff8800c3100000, task ffff8800cc01cbc0)
>> Mar  9 00:24:10 memoryana kernel: Stack:
>> Mar  9 00:24:10 memoryana kernel: 0000000000000000 00000000008e8302 
>> ffff8800cc4c3400 ffff8800c3135b60
>> Mar  9 00:24:10 memoryana kernel: <0> 00000000000cc302 ffffffff810b0382 
>> 00007f60ad841876 ffff8800c30b4c10
>> Mar  9 00:24:10 memoryana kernel: <0> 00000000000100e0 0000000000000000 
>> ffff8800c374cc60 ffffffff810b3595
>> Mar  9 00:24:10 memoryana kernel: Call Trace:
>> Mar  9 00:24:10 memoryana kernel: [<ffffffff810b0382>] ? 
>> __pte_alloc+0xf2/0x120
>> Mar  9 00:24:10 memoryana kernel: [<ffffffff810b3595>] ? 
>> handle_mm_fault+0xa45/0xab0
>> Mar  9 00:24:10 memoryana kernel: [<ffffffff8153cfe5>] ? 
>> page_fault+0x25/0x30
>> Mar  9 00:24:10 memoryana kernel: [<ffffffff8153d21a>] ? 
>> error_exit+0x2a/0x60
>> Mar  9 00:24:10 memoryana kernel: [<ffffffff8101481d>] ? 
>> retint_restore_args+0x5/0x6
>> Mar  9 00:24:10 memoryana kernel: [<ffffffff81038631>] ? 
>> do_page_fault+0x121/0x3c0
>> Mar  9 00:24:10 memoryana kernel: [<ffffffff812a2e0d>] ? 
>> __put_user_4+0x1d/0x30
>> Mar  9 00:24:10 memoryana kernel: [<ffffffff8153cfe5>] ? 
>> page_fault+0x25/0x30
>> Mar  9 00:24:10 memoryana kernel: Code: 57 c7 75 00 00 48 89 f0 89 3c 24 
>> 74 27 48 89 44 24 08 48 89 e7 be 01 00 00 00 31 d2 41 ba f0 7f 00 00 e8 
>> d3 be ff ff 85 c0 74 04 <0f> 0b eb fe 48 83 c4 28 c3 48 89 f7 e8 6e f7 
>> ff ff 48 83 f8 ff
>> Mar  9 00:24:10 memoryana kernel: RIP  [<ffffffff8100d471>] 
>> pin_pagetable_pfn+0x31/0x60
>> Mar  9 00:24:10 memoryana kernel: RSP <ffff8800c3101df8>
>> Mar  9 00:24:10 memoryana kernel: ---[ end trace f4eae184c1a9f532 ]---
>> Mar  9 00:24:11 memoryana multipathd: --------shut down-------
>> 
>> -- 
>> Andreas Olsowski
>> Leuphana Universität Lüneburg
>> Rechen- und Medienzentrum
>> Scharnhorststraße 1, C7.015
>> 21335 Lüneburg
>> 
>> Tel: ++49 4131 677 1309
>> 
>> 



>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xensource.com
>> http://lists.xensource.com/xen-devel





-- 
Best regards,
 Sander                            mailto:linux@eikelenboom.it

Re: kernel BUG at arch/x86/xen/mmu.c:1860!

[Teck Choon Giam]

[-- Attachment #1.1: Type: text/plain, Size: 39231 bytes --]

On Sat, Mar 12, 2011 at 3:59 AM, Sander Eikelenboom <linux@eikelenboom.it>wrote:

> Haven't seen this one on my systen. I do use LVM, but i'm only using LVM as
> device mapper, without software raid etc.
> Also using debian lenny/squeeze together with custom compiled xen and
> xen-2.6.32.x pvops kernel
>
> So I would suspect the multipathing / raid part.
>

>From one of my test server, below is grep output:

# grep -A 45 'cut here' /var/log/messages*|grep 'kernel: Process'
/var/log/messages-Mar 10 00:53:00 xen06 kernel: Process mpath_wait (pid:
9513, threadinfo ffff880010f38000, task ffff880010c4c100)
/var/log/messages-Mar 10 02:52:33 xen06 kernel: Process clvmd (pid: 3240,
threadinfo ffff880035a14000, task ffff88003b49c340)
/var/log/messages-Mar 10 02:52:33 xen06 kernel: Process syslogd (pid: 24279,
threadinfo ffff88003bb3e000, task ffff880016f3a1c0)
/var/log/messages-Mar 10 03:40:52 xen06 kernel: Process mpath_wait (pid:
16487, threadinfo ffff88003becc000, task ffff88003bdb8480)
/var/log/messages.2-Feb 23 22:56:56 xen06 kernel: Process mpath_wait (pid:
15222, threadinfo ffff8800265b8000, task ffff88003b940200)
/var/log/messages.2-Feb 24 01:03:17 xen06 kernel: Process dmsetup (pid: 509,
threadinfo ffff88003ced4000, task ffff88003025c4c0)
/var/log/messages.2-Feb 26 23:20:22 xen06 kernel: Process mpath_wait (pid:
10154, threadinfo ffff880025604000, task ffff8800254804c0)
/var/log/messages.4-Feb  8 04:30:10 xen06 kernel: Process mpath_wait (pid:
11646, threadinfo ffff88001e722000, task ffff880025222440)
/var/log/messages.4-Feb  9 16:43:11 xen06 kernel: Process lvremove (pid:
24469, threadinfo ffff88003b930000, task ffff880025ab03c0)
/var/log/messages.4-Feb  9 22:02:09 xen06 kernel: Process grep (pid: 10327,
threadinfo ffff880025056000, task ffff880024814240)
/var/log/messages.4-Feb  9 22:13:37 xen06 kernel: Process dmsetup (pid:
6287, threadinfo ffff88003cf64000, task ffff880025fe8740)
/var/log/messages.4-Feb  9 22:16:55 xen06 kernel: Process clvmd (pid: 3204,
threadinfo ffff88003ccb8000, task ffff88003a436440)

Whereby output related to 'kernel: BUG: unable to handle kernel paging
request at':

# grep -B 20 -A 50 'kernel: BUG: unable to handle kernel paging request at'
/var/log/messages*
/var/log/messages.2-Feb 24 00:59:13 xen06 lvm[2379]: No longer monitoring
snapshot XenGroup-testcrash5--snapshot
/var/log/messages.2-Feb 24 00:59:14 xen06 lvm[2379]: Monitoring snapshot
XenGroup-testcrash1--snapshot
/var/log/messages.2-Feb 24 00:59:14 xen06 kernel: kjournald starting.
Commit interval 5 seconds
/var/log/messages.2-Feb 24 00:59:14 xen06 kernel: EXT3 FS on dm-0, internal
journal
/var/log/messages.2-Feb 24 00:59:14 xen06 kernel: EXT3-fs: mounted
filesystem with ordered data mode.
/var/log/messages.2-Feb 24 00:59:14 xen06 lvm[2379]: No longer monitoring
snapshot XenGroup-testcrash1--snapshot
/var/log/messages.2-Feb 24 00:59:15 xen06 lvm[2379]: Monitoring snapshot
XenGroup-testcrash2--snapshot
/var/log/messages.2-Feb 24 00:59:16 xen06 kernel: kjournald starting.
Commit interval 5 seconds
/var/log/messages.2-Feb 24 00:59:16 xen06 kernel: EXT3 FS on dm-0, internal
journal
/var/log/messages.2-Feb 24 00:59:16 xen06 kernel: EXT3-fs: mounted
filesystem with ordered data mode.
/var/log/messages.2-Feb 24 00:59:16 xen06 lvm[2379]: No longer monitoring
snapshot XenGroup-testcrash2--snapshot
/var/log/messages.2-Feb 24 00:59:17 xen06 lvm[2379]: Monitoring snapshot
XenGroup-testcrash3--snapshot
/var/log/messages.2-Feb 24 00:59:17 xen06 kernel: kjournald starting.
Commit interval 5 seconds
/var/log/messages.2-Feb 24 00:59:17 xen06 kernel: EXT3 FS on dm-0, internal
journal
/var/log/messages.2-Feb 24 00:59:17 xen06 kernel: EXT3-fs: mounted
filesystem with ordered data mode.
/var/log/messages.2-Feb 24 00:59:18 xen06 lvm[2379]: No longer monitoring
snapshot XenGroup-testcrash3--snapshot
/var/log/messages.2-Feb 24 00:59:19 xen06 lvm[2379]: Monitoring snapshot
XenGroup-testcrash4--snapshot
/var/log/messages.2-Feb 24 00:59:19 xen06 kernel: kjournald starting.
Commit interval 5 seconds
/var/log/messages.2-Feb 24 00:59:19 xen06 kernel: EXT3 FS on dm-0, internal
journal
/var/log/messages.2-Feb 24 00:59:19 xen06 kernel: EXT3-fs: mounted
filesystem with ordered data mode.
/var/log/messages.2:Feb 24 00:59:19 xen06 kernel: BUG: unable to handle
kernel paging request at ffff88002561f3f8
/var/log/messages.2-Feb 24 00:59:19 xen06 kernel: IP: [<ffffffff8100e2d4>]
xen_set_pmd+0x16/0x2b
/var/log/messages.2-Feb 24 00:59:19 xen06 kernel: PGD 1002067 PUD 1006067
PMD 1f1067 PTE 801000002561f065
/var/log/messages.2-Feb 24 00:59:19 xen06 kernel: Oops: 0003 [#1] SMP
/var/log/messages.2-Feb 24 00:59:19 xen06 kernel: last sysfs file:
/sys/block/dm-13/dev
/var/log/messages.2-Feb 24 00:59:19 xen06 kernel: CPU 1
/var/log/messages.2-Feb 24 00:59:19 xen06 kernel: Modules linked in: dlm
configfs xt_physdev iptable_filter ip_tables x_tables bridge stp be2iscsi
iscsi_tcp bnx2i cnic uio ipv6 cxgb3i cxgb3 mdio libiscsi_tcp libiscsi
scsi_transport_iscsi dm_multipath scsi_dh video backlight output sbs sbshc
power_meter hwmon battery acpi_memhotplug xen_acpi_memhotplug ac parport_pc
lp parport sg ide_cd_mod cdrom tg3 serio_raw libphy button tpm_tis tpm
tpm_bios pcspkr shpchp i2c_i801 i2c_core iTCO_wdt dm_snapshot dm_zero
dm_mirror dm_region_hash dm_log dm_mod ata_piix libata sd_mod scsi_mod raid1
ext3 jbd uhci_hcd ohci_hcd ehci_hcd [last unloaded: microcode]
/var/log/messages.2-Feb 24 00:59:19 xen06 kernel: Pid: 16520, comm: udevd
Not tainted 2.6.32.29-0.xen.pvops.choon.centos5 #1 PowerEdge 860
/var/log/messages.2-Feb 24 00:59:19 xen06 kernel: RIP:
e030:[<ffffffff8100e2d4>]  [<ffffffff8100e2d4>] xen_set_pmd+0x16/0x2b
/var/log/messages.2-Feb 24 00:59:19 xen06 kernel: RSP:
e02b:ffff880025655b88  EFLAGS: 00010246
/var/log/messages.2-Feb 24 00:59:19 xen06 kernel: RAX: 0000000000000000 RBX:
ffff88002561f3f8 RCX: 00007f9a10c63000
/var/log/messages.2-Feb 24 00:59:19 xen06 kernel: RDX: ffffea0000000000 RSI:
0000000000000000 RDI: ffff88002561f3f8
/var/log/messages.2-Feb 24 00:59:19 xen06 kernel: RBP: ffff880025655b98 R08:
00007f9a10fc0000 R09: ffff88003e64d609
/var/log/messages.2-Feb 24 00:59:19 xen06 kernel: R10: ffff88003cce22c0 R11:
ffffffff813391a0 R12: 0000000000000000
/var/log/messages.2-Feb 24 00:59:19 xen06 lvm[2379]: No longer monitoring
snapshot XenGroup-testcrash4--snapshot
/var/log/messages.2-Feb 24 00:59:19 xen06 kernel: R13: 00007f9a10000000 R14:
ffff88003cc6a7f8 R15: ffff88002550c340
/var/log/messages.2-Feb 24 00:59:19 xen06 kernel: FS:
00007f9a10a41710(0000) GS:ffff88002806c000(0000) knlGS:0000000000000000
/var/log/messages.2-Feb 24 00:59:19 xen06 udevd-event[16460]: run_program:
'/sbin/dmsetup' abnormal exit
/var/log/messages.2-Feb 24 00:59:19 xen06 kernel: CS:  e033 DS: 0000 ES:
0000 CR0: 000000008005003b
/var/log/messages.2-Feb 24 00:59:19 xen06 kernel: CR2: ffff88002561f3f8 CR3:
000000003bb46000 CR4: 0000000000002660
/var/log/messages.2-Feb 24 00:59:19 xen06 kernel: DR0: 0000000000000000 DR1:
0000000000000000 DR2: 0000000000000000
/var/log/messages.2-Feb 24 00:59:19 xen06 kernel: DR3: 0000000000000000 DR6:
00000000ffff0ff0 DR7: 0000000000000400
/var/log/messages.2-Feb 24 00:59:19 xen06 kernel: Process udevd (pid: 16520,
threadinfo ffff880025654000, task ffff88003cce22c0)
/var/log/messages.2-Feb 24 00:59:19 xen06 kernel: Stack:
/var/log/messages.2-Feb 24 00:59:19 xen06 kernel:  ffff88002561f3f8
0000000025690067 ffff880025655c48 ffffffff810a60b0
/var/log/messages.2-Feb 24 00:59:19 xen06 kernel: <0> ffff880025655ac8
000000000000001b 00007f9a10e00000 0000000000000000
/var/log/messages.2-Feb 24 00:59:19 xen06 kernel: <0> 00007f9a10c63000
ffff88002807a5c0 00007f9a10c63000 ffff880025655cb8
/var/log/messages.2-Feb 24 00:59:19 xen06 kernel: Call Trace:
/var/log/messages.2-Feb 24 00:59:19 xen06 kernel:  [<ffffffff810a60b0>]
free_pgd_range+0x3b3/0x401
/var/log/messages.2-Feb 24 00:59:19 xen06 kernel:  [<ffffffff810a63b1>]
free_pgtables+0xb6/0xce
/var/log/messages.2-Feb 24 00:59:19 xen06 kernel:  [<ffffffff810a84a4>]
exit_mmap+0xf0/0x14b
/var/log/messages.2-Feb 24 00:59:19 xen06 kernel:  [<ffffffff81048888>]
mmput+0x46/0xb2
/var/log/messages.2-Feb 24 00:59:19 xen06 kernel:  [<ffffffff810c8cd9>]
flush_old_exec+0x463/0x536
/var/log/messages.2-Feb 24 00:59:19 xen06 kernel:  [<ffffffff810fbf99>]
load_elf_binary+0x365/0x171b
/var/log/messages.2-Feb 24 00:59:19 xen06 kernel:  [<ffffffff810faa30>] ?
load_misc_binary+0x61/0x322
/var/log/messages.2-Feb 24 00:59:19 xen06 kernel:  [<ffffffff810a50dd>] ?
get_user_pages+0x44/0x46
/var/log/messages.2-Feb 24 00:59:19 xen06 kernel:  [<ffffffff810c795e>] ?
put_arg_page+0x9/0xb
/var/log/messages.2-Feb 24 00:59:19 xen06 kernel:  [<ffffffff810c7e0b>]
search_binary_handler+0xde/0x268
/var/log/messages.2-Feb 24 00:59:19 xen06 kernel:  [<ffffffff810c9208>]
do_execve+0x193/0x262
/var/log/messages.2-Feb 24 00:59:19 xen06 kernel:  [<ffffffff810113af>]
sys_execve+0x3e/0x58
/var/log/messages.2-Feb 24 00:59:19 xen06 kernel:  [<ffffffff81012f1a>]
stub_execve+0x6a/0xc0
/var/log/messages.2-Feb 24 00:59:19 xen06 kernel: Code: 8b 3c 25 70 c5 00 00
ff 14 25 68 a2 5c 81 48 83 c4 18 5b c9 c3 55 48 89 e5 41 54 49 89 f4 53 48
89 fb e8 79 e3 ff ff 84 c0 75 05 <4c> 89 23 eb 0b 4c 89 e6 48 89 df e8 69 ff
ff ff 5b 41 5c c9 c3
/var/log/messages.2-Feb 24 00:59:19 xen06 kernel: RIP  [<ffffffff8100e2d4>]
xen_set_pmd+0x16/0x2b
/var/log/messages.2-Feb 24 00:59:19 xen06 kernel:  RSP <ffff880025655b88>
/var/log/messages.2-Feb 24 00:59:19 xen06 kernel: CR2: ffff88002561f3f8
/var/log/messages.2-Feb 24 00:59:19 xen06 kernel: ---[ end trace
5ea31e622470b518 ]---
/var/log/messages.2-Feb 24 00:59:20 xen06 lvm[2379]: Monitoring snapshot
XenGroup-testcrash5--snapshot
/var/log/messages.2-Feb 24 00:59:21 xen06 kernel: kjournald starting.
Commit interval 5 seconds
/var/log/messages.2-Feb 24 00:59:21 xen06 kernel: EXT3 FS on dm-0, internal
journal
/var/log/messages.2-Feb 24 00:59:21 xen06 kernel: EXT3-fs: mounted
filesystem with ordered data mode.
/var/log/messages.2-Feb 24 00:59:21 xen06 lvm[2379]: No longer monitoring
snapshot XenGroup-testcrash5--snapshot
--
/var/log/messages.4-Feb  8 03:58:30 xen06 lvm[3961]: Monitoring snapshot
XenGroup-testcrash4--snapshot
/var/log/messages.4-Feb  8 03:58:30 xen06 kernel: kjournald starting.
Commit interval 5 seconds
/var/log/messages.4-Feb  8 03:58:30 xen06 kernel: EXT3 FS on dm-11, internal
journal
/var/log/messages.4-Feb  8 03:58:30 xen06 kernel: EXT3-fs: mounted
filesystem with ordered data mode.
/var/log/messages.4-Feb  8 03:58:30 xen06 lvm[3961]: No longer monitoring
snapshot XenGroup-testcrash4--snapshot
/var/log/messages.4-Feb  8 03:58:31 xen06 lvm[3961]: Monitoring snapshot
XenGroup-testcrash5--snapshot
/var/log/messages.4-Feb  8 03:58:31 xen06 kernel: kjournald starting.
Commit interval 5 seconds
/var/log/messages.4-Feb  8 03:58:31 xen06 kernel: EXT3 FS on dm-11, internal
journal
/var/log/messages.4-Feb  8 03:58:31 xen06 kernel: EXT3-fs: mounted
filesystem with ordered data mode.
/var/log/messages.4-Feb  8 03:58:32 xen06 lvm[3961]: No longer monitoring
snapshot XenGroup-testcrash5--snapshot
/var/log/messages.4-Feb  8 03:58:33 xen06 lvm[3961]: Monitoring snapshot
XenGroup-testcrash1--snapshot
/var/log/messages.4-Feb  8 03:58:33 xen06 kernel: kjournald starting.
Commit interval 5 seconds
/var/log/messages.4-Feb  8 03:58:33 xen06 kernel: EXT3 FS on dm-11, internal
journal
/var/log/messages.4-Feb  8 03:58:33 xen06 kernel: EXT3-fs: mounted
filesystem with ordered data mode.
/var/log/messages.4-Feb  8 03:58:33 xen06 lvm[3961]: No longer monitoring
snapshot XenGroup-testcrash1--snapshot
/var/log/messages.4-Feb  8 03:58:34 xen06 lvm[3961]: Monitoring snapshot
XenGroup-testcrash2--snapshot
/var/log/messages.4-Feb  8 03:58:34 xen06 kernel: kjournald starting.
Commit interval 5 seconds
/var/log/messages.4-Feb  8 03:58:34 xen06 kernel: EXT3 FS on dm-11, internal
journal
/var/log/messages.4-Feb  8 03:58:34 xen06 kernel: EXT3-fs: mounted
filesystem with ordered data mode.
/var/log/messages.4-Feb  8 03:58:34 xen06 lvm[3961]: No longer monitoring
snapshot XenGroup-testcrash2--snapshot
/var/log/messages.4:Feb  8 03:58:34 xen06 kernel: BUG: unable to handle
kernel paging request at ffff880030081010
/var/log/messages.4-Feb  8 03:58:34 xen06 kernel: IP: [<ffffffff8100e2d4>]
xen_set_pmd+0x16/0x2b
/var/log/messages.4-Feb  8 03:58:34 xen06 kernel: PGD 1002067 PUD 1006067
PMD 246067 PTE 8010000030081065
/var/log/messages.4-Feb  8 03:58:34 xen06 kernel: Oops: 0003 [#1] SMP
/var/log/messages.4-Feb  8 03:58:34 xen06 kernel: last sysfs file:
/sys/devices/system/cpu/cpu3/cache/index2/shared_cpu_map
/var/log/messages.4-Feb  8 03:58:34 xen06 kernel: CPU 3
/var/log/messages.4-Feb  8 03:58:34 xen06 kernel: Modules linked in:
ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state
nf_conntrack ipt_REJECT xt_tcpudp gfs2 dlm configfs xt_physdev
iptable_filter ip_tables x_tables bridge stp be2iscsi iscsi_tcp bnx2i cnic
uio ipv6 cxgb3i cxgb3 mdio libiscsi_tcp libiscsi scsi_transport_iscsi
dm_multipath scsi_dh video backlight output sbs sbshc power_meter hwmon
battery acpi_memhotplug xen_acpi_memhotplug ac parport_pc lp parport sg
ide_cd_mod cdrom serio_raw tg3 button libphy tpm_tis tpm tpm_bios i2c_i801
pcspkr i2c_core iTCO_wdt shpchp dm_snapshot dm_zero dm_mirror dm_region_hash
dm_log dm_mod ata_piix libata sd_mod scsi_mod raid1 ext3 jbd uhci_hcd
ohci_hcd ehci_hcd [last unloaded: microcode]
/var/log/messages.4-Feb  8 03:58:34 xen06 kernel: Pid: 32051, comm:
mpath_wait Not tainted 2.6.32.28-0.xen.pvops.choon.centos5 #1 PowerEdge 860
/var/log/messages.4-Feb  8 03:58:34 xen06 kernel: RIP:
e030:[<ffffffff8100e2d4>]  [<ffffffff8100e2d4>] xen_set_pmd+0x16/0x2b
/var/log/messages.4-Feb  8 03:58:34 xen06 kernel: RSP:
e02b:ffff880030c19b88  EFLAGS: 00010246
/var/log/messages.4-Feb  8 03:58:34 xen06 kernel: RAX: 0000000000000000 RBX:
ffff880030081010 RCX: 00000000008c3000
/var/log/messages.4-Feb  8 03:58:34 xen06 kernel: RDX: ffffea0000000000 RSI:
0000000000000000 RDI: ffff880030081010
/var/log/messages.4-Feb  8 03:58:34 xen06 kernel: RBP: ffff880030c19b98 R08:
0000000002150000 R09: ffff88003e6f3e89
/var/log/messages.4-Feb  8 03:58:34 xen06 kernel: R10: ffff880030e78040 R11:
ffffffff813381a0 R12: 0000000000000000
/var/log/messages.4-Feb  8 03:58:34 xen06 kernel: R13: 0000000000600000 R14:
ffff880036dc5000 R15: ffff880030994000
/var/log/messages.4-Feb  8 03:58:34 xen06 kernel: FS:
00007f147d9606e0(0000) GS:ffff8800280a6000(0000) knlGS:0000000000000000
/var/log/messages.4-Feb  8 03:58:34 xen06 kernel: CS:  e033 DS: 0000 ES:
0000 CR0: 000000008005003b
/var/log/messages.4-Feb  8 03:58:34 xen06 kernel: CR2: ffff880030081010 CR3:
000000003ba55000 CR4: 0000000000002660
/var/log/messages.4-Feb  8 03:58:34 xen06 kernel: DR0: 0000000000000000 DR1:
0000000000000000 DR2: 0000000000000000
/var/log/messages.4-Feb  8 03:58:34 xen06 kernel: DR3: 0000000000000000 DR6:
00000000ffff0ff0 DR7: 0000000000000400
/var/log/messages.4-Feb  8 03:58:34 xen06 kernel: Process mpath_wait (pid:
32051, threadinfo ffff880030c18000, task ffff88003c93c280)
/var/log/messages.4-Feb  8 03:58:34 xen06 kernel: Stack:
/var/log/messages.4-Feb  8 03:58:34 xen06 kernel:  ffff880030081010
000000003b7e4067 ffff880030c19c48 ffffffff810a5e32
/var/log/messages.4-Feb  8 03:58:34 xen06 kernel: <0> ffff880030c19ac8
000000000000001f 0000000002000000 0000000000000000
/var/log/messages.4-Feb  8 03:58:34 xen06 kernel: <0> 00000000008c3000
ffff8800280b45c0 00000000008c3000 ffff880030c19cb8
/var/log/messages.4-Feb  8 03:58:34 xen06 kernel: Call Trace:
/var/log/messages.4-Feb  8 03:58:34 xen06 kernel:  [<ffffffff810a5e32>]
free_pgd_range+0x3b3/0x401
/var/log/messages.4-Feb  8 03:58:34 xen06 kernel:  [<ffffffff810a6133>]
free_pgtables+0xb6/0xce
/var/log/messages.4-Feb  8 03:58:34 xen06 kernel:  [<ffffffff810a8224>]
exit_mmap+0xf0/0x14b
/var/log/messages.4-Feb  8 03:58:34 xen06 kernel:  [<ffffffff81048648>]
mmput+0x46/0xb2
/var/log/messages.4-Feb  8 03:58:34 xen06 kernel:  [<ffffffff810c8a56>]
flush_old_exec+0x460/0x533
/var/log/messages.4-Feb  8 03:58:34 xen06 kernel:  [<ffffffff810fbd15>]
load_elf_binary+0x365/0x171c
/var/log/messages.4-Feb  8 03:58:34 xen06 kernel:  [<ffffffff810fa7ac>] ?
load_misc_binary+0x61/0x322
/var/log/messages.4-Feb  8 03:58:34 xen06 kernel:  [<ffffffff810a4e5f>] ?
get_user_pages+0x44/0x46
/var/log/messages.4-Feb  8 03:58:34 xen06 kernel:  [<ffffffff810c76de>] ?
put_arg_page+0x9/0xb
/var/log/messages.4-Feb  8 03:58:34 xen06 kernel:  [<ffffffff810c7b8b>]
search_binary_handler+0xde/0x268
/var/log/messages.4-Feb  8 03:58:34 xen06 kernel:  [<ffffffff810c8f85>]
do_execve+0x193/0x262
/var/log/messages.4-Feb  8 03:58:34 xen06 kernel:  [<ffffffff810113af>]
sys_execve+0x3e/0x58
/var/log/messages.4-Feb  8 03:58:34 xen06 kernel:  [<ffffffff81012f1a>]
stub_execve+0x6a/0xc0
/var/log/messages.4-Feb  8 03:58:34 xen06 kernel: Code: 8b 3c 25 70 c5 00 00
ff 14 25 68 82 5c 81 48 83 c4 18 5b c9 c3 55 48 89 e5 41 54 49 89 f4 53 48
89 fb e8 79 e3 ff ff 84 c0 75 05 <4c> 89 23 eb 0b 4c 89 e6 48 89 df e8 69 ff
ff ff 5b 41 5c c9 c3
/var/log/messages.4-Feb  8 03:58:34 xen06 kernel: RIP  [<ffffffff8100e2d4>]
xen_set_pmd+0x16/0x2b
/var/log/messages.4-Feb  8 03:58:34 xen06 kernel:  RSP <ffff880030c19b88>
/var/log/messages.4-Feb  8 03:58:34 xen06 kernel: CR2: ffff880030081010
/var/log/messages.4-Feb  8 03:58:34 xen06 kernel: ---[ end trace
f2d7fcc13c4cdf6a ]---
/var/log/messages.4:Feb  8 03:58:34 xen06 kernel: BUG: unable to handle
kernel paging request at ffff8800313e1b00
/var/log/messages.4-Feb  8 03:58:34 xen06 kernel: IP: [<ffffffff810377f2>]
ptep_set_access_flags+0x27/0x4a
/var/log/messages.4-Feb  8 03:58:35 xen06 kernel: PGD 1002067 PUD 1006067
PMD 24f067 PTE 80100000313e1065
/var/log/messages.4-Feb  8 03:58:35 xen06 kernel: Oops: 0003 [#2] SMP
/var/log/messages.4-Feb  8 03:58:35 xen06 kernel: last sysfs file:
/sys/devices/system/cpu/cpu3/cache/index2/shared_cpu_map
/var/log/messages.4-Feb  8 03:58:35 xen06 kernel: CPU 1
/var/log/messages.4-Feb  8 03:58:35 xen06 kernel: Modules linked in:
ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state
nf_conntrack ipt_REJECT xt_tcpudp gfs2 dlm configfs xt_physdev
iptable_filter ip_tables x_tables bridge stp be2iscsi iscsi_tcp bnx2i cnic
uio ipv6 cxgb3i cxgb3 mdio libiscsi_tcp libiscsi scsi_transport_iscsi
dm_multipath scsi_dh video backlight output sbs sbshc power_meter hwmon
battery acpi_memhotplug xen_acpi_memhotplug ac parport_pc lp parport sg
ide_cd_mod cdrom serio_raw tg3 button libphy tpm_tis tpm tpm_bios i2c_i801
pcspkr i2c_core iTCO_wdt shpchp dm_snapshot dm_zero dm_mirror dm_region_hash
dm_log dm_mod ata_piix libata sd_mod scsi_mod raid1 ext3 jbd uhci_hcd
ohci_hcd ehci_hcd [last unloaded: microcode]
/var/log/messages.4-Feb  8 03:58:35 xen06 kernel: Pid: 32053, comm:
mpath_wait Tainted: G      D    2.6.32.28-0.xen.pvops.choon.centos5 #1
PowerEdge 860
/var/log/messages.4-Feb  8 03:58:35 xen06 kernel: RIP:
e030:[<ffffffff810377f2>]  [<ffffffff810377f2>]
ptep_set_access_flags+0x27/0x4a
/var/log/messages.4-Feb  8 03:58:35 xen06 kernel: RSP:
e02b:ffff8800307c7cb8  EFLAGS: 00210202
/var/log/messages.4-Feb  8 03:58:35 xen06 kernel: RAX: ffff8800304b4870 RBX:
00007f147d960770 RCX: 80000001f6d6f167
/var/log/messages.4-Feb  8 03:58:35 xen06 kernel: RDX: ffff8800313e1b00 RSI:
00007f147d960770 RDI: ffff8800304b4870
/var/log/messages.4-Feb  8 03:58:35 xen06 kernel: RBP: ffff8800307c7cd8 R08:
0000000000000001 R09: ffffea0000ac5948
/var/log/messages.4-Feb  8 03:58:35 xen06 kernel: R10: 3138666666666666 R11:
000000000002496a R12: 0000000000000001
/var/log/messages.4-Feb  8 03:58:35 xen06 kernel: R13: ffff8800304b4870 R14:
0000000000000008 R15: 0000000000000000
/var/log/messages.4-Feb  8 03:58:35 xen06 kernel: FS:
00007f147d9606e0(0000) GS:ffff88002806c000(0000) knlGS:0000000000000000
/var/log/messages.4-Feb  8 03:58:35 xen06 kernel: CS:  e033 DS: 0000 ES:
0000 CR0: 000000008005003b
/var/log/messages.4-Feb  8 03:58:35 xen06 kernel: CR2: ffff8800313e1b00 CR3:
0000000031261000 CR4: 0000000000002660
/var/log/messages.4-Feb  8 03:58:35 xen06 kernel: DR0: 0000000000000000 DR1:
0000000000000000 DR2: 0000000000000000
/var/log/messages.4-Feb  8 03:58:35 xen06 kernel: DR3: 0000000000000000 DR6:
00000000ffff0ff0 DR7: 0000000000000400
/var/log/messages.4-Feb  8 03:58:35 xen06 kernel: Process mpath_wait (pid:
32053, threadinfo ffff8800307c6000, task ffff880030ece200)
/var/log/messages.4-Feb  8 03:58:35 xen06 kernel: Stack:
/var/log/messages.4-Feb  8 03:58:35 xen06 kernel:  80000001f6d6f145
0000000000000000 ffffea0000ac5948 ffffea0000800f30
/var/log/messages.4-Feb  8 03:58:35 xen06 kernel: <0> ffff8800307c7d88
ffffffff810a2a19 0000000000000001 ffff8800307c7de8
/var/log/messages.4-Feb  8 03:58:35 xen06 kernel: <0> ffffffff8100f0b1
ffff8800307c7d98 ffff8800305a8f60 ffff8800313e1b00
/var/log/messages.4-Feb  8 03:58:35 xen06 kernel: Call Trace:
/var/log/messages.4-Feb  8 03:58:35 xen06 kernel:  [<ffffffff810a2a19>]
do_wp_page+0x2fd/0x717
/var/log/messages.4-Feb  8 03:58:35 xen06 kernel:  [<ffffffff8100f0b1>] ?
xen_force_evtchn_callback+0xd/0xf
/var/log/messages.4-Feb  8 03:58:35 xen06 kernel:  [<ffffffff8100cde0>] ?
xen_pte_val+0x64/0x6e
/var/log/messages.4-Feb  8 03:58:35 xen06 kernel:  [<ffffffff8100c3e9>] ?
__raw_callee_save_xen_pte_val+0x15/0x23
/var/log/messages.4-Feb  8 03:58:35 xen06 kernel:  [<ffffffff810a4933>]
handle_mm_fault+0x84a/0x8b9
/var/log/messages.4-Feb  8 03:58:35 xen06 kernel:  [<ffffffff8131be2d>]
do_page_fault+0x252/0x2e2
/var/log/messages.4-Feb  8 03:58:35 xen06 kernel:  [<ffffffff8100f7f2>] ?
check_events+0x12/0x20
/var/log/messages.4-Feb  8 03:58:35 xen06 kernel:  [<ffffffff81319db5>]
page_fault+0x25/0x30
/var/log/messages.4-Feb  8 03:58:35 xen06 kernel:  [<ffffffff8116dc9d>] ?
__put_user_4+0x1d/0x30
/var/log/messages.4-Feb  8 03:58:35 xen06 kernel:  [<ffffffff81047d2c>] ?
schedule_tail+0x98/0x9d
/var/log/messages.4-Feb  8 03:58:35 xen06 kernel:  [<ffffffff810129c3>]
ret_from_fork+0x13/0x80
/var/log/messages.4-Feb  8 03:58:35 xen06 kernel: Code: 41 5d c9 c3 55 48 89
e5 41 55 49 89 fd 41 54 45 31 e4 53 48 89 f3 48 83 ec 08 48 39 0a 41 0f 95
c4 45 85 e4 74 1d 45 85 c0 74 18 <48> 89 0a 48 8b 3f 0f 1f 80 00 00 00 00 48
89 de 4c 89 ef e8 37
/var/log/messages.4-Feb  8 03:58:35 xen06 kernel: RIP  [<ffffffff810377f2>]
ptep_set_access_flags+0x27/0x4a
/var/log/messages.4-Feb  8 03:58:35 xen06 kernel:  RSP <ffff8800307c7cb8>
/var/log/messages.4-Feb  8 03:58:35 xen06 kernel: CR2: ffff8800313e1b00
/var/log/messages.4-Feb  8 03:58:35 xen06 kernel: ---[ end trace
f2d7fcc13c4cdf6b ]---
/var/log/messages.4-Feb  8 04:26:35 xen06 syslogd 1.4.1: restart.
/var/log/messages.4-Feb  8 04:26:35 xen06 kernel: klogd 1.4.1, log source =
/proc/kmsg started.
/var/log/messages.4-Feb  8 04:26:35 xen06 kernel: Linux version
2.6.32.28-0.xen.pvops.choon.centos5 (mockbuild@builder5.choon.net) (gcc
version 4.1.2 20080704 (Red Hat 4.1.2-48)) #1 SMP Tue Feb 8 03:26:17 SGT
2011
/var/log/messages.4-Feb  8 04:26:35 xen06 kernel: Command line: ro
root=/dev/md1 panic=5 panic_timeout=5
/var/log/messages.4-Feb  8 04:26:35 xen06 kernel: KERNEL supported cpus:
/var/log/messages.4-Feb  8 04:26:35 xen06 kernel:   Intel GenuineIntel
/var/log/messages.4-Feb  8 04:26:35 xen06 kernel:   AMD AuthenticAMD
/var/log/messages.4-Feb  8 04:26:35 xen06 kernel:   Centaur CentaurHauls
/var/log/messages.4-Feb  8 04:26:35 xen06 kernel: released 0 pages of unused
memory
--
/var/log/messages.4-Feb  9 07:44:11 xen06 lvm[3802]: Monitoring snapshot
XenGroup-testcrash2--snapshot
/var/log/messages.4-Feb  9 07:44:11 xen06 kernel: kjournald starting.
Commit interval 5 seconds
/var/log/messages.4-Feb  9 07:44:11 xen06 kernel: EXT3 FS on dm-11, internal
journal
/var/log/messages.4-Feb  9 07:44:11 xen06 kernel: EXT3-fs: mounted
filesystem with ordered data mode.
/var/log/messages.4-Feb  9 07:44:11 xen06 lvm[3802]: No longer monitoring
snapshot XenGroup-testcrash2--snapshot
/var/log/messages.4-Feb  9 07:44:12 xen06 lvm[3802]: Monitoring snapshot
XenGroup-testcrash3--snapshot
/var/log/messages.4-Feb  9 07:44:12 xen06 kernel: kjournald starting.
Commit interval 5 seconds
/var/log/messages.4-Feb  9 07:44:12 xen06 kernel: EXT3 FS on dm-11, internal
journal
/var/log/messages.4-Feb  9 07:44:12 xen06 kernel: EXT3-fs: mounted
filesystem with ordered data mode.
/var/log/messages.4-Feb  9 07:44:12 xen06 lvm[3802]: No longer monitoring
snapshot XenGroup-testcrash3--snapshot
/var/log/messages.4-Feb  9 07:44:13 xen06 lvm[3802]: Monitoring snapshot
XenGroup-testcrash4--snapshot
/var/log/messages.4-Feb  9 07:44:13 xen06 kernel: kjournald starting.
Commit interval 5 seconds
/var/log/messages.4-Feb  9 07:44:13 xen06 kernel: EXT3 FS on dm-11, internal
journal
/var/log/messages.4-Feb  9 07:44:13 xen06 kernel: EXT3-fs: mounted
filesystem with ordered data mode.
/var/log/messages.4-Feb  9 07:44:13 xen06 lvm[3802]: No longer monitoring
snapshot XenGroup-testcrash4--snapshot
/var/log/messages.4-Feb  9 07:44:14 xen06 lvm[3802]: Monitoring snapshot
XenGroup-testcrash5--snapshot
/var/log/messages.4-Feb  9 07:44:14 xen06 kernel: kjournald starting.
Commit interval 5 seconds
/var/log/messages.4-Feb  9 07:44:15 xen06 kernel: EXT3 FS on dm-11, internal
journal
/var/log/messages.4-Feb  9 07:44:15 xen06 kernel: EXT3-fs: mounted
filesystem with ordered data mode.
/var/log/messages.4-Feb  9 07:44:15 xen06 lvm[3802]: No longer monitoring
snapshot XenGroup-testcrash5--snapshot
/var/log/messages.4:Feb  9 07:53:59 xen06 kernel: BUG: unable to handle
kernel paging request at ffff88003cc0c010
/var/log/messages.4-Feb  9 07:53:59 xen06 kernel: IP: [<ffffffff8100e2d4>]
xen_set_pmd+0x16/0x2b
/var/log/messages.4-Feb  9 07:53:59 xen06 kernel: PGD 1002067 PUD 1006067
PMD 2ac067 PTE 801000003cc0c065
/var/log/messages.4-Feb  9 07:53:59 xen06 kernel: Oops: 0003 [#1] SMP
/var/log/messages.4-Feb  9 07:53:59 xen06 kernel: last sysfs file:
/sys/devices/system/cpu/cpu3/cache/index2/shared_cpu_map
/var/log/messages.4-Feb  9 07:53:59 xen06 kernel: CPU 0
/var/log/messages.4-Feb  9 07:53:59 xen06 kernel: Modules linked in: dlm
configfs xt_physdev iptable_filter ip_tables x_tables bridge stp be2iscsi
iscsi_tcp bnx2i cnic uio ipv6 cxgb3i cxgb3 mdio libiscsi_tcp libiscsi
scsi_transport_iscsi dm_multipath scsi_dh video backlight output sbs sbshc
power_meter hwmon battery acpi_memhotplug xen_acpi_memhotplug ac parport_pc
lp parport sg ide_cd_mod cdrom serio_raw tg3 libphy button tpm_tis tpm
tpm_bios pcspkr i2c_i801 shpchp i2c_core iTCO_wdt dm_snapshot dm_zero
dm_mirror dm_region_hash dm_log dm_mod ata_piix libata sd_mod scsi_mod raid1
ext3 jbd uhci_hcd ohci_hcd ehci_hcd [last unloaded: microcode]
/var/log/messages.4-Feb  9 07:53:59 xen06 kernel: Pid: 11005, comm:
mpath_wait Not tainted 2.6.32.28-0.xen.pvops.choon.centos5 #1 PowerEdge 860
/var/log/messages.4-Feb  9 07:53:59 xen06 kernel: RIP:
e030:[<ffffffff8100e2d4>]  [<ffffffff8100e2d4>] xen_set_pmd+0x16/0x2b
/var/log/messages.4-Feb  9 07:53:59 xen06 kernel: RSP:
e02b:ffff88003b7d3b88  EFLAGS: 00010246
/var/log/messages.4-Feb  9 07:53:59 xen06 kernel: RAX: 0000000000000000 RBX:
ffff88003cc0c010 RCX: 00000000008c3000
/var/log/messages.4-Feb  9 07:53:59 xen06 kernel: RDX: ffffea0000000000 RSI:
0000000000000000 RDI: ffff88003cc0c010
/var/log/messages.4-Feb  9 07:53:59 xen06 kernel: RBP: ffff88003b7d3b98 R08:
0000000001244000 R09: ffff88003f009f09
/var/log/messages.4-Feb  9 07:53:59 xen06 kernel: R10: ffff8800269e3ac0 R11:
ffffffff813381a0 R12: 0000000000000000
/var/log/messages.4-Feb  9 07:53:59 xen06 kernel: R13: 0000000000600000 R14:
ffff8800269ca000 R15: ffff8800268bb000
/var/log/messages.4-Feb  9 07:53:59 xen06 kernel: FS:
00007fa7f8de76e0(0000) GS:ffff88002804f000(0000) knlGS:0000000000000000
/var/log/messages.4-Feb  9 07:53:59 xen06 kernel: CS:  e033 DS: 0000 ES:
0000 CR0: 000000008005003b
/var/log/messages.4-Feb  9 07:53:59 xen06 kernel: CR2: ffff88003cc0c010 CR3:
00000000306b0000 CR4: 0000000000002660
/var/log/messages.4-Feb  9 07:53:59 xen06 kernel: DR0: 0000000000000000 DR1:
0000000000000000 DR2: 0000000000000000
/var/log/messages.4-Feb  9 07:53:59 xen06 kernel: DR3: 0000000000000000 DR6:
00000000ffff0ff0 DR7: 0000000000000400
/var/log/messages.4-Feb  9 07:53:59 xen06 kernel: Process mpath_wait (pid:
11005, threadinfo ffff88003b7d2000, task ffff8800307aa480)
/var/log/messages.4-Feb  9 07:53:59 xen06 kernel: Stack:
/var/log/messages.4-Feb  9 07:53:59 xen06 kernel:  ffff88003cc0c010
000000003cfee067 ffff88003b7d3c48 ffffffff810a5e32
/var/log/messages.4-Feb  9 07:53:59 xen06 kernel: <0> 000000000000e02b
0000000000000060 0000000001200000 0000000000000000
/var/log/messages.4-Feb  9 07:53:59 xen06 kernel: <0> 00000000008c3000
ffff88002805d5c0 00000000008c3000 ffff88003b7d3cb8
/var/log/messages.4-Feb  9 07:53:59 xen06 kernel: Call Trace:
/var/log/messages.4-Feb  9 07:53:59 xen06 kernel:  [<ffffffff810a5e32>]
free_pgd_range+0x3b3/0x401
/var/log/messages.4-Feb  9 07:53:59 xen06 kernel:  [<ffffffff810a6133>]
free_pgtables+0xb6/0xce
/var/log/messages.4-Feb  9 07:53:59 xen06 kernel:  [<ffffffff810a8224>]
exit_mmap+0xf0/0x14b
/var/log/messages.4-Feb  9 07:53:59 xen06 kernel:  [<ffffffff81048648>]
mmput+0x46/0xb2
/var/log/messages.4-Feb  9 07:53:59 xen06 kernel:  [<ffffffff810c8a56>]
flush_old_exec+0x460/0x533
/var/log/messages.4-Feb  9 07:53:59 xen06 kernel:  [<ffffffff810fbd15>]
load_elf_binary+0x365/0x171c
/var/log/messages.4-Feb  9 07:53:59 xen06 kernel:  [<ffffffff810fa7ac>] ?
load_misc_binary+0x61/0x322
/var/log/messages.4-Feb  9 07:53:59 xen06 kernel:  [<ffffffff810a4e5f>] ?
get_user_pages+0x44/0x46
/var/log/messages.4-Feb  9 07:53:59 xen06 kernel:  [<ffffffff810c76de>] ?
put_arg_page+0x9/0xb
/var/log/messages.4-Feb  9 07:53:59 xen06 kernel:  [<ffffffff810c7b8b>]
search_binary_handler+0xde/0x268
/var/log/messages.4-Feb  9 07:53:59 xen06 kernel:  [<ffffffff810c8f85>]
do_execve+0x193/0x262
/var/log/messages.4-Feb  9 07:53:59 xen06 kernel:  [<ffffffff810113af>]
sys_execve+0x3e/0x58
/var/log/messages.4-Feb  9 07:53:59 xen06 kernel:  [<ffffffff81012f1a>]
stub_execve+0x6a/0xc0
/var/log/messages.4-Feb  9 07:53:59 xen06 kernel: Code: 8b 3c 25 70 c5 00 00
ff 14 25 68 82 5c 81 48 83 c4 18 5b c9 c3 55 48 89 e5 41 54 49 89 f4 53 48
89 fb e8 79 e3 ff ff 84 c0 75 05 <4c> 89 23 eb 0b 4c 89 e6 48 89 df e8 69 ff
ff ff 5b 41 5c c9 c3
/var/log/messages.4-Feb  9 07:53:59 xen06 kernel: RIP  [<ffffffff8100e2d4>]
xen_set_pmd+0x16/0x2b
/var/log/messages.4-Feb  9 07:53:59 xen06 kernel:  RSP <ffff88003b7d3b88>
/var/log/messages.4-Feb  9 07:53:59 xen06 kernel: CR2: ffff88003cc0c010
/var/log/messages.4-Feb  9 07:53:59 xen06 kernel: ---[ end trace
b5fffc59f6c8f974 ]---
/var/log/messages.4-Feb  9 16:31:21 xen06 lvm[3802]: Monitoring snapshot
XenGroup-testcrash1--snapshot
/var/log/messages.4-Feb  9 16:31:21 xen06 kernel: kjournald starting.
Commit interval 5 seconds
/var/log/messages.4-Feb  9 16:31:21 xen06 kernel: EXT3 FS on dm-11, internal
journal
/var/log/messages.4-Feb  9 16:31:21 xen06 kernel: EXT3-fs: mounted
filesystem with ordered data mode.
/var/log/messages.4-Feb  9 16:31:21 xen06 lvm[3802]: No longer monitoring
snapshot XenGroup-testcrash1--snapshot
/var/log/messages.4-Feb  9 16:31:22 xen06 lvm[3802]: Monitoring snapshot
XenGroup-testcrash2--snapshot
/var/log/messages.4-Feb  9 16:31:22 xen06 kernel: kjournald starting.
Commit interval 5 seconds
--
/var/log/messages.4-Feb  9 16:35:34 xen06 lvm[3802]: Monitoring snapshot
XenGroup-testcrash1--snapshot
/var/log/messages.4-Feb  9 16:35:34 xen06 kernel: kjournald starting.
Commit interval 5 seconds
/var/log/messages.4-Feb  9 16:35:34 xen06 kernel: EXT3 FS on dm-11, internal
journal
/var/log/messages.4-Feb  9 16:35:34 xen06 kernel: EXT3-fs: mounted
filesystem with ordered data mode.
/var/log/messages.4-Feb  9 16:35:35 xen06 lvm[3802]: No longer monitoring
snapshot XenGroup-testcrash1--snapshot
/var/log/messages.4-Feb  9 16:35:36 xen06 lvm[3802]: Monitoring snapshot
XenGroup-testcrash2--snapshot
/var/log/messages.4-Feb  9 16:35:36 xen06 kernel: kjournald starting.
Commit interval 5 seconds
/var/log/messages.4-Feb  9 16:35:36 xen06 kernel: EXT3 FS on dm-11, internal
journal
/var/log/messages.4-Feb  9 16:35:36 xen06 kernel: EXT3-fs: mounted
filesystem with ordered data mode.
/var/log/messages.4-Feb  9 16:35:36 xen06 lvm[3802]: No longer monitoring
snapshot XenGroup-testcrash2--snapshot
/var/log/messages.4-Feb  9 16:35:37 xen06 lvm[3802]: Monitoring snapshot
XenGroup-testcrash3--snapshot
/var/log/messages.4-Feb  9 16:35:37 xen06 kernel: kjournald starting.
Commit interval 5 seconds
/var/log/messages.4-Feb  9 16:35:37 xen06 kernel: EXT3 FS on dm-11, internal
journal
/var/log/messages.4-Feb  9 16:35:37 xen06 kernel: EXT3-fs: mounted
filesystem with ordered data mode.
/var/log/messages.4-Feb  9 16:35:37 xen06 lvm[3802]: No longer monitoring
snapshot XenGroup-testcrash3--snapshot
/var/log/messages.4-Feb  9 16:35:38 xen06 lvm[3802]: Monitoring snapshot
XenGroup-testcrash4--snapshot
/var/log/messages.4-Feb  9 16:35:38 xen06 kernel: kjournald starting.
Commit interval 5 seconds
/var/log/messages.4-Feb  9 16:35:39 xen06 kernel: EXT3 FS on dm-11, internal
journal
/var/log/messages.4-Feb  9 16:35:39 xen06 kernel: EXT3-fs: mounted
filesystem with ordered data mode.
/var/log/messages.4-Feb  9 16:35:39 xen06 lvm[3802]: No longer monitoring
snapshot XenGroup-testcrash4--snapshot
/var/log/messages.4:Feb  9 16:35:39 xen06 kernel: BUG: unable to handle
kernel paging request at ffff880026b00418
/var/log/messages.4-Feb  9 16:35:39 xen06 kernel: IP: [<ffffffff8100e2d4>]
xen_set_pmd+0x16/0x2b
/var/log/messages.4-Feb  9 16:35:39 xen06 kernel: PGD 1002067 PUD 1006067
PMD 1fb067 PTE 8010000026b00065
/var/log/messages.4-Feb  9 16:35:39 xen06 kernel: Oops: 0003 [#2] SMP
/var/log/messages.4-Feb  9 16:35:39 xen06 kernel: last sysfs file:
/sys/devices/system/cpu/cpu3/cache/index2/shared_cpu_map
/var/log/messages.4-Feb  9 16:35:39 xen06 kernel: CPU 2
/var/log/messages.4-Feb  9 16:35:39 xen06 kernel: Modules linked in: dlm
configfs xt_physdev iptable_filter ip_tables x_tables bridge stp be2iscsi
iscsi_tcp bnx2i cnic uio ipv6 cxgb3i cxgb3 mdio libiscsi_tcp libiscsi
scsi_transport_iscsi dm_multipath scsi_dh video backlight output sbs sbshc
power_meter hwmon battery acpi_memhotplug xen_acpi_memhotplug ac parport_pc
lp parport sg ide_cd_mod cdrom serio_raw tg3 libphy button tpm_tis tpm
tpm_bios pcspkr i2c_i801 shpchp i2c_core iTCO_wdt dm_snapshot dm_zero
dm_mirror dm_region_hash dm_log dm_mod ata_piix libata sd_mod scsi_mod raid1
ext3 jbd uhci_hcd ohci_hcd ehci_hcd [last unloaded: microcode]
/var/log/messages.4-Feb  9 16:35:39 xen06 kernel: Pid: 25953, comm: udevd
Tainted: G      D    2.6.32.28-0.xen.pvops.choon.centos5 #1 PowerEdge 860
/var/log/messages.4-Feb  9 16:35:39 xen06 kernel: RIP:
e030:[<ffffffff8100e2d4>]  [<ffffffff8100e2d4>] xen_set_pmd+0x16/0x2b
/var/log/messages.4-Feb  9 16:35:39 xen06 kernel: RSP:
e02b:ffff88002682fb88  EFLAGS: 00010246
/var/log/messages.4-Feb  9 16:35:39 xen06 kernel: RAX: 0000000000000000 RBX:
ffff880026b00418 RCX: 00007f71114ac000
/var/log/messages.4-Feb  9 16:35:39 xen06 kernel: RDX: ffffea0000000000 RSI:
0000000000000000 RDI: ffff880026b00418
/var/log/messages.4-Feb  9 16:35:39 xen06 kernel: RBP: ffff88002682fb98 R08:
00007f7112d14000 R09: ffff88003e6e5a89
/var/log/messages.4-Feb  9 16:35:39 xen06 kernel: R10: ffff88003d2d0580 R11:
ffffffff813381a0 R12: 0000000000000000
/var/log/messages.4-Feb  9 16:35:39 xen06 kernel: R13: 00007f7110800000 R14:
ffff880026b047f0 R15: ffff880025943e20
/var/log/messages.4-Feb  9 16:35:39 xen06 kernel: FS:
00007f711128a710(0000) GS:ffff880028089000(0000) knlGS:0000000000000000
/var/log/messages.4-Feb  9 16:35:39 xen06 kernel: CS:  e033 DS: 0000 ES:
0000 CR0: 000000008005003b
/var/log/messages.4-Feb  9 16:35:39 xen06 kernel: CR2: ffff880026b00418 CR3:
0000000026946000 CR4: 0000000000002660
/var/log/messages.4-Feb  9 16:35:39 xen06 kernel: DR0: 0000000000000000 DR1:
0000000000000000 DR2: 0000000000000000
/var/log/messages.4-Feb  9 16:35:39 xen06 kernel: DR3: 0000000000000000 DR6:
00000000ffff0ff0 DR7: 0000000000000400
/var/log/messages.4-Feb  9 16:35:39 xen06 kernel: Process udevd (pid: 25953,
threadinfo ffff88002682e000, task ffff8800269ae380)
/var/log/messages.4-Feb  9 16:35:39 xen06 kernel: Stack:
/var/log/messages.4-Feb  9 16:35:39 xen06 kernel:  ffff880026b00418
000000003d32d067 ffff88002682fc48 ffffffff810a5e32
/var/log/messages.4-Feb  9 16:35:39 xen06 kernel: <0> ffff88002682fac8
000000000000001c 00007f7112c00000 0000000000000000
/var/log/messages.4-Feb  9 16:35:39 xen06 udevd-event[25941]: run_program:
'/sbin/dmsetup' abnormal exit
/var/log/messages.4-Feb  9 16:35:39 xen06 kernel: <0> 00007f71114ac000
ffff8800280975c0 00007f71114ac000 ffff88002682fcb8
/var/log/messages.4-Feb  9 16:35:39 xen06 kernel: Call Trace:
/var/log/messages.4-Feb  9 16:35:39 xen06 kernel:  [<ffffffff810a5e32>]
free_pgd_range+0x3b3/0x401
/var/log/messages.4-Feb  9 16:35:39 xen06 kernel:  [<ffffffff810a6133>]
free_pgtables+0xb6/0xce
/var/log/messages.4-Feb  9 16:35:39 xen06 kernel:  [<ffffffff810a8224>]
exit_mmap+0xf0/0x14b
/var/log/messages.4-Feb  9 16:35:39 xen06 kernel:  [<ffffffff81048648>]
mmput+0x46/0xb2
/var/log/messages.4-Feb  9 16:35:39 xen06 kernel:  [<ffffffff810c8a56>]
flush_old_exec+0x460/0x533
/var/log/messages.4-Feb  9 16:35:39 xen06 kernel:  [<ffffffff810fbd15>]
load_elf_binary+0x365/0x171c
/var/log/messages.4-Feb  9 16:35:39 xen06 kernel:  [<ffffffff810fa7ac>] ?
load_misc_binary+0x61/0x322
/var/log/messages.4-Feb  9 16:35:39 xen06 kernel:  [<ffffffff810a4e5f>] ?
get_user_pages+0x44/0x46
/var/log/messages.4-Feb  9 16:35:39 xen06 kernel:  [<ffffffff810c76de>] ?
put_arg_page+0x9/0xb
/var/log/messages.4-Feb  9 16:35:39 xen06 kernel:  [<ffffffff810c7b8b>]
search_binary_handler+0xde/0x268
/var/log/messages.4-Feb  9 16:35:39 xen06 kernel:  [<ffffffff810c8f85>]
do_execve+0x193/0x262
/var/log/messages.4-Feb  9 16:35:39 xen06 kernel:  [<ffffffff810113af>]
sys_execve+0x3e/0x58
/var/log/messages.4-Feb  9 16:35:39 xen06 kernel:  [<ffffffff81012f1a>]
stub_execve+0x6a/0xc0
/var/log/messages.4-Feb  9 16:35:39 xen06 kernel: Code: 8b 3c 25 70 c5 00 00
ff 14 25 68 82 5c 81 48 83 c4 18 5b c9 c3 55 48 89 e5 41 54 49 89 f4 53 48
89 fb e8 79 e3 ff ff 84 c0 75 05 <4c> 89 23 eb 0b 4c 89 e6 48 89 df e8 69 ff
ff ff 5b 41 5c c9 c3
/var/log/messages.4-Feb  9 16:35:39 xen06 kernel: RIP  [<ffffffff8100e2d4>]
xen_set_pmd+0x16/0x2b
/var/log/messages.4-Feb  9 16:35:39 xen06 kernel:  RSP <ffff88002682fb88>
/var/log/messages.4-Feb  9 16:35:39 xen06 kernel: CR2: ffff880026b00418
/var/log/messages.4-Feb  9 16:35:39 xen06 kernel: ---[ end trace
b5fffc59f6c8f975 ]---
/var/log/messages.4-Feb  9 16:35:40 xen06 lvm[3802]: Monitoring snapshot
XenGroup-testcrash5--snapshot
/var/log/messages.4-Feb  9 16:35:40 xen06 kernel: kjournald starting.
Commit interval 5 seconds
/var/log/messages.4-Feb  9 16:35:40 xen06 kernel: EXT3 FS on dm-11, internal
journal
/var/log/messages.4-Feb  9 16:35:40 xen06 kernel: EXT3-fs: mounted
filesystem with ordered data mode.
/var/log/messages.4-Feb  9 16:35:40 xen06 lvm[3802]: No longer monitoring
snapshot XenGroup-testcrash5--snapshot
/var/log/messages.4-Feb  9 16:35:41 xen06 lvm[3802]: Monitoring snapshot
XenGroup-testcrash1--snapshot

Thanks.

Kindest regards,
Giam Teck Choon

[-- Attachment #1.2: Type: text/html, Size: 41656 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: kernel BUG at arch/x86/xen/mmu.c:1860!

[Teck Choon Giam]

[-- Attachment #1.1: Type: text/plain, Size: 569 bytes --]

On Sat, Mar 12, 2011 at 3:59 AM, Sander Eikelenboom <linux@eikelenboom.it>wrote:

> Haven't seen this one on my systen. I do use LVM, but i'm only using LVM as
> device mapper, without software raid etc.
> Also using debian lenny/squeeze together with custom compiled xen and
> xen-2.6.32.x pvops kernel
>
> So I would suspect the multipathing / raid part.
>

If you do not use LVM snapshot create/delete then you mostly won't hit this
BUG.  LVM create/delete might but very rare but I hit this before... so
unlucky for me :(

Thanks.

Kindest regards,
Giam Teck Choon

[-- Attachment #1.2: Type: text/html, Size: 879 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: kernel BUG at arch/x86/xen/mmu.c:1860!

[Sander Eikelenboom]

You did have some test script for creating/delete snapshots that triggers it ?

Could try to test it with 2.6.38 as dom0 now, i you could resend that.



Friday, March 11, 2011, 9:45:31 PM, you wrote:

> On Sat, Mar 12, 2011 at 3:59 AM, Sander Eikelenboom <linux@eikelenboom.it>wrote:

>> Haven't seen this one on my systen. I do use LVM, but i'm only using LVM as
>> device mapper, without software raid etc.
>> Also using debian lenny/squeeze together with custom compiled xen and
>> xen-2.6.32.x pvops kernel
>>
>> So I would suspect the multipathing / raid part.
>>

> If you do not use LVM snapshot create/delete then you mostly won't hit this
> BUG.  LVM create/delete might but very rare but I hit this before... so
> unlucky for me :(

> Thanks.

> Kindest regards,
> Giam Teck Choon



-- 
Best regards,
 Sander                            mailto:linux@eikelenboom.it

Re: kernel BUG at arch/x86/xen/mmu.c:1860!

[Teck Choon Giam]

[-- Attachment #1.1: Type: text/plain, Size: 401 bytes --]

On Sat, Mar 12, 2011 at 5:02 AM, Sander Eikelenboom <linux@eikelenboom.it>wrote:

> You did have some test script for creating/delete snapshots that triggers
> it ?
>

Yes, I do and it is posted in this thread.


>
> Could try to test it with 2.6.38 as dom0 now, i you could resend that.
>

According to Andreas Olsowski testing, 2.6.38 doesn't suffer this.

Thanks.

Kindest regards,
Giam Teck Choon

[-- Attachment #1.2: Type: text/html, Size: 864 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: kernel BUG at arch/x86/xen/mmu.c:1860!

[Konrad Rzeszutek Wilk]

> > 4000000000000000: caf=8000000000000003 taf=4000000000000001
> > (XEN) mm.c:2825:d0 Error while installing new mfn 3b286
> > (XEN) mm.c:2794:d0 Mfn 35620 not pinned
> > (XEN) mm.c:2794:d0 Mfn 3b286 not pinned
> >
> > Takes a bit of time to reproduce it, but I can reproduce it on my CentOS
> > 5.5 OS.
> >
> 
> Yep... it takes sometime to hit this bug and if you run my test script
> without any guest running will be faster to reproduce it.
> 
> May I know what is your normal development platform/distribution for xen?  I

It used to be Fedora Core 13, but then I switched to Ubuntu 10.10 and 11.04.

> hope this isn't just distribution specific issue for PVOPS
> stable/xen-2.6.32.x Dom0... ... mostly not as from this thread alone I see
> someone saying they hit this bug as well and I believe they are not running
> CentOS 5.x... guess is debian based distribution.

I am trying to eliminate this being a userspace tool version issue. I am running a
similar test on a Fedora Core 13 box.

Re: kernel BUG at arch/x86/xen/mmu.c:1860!

[Christophe Saout]

Hello thread,

hijacking this thread since I am running into the same issue on a new
machine.

> >      > While doing LVM snapshot for migration and get the following:
> >      >
> >      > Dec 26 15:58:29 xen01 kernel: ------------[ cut here ]------------
> >      > Dec 26 15:58:29 xen01 kernel: kernel BUG at arch/x86/xen/mmu.c:1860!
> >      > Dec 26 15:58:29 xen01 kernel: invalid opcode: 0000 [#1] SMP
> >      > Dec 26 15:58:29 xen01 kernel: last sysfs file: /sys/block/dm-26/dev
> >      > Dec 26 15:58:29 xen01 kernel: CPU 0
> >      > Dec 26 15:58:29 xen01 kernel: Modules linked in: ipt_MASQUERADE
>
> It would be very good to track this down and get it fixed.. 
> hopefully you're able to help a bit and try some things to debug it.
> 
> Konrad maybe has some ideas to try.. 

I am seeing this with an lvcreate here, so I guess it's somehow related
to device-mapper stuff in general.

It doesn't look like this has been resolved yet.  Somewhere I saw a
request for the hypervisor message related to the pinning failure.

Here it is:

(XEN) mm.c:2364:d0 Bad type (saw 7400000000000001 != exp 1000000000000000) for mfn 41114f (pfn d514f)
(XEN) mm.c:2733:d0 Error while pinning mfn 41114f

I have a bit of experience in debugging things, so if I can help someone
with more information...

Cheers,
	Christophe

Re: kernel BUG at arch/x86/xen/mmu.c:1860!

[Christophe Saout]

Hi again,

> > >      > While doing LVM snapshot for migration and get the following:
> > >      >
> > >      > Dec 26 15:58:29 xen01 kernel: ------------[ cut here ]------------
> > >      > Dec 26 15:58:29 xen01 kernel: kernel BUG at arch/x86/xen/mmu.c:1860!
> > >      > Dec 26 15:58:29 xen01 kernel: invalid opcode: 0000 [#1] SMP
> > >      > Dec 26 15:58:29 xen01 kernel: last sysfs file: /sys/block/dm-26/dev
> > >      > Dec 26 15:58:29 xen01 kernel: CPU 0
> > >      > Dec 26 15:58:29 xen01 kernel: Modules linked in: ipt_MASQUERADE
> >
> > It would be very good to track this down and get it fixed.. 
> > hopefully you're able to help a bit and try some things to debug it.
> > 
> > Konrad maybe has some ideas to try.. 
> 
> I am seeing this with an lvcreate here, so I guess it's somehow related
> to device-mapper stuff in general.
> 
> It doesn't look like this has been resolved yet.  Somewhere I saw a
> request for the hypervisor message related to the pinning failure.
> 
> Here it is:
> 
> (XEN) mm.c:2364:d0 Bad type (saw 7400000000000001 != exp 1000000000000000) for mfn 41114f (pfn d514f)
> (XEN) mm.c:2733:d0 Error while pinning mfn 41114f
> 
> I have a bit of experience in debugging things, so if I can help someone
> with more information...

Additional information: This happened with a number of commands now.
However, I am running a multipath setup and every time the crash seemed
to be caused in the process context of the multipath daemon.  I think
the daemon listens to events from the device-mapper subsystem to watch
for changes and the problem somehow arises from there, since on another
machine with the same XEN/Dom0 version without such a daemon I never had
any troubles with LVM.

 [<ffffffff810052e2>] pin_pagetable_pfn+0x52/0x60    
 [<ffffffff81006f5c>] xen_alloc_ptpage+0x9c/0xa0
 [<ffffffff81006f8e>] xen_alloc_pte+0xe/0x10
 [<ffffffff810decde>] __pte_alloc+0x7e/0xf0
 [<ffffffff810e15c5>] handle_mm_fault+0x855/0x930
 [<ffffffff8102dd9e>] ? pvclock_clocksource_read+0x4e/0x100
 [<ffffffff810e734c>] ? do_mmap_pgoff+0x33c/0x380
 [<ffffffff81452b96>] do_page_fault+0x116/0x3e0
 [<ffffffff8144ff65>] page_fault+0x25/0x30

Cheers,
	Christophe

Re: kernel BUG at arch/x86/xen/mmu.c:1860!

[benco]

Hi,

the same issue on old IBM x335 servers with QLogic ISP2312 HBAs and
multipath setup. Newer hardware (x3550M2 + QLA2462 HBA) seems to be fine.
I've tried several versions of pvops kernel and xen hypervisor from 4.x line
with the same results.

Roman

On Tue, Jan 04, 2011 at 04:19:02PM +0100, Christophe Saout wrote:
> Hi again,
> 
> > > >      > While doing LVM snapshot for migration and get the following:
> > > >      >
> > > >      > Dec 26 15:58:29 xen01 kernel: ------------[ cut here ]------------
> > > >      > Dec 26 15:58:29 xen01 kernel: kernel BUG at arch/x86/xen/mmu.c:1860!
> > > >      > Dec 26 15:58:29 xen01 kernel: invalid opcode: 0000 [#1] SMP
> > > >      > Dec 26 15:58:29 xen01 kernel: last sysfs file: /sys/block/dm-26/dev
> > > >      > Dec 26 15:58:29 xen01 kernel: CPU 0
> > > >      > Dec 26 15:58:29 xen01 kernel: Modules linked in: ipt_MASQUERADE
> > >
> > > It would be very good to track this down and get it fixed.. 
> > > hopefully you're able to help a bit and try some things to debug it.
> > > 
> > > Konrad maybe has some ideas to try.. 
> > 
> > I am seeing this with an lvcreate here, so I guess it's somehow related
> > to device-mapper stuff in general.
> > 
> > It doesn't look like this has been resolved yet.  Somewhere I saw a
> > request for the hypervisor message related to the pinning failure.
> > 
> > Here it is:
> > 
> > (XEN) mm.c:2364:d0 Bad type (saw 7400000000000001 != exp 1000000000000000) for mfn 41114f (pfn d514f)
> > (XEN) mm.c:2733:d0 Error while pinning mfn 41114f
> > 
> > I have a bit of experience in debugging things, so if I can help someone
> > with more information...
> 
> Additional information: This happened with a number of commands now.
> However, I am running a multipath setup and every time the crash seemed
> to be caused in the process context of the multipath daemon.  I think
> the daemon listens to events from the device-mapper subsystem to watch
> for changes and the problem somehow arises from there, since on another
> machine with the same XEN/Dom0 version without such a daemon I never had
> any troubles with LVM.
> 
>  [<ffffffff810052e2>] pin_pagetable_pfn+0x52/0x60    
>  [<ffffffff81006f5c>] xen_alloc_ptpage+0x9c/0xa0
>  [<ffffffff81006f8e>] xen_alloc_pte+0xe/0x10
>  [<ffffffff810decde>] __pte_alloc+0x7e/0xf0
>  [<ffffffff810e15c5>] handle_mm_fault+0x855/0x930
>  [<ffffffff8102dd9e>] ? pvclock_clocksource_read+0x4e/0x100
>  [<ffffffff810e734c>] ? do_mmap_pgoff+0x33c/0x380
>  [<ffffffff81452b96>] do_page_fault+0x116/0x3e0
>  [<ffffffff8144ff65>] page_fault+0x25/0x30
> 
> Cheers,
> 	Christophe
> 
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

-- 
----------------------------------------------------------------------
  ,''`.       [benco] | mailto: benco@acid.sk | silc: /msg benco
 : :' :  -------------------------------------------------------------
 `. `'           GPG publickey: http://www.acid.sk/pubkey.asc
   `-      KF  =  0DF6 0592 74D2 F17A DACF  A5C3 1720 CB7C F54C F429

Re: kernel BUG at arch/x86/xen/mmu.c:1860!

[Christophe Saout]

Hi once more,


> > It doesn't look like this has been resolved yet.  Somewhere I saw a
> > request for the hypervisor message related to the pinning failure.
> > 
> > Here it is:
> > 
> > (XEN) mm.c:2364:d0 Bad type (saw 7400000000000001 != exp 1000000000000000) for mfn 41114f (pfn d514f)
> > (XEN) mm.c:2733:d0 Error while pinning mfn 41114f
> > 
> > I have a bit of experience in debugging things, so if I can help someone
> > with more information...
>  [<ffffffff810052e2>] pin_pagetable_pfn+0x52/0x60    
>  [<ffffffff81006f5c>] xen_alloc_ptpage+0x9c/0xa0
>  [<ffffffff81006f8e>] xen_alloc_pte+0xe/0x10
>  [<ffffffff810decde>] __pte_alloc+0x7e/0xf0
>  [<ffffffff810e15c5>] handle_mm_fault+0x855/0x930
>  [<ffffffff8102dd9e>] ? pvclock_clocksource_read+0x4e/0x100
>  [<ffffffff810e734c>] ? do_mmap_pgoff+0x33c/0x380
>  [<ffffffff81452b96>] do_page_fault+0x116/0x3e0
>  [<ffffffff8144ff65>] page_fault+0x25/0x30

> Additional information: This happened with a number of commands now.
> However, I am running a multipath setup and every time the crash
> seemed to be caused in the process context of the multipath daemon. 
> I think the daemon listens to events from the device-mapper subsystem
> to watch for changes and the problem somehow arises from there, since
> on another machine with the same XEN/Dom0 version without such a
> daemon I never had any troubles with LVM.

On further investigation is seems that most of the time the issue is not
caused by the daemon, but by the "multipath" tool, which is used a lot
by udev to identify properties of block devices.

When I start stracing udevd (following forks), I'm not able to reproduce
the crash anymore.  So I was hoping to find out what the process was
doing before the crash occurs, but since my attempts to trace the
process masks the bug, I can't. :(

(without strace, the bug is very common, about every third "lvcreate"
command.  Every lvcreate command triggers about 20 multipath
invocations)

	Christophe

Re: kernel BUG at arch/x86/xen/mmu.c:1860!

[Teck Choon Giam]

[-- Attachment #1.1: Type: text/plain, Size: 2887 bytes --]

On Wed, Jan 5, 2011 at 2:40 AM, Christophe Saout <christophe@saout.de>wrote:

> Hi once more,
>
>
> > > It doesn't look like this has been resolved yet.  Somewhere I saw a
> > > request for the hypervisor message related to the pinning failure.
> > >
> > > Here it is:
> > >
> > > (XEN) mm.c:2364:d0 Bad type (saw 7400000000000001 != exp
> 1000000000000000) for mfn 41114f (pfn d514f)
> > > (XEN) mm.c:2733:d0 Error while pinning mfn 41114f
> > >
> > > I have a bit of experience in debugging things, so if I can help
> someone
> > > with more information...
> >  [<ffffffff810052e2>] pin_pagetable_pfn+0x52/0x60
> >  [<ffffffff81006f5c>] xen_alloc_ptpage+0x9c/0xa0
> >  [<ffffffff81006f8e>] xen_alloc_pte+0xe/0x10
> >  [<ffffffff810decde>] __pte_alloc+0x7e/0xf0
> >  [<ffffffff810e15c5>] handle_mm_fault+0x855/0x930
> >  [<ffffffff8102dd9e>] ? pvclock_clocksource_read+0x4e/0x100
> >  [<ffffffff810e734c>] ? do_mmap_pgoff+0x33c/0x380
> >  [<ffffffff81452b96>] do_page_fault+0x116/0x3e0
> >  [<ffffffff8144ff65>] page_fault+0x25/0x30
>
> > Additional information: This happened with a number of commands now.
> > However, I am running a multipath setup and every time the crash
> > seemed to be caused in the process context of the multipath daemon.
> > I think the daemon listens to events from the device-mapper subsystem
> > to watch for changes and the problem somehow arises from there, since
> > on another machine with the same XEN/Dom0 version without such a
> > daemon I never had any troubles with LVM.
>
> On further investigation is seems that most of the time the issue is not
> caused by the daemon, but by the "multipath" tool, which is used a lot
> by udev to identify properties of block devices.
>
> When I start stracing udevd (following forks), I'm not able to reproduce
> the crash anymore.  So I was hoping to find out what the process was
> doing before the crash occurs, but since my attempts to trace the
> process masks the bug, I can't. :(
>
> (without strace, the bug is very common, about every third "lvcreate"
> command.  Every lvcreate command triggers about 20 multipath
> invocations)
>
>
I am able to prevent that bug for 8 days (till now) by implementing sleep 5
seconds then syc then sleep 5 seconds then sync repeating this for 60
seconds while doing lvm snapshot for 10 domUs.  I mean:

1. lvm snapshot domU (lvcreate)
2. mount lvm snapsho domUt
3. rsync to backup domU
4. umount lvm snapshot domU
5. remove lvm snapshot domU (lvremove)
6. sync (start countdown of 60 seconds and every 5 seconds interval doing
sync)
7. sleep 5
8. sync
9. sleep 5
10. sync
11. sleep 5
12. sync
.... until it hits 0 second countdown
Then next domU repeat the cycle.

Doing the above I am able to prevent such crash or bug to pop up for 8 days
(8 such daily LVM snapshot backup for all domUs) which I posted in this
thread.

Thanks.

Kindest regards,
Giam Teck Choon

[-- Attachment #1.2: Type: text/html, Size: 3622 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: kernel BUG at arch/x86/xen/mmu.c:1860!

[benco]

Hi,

I use the same workaround to start domUs, since the issue mentioned bellow
ocured also during domU start via xendomains script.

Another scenario causing this issue (at least for me) are cron scripts. I was
unable to find out which one is responsible (crash every 2-3 days), but the
issue went away with disabled cronscripts.

Roman

On Wed, Jan 05, 2011 at 03:32:58AM +0800, Teck Choon Giam wrote:
> On Wed, Jan 5, 2011 at 2:40 AM, Christophe Saout <christophe@saout.de>wrote:
> 
> > Hi once more,
> >
> >
> > > > It doesn't look like this has been resolved yet.  Somewhere I saw a
> > > > request for the hypervisor message related to the pinning failure.
> > > >
> > > > Here it is:
> > > >
> > > > (XEN) mm.c:2364:d0 Bad type (saw 7400000000000001 != exp
> > 1000000000000000) for mfn 41114f (pfn d514f)
> > > > (XEN) mm.c:2733:d0 Error while pinning mfn 41114f
> > > >
> > > > I have a bit of experience in debugging things, so if I can help
> > someone
> > > > with more information...
> > >  [<ffffffff810052e2>] pin_pagetable_pfn+0x52/0x60
> > >  [<ffffffff81006f5c>] xen_alloc_ptpage+0x9c/0xa0
> > >  [<ffffffff81006f8e>] xen_alloc_pte+0xe/0x10
> > >  [<ffffffff810decde>] __pte_alloc+0x7e/0xf0
> > >  [<ffffffff810e15c5>] handle_mm_fault+0x855/0x930
> > >  [<ffffffff8102dd9e>] ? pvclock_clocksource_read+0x4e/0x100
> > >  [<ffffffff810e734c>] ? do_mmap_pgoff+0x33c/0x380
> > >  [<ffffffff81452b96>] do_page_fault+0x116/0x3e0
> > >  [<ffffffff8144ff65>] page_fault+0x25/0x30
> >
> > > Additional information: This happened with a number of commands now.
> > > However, I am running a multipath setup and every time the crash
> > > seemed to be caused in the process context of the multipath daemon.
> > > I think the daemon listens to events from the device-mapper subsystem
> > > to watch for changes and the problem somehow arises from there, since
> > > on another machine with the same XEN/Dom0 version without such a
> > > daemon I never had any troubles with LVM.
> >
> > On further investigation is seems that most of the time the issue is not
> > caused by the daemon, but by the "multipath" tool, which is used a lot
> > by udev to identify properties of block devices.
> >
> > When I start stracing udevd (following forks), I'm not able to reproduce
> > the crash anymore.  So I was hoping to find out what the process was
> > doing before the crash occurs, but since my attempts to trace the
> > process masks the bug, I can't. :(
> >
> > (without strace, the bug is very common, about every third "lvcreate"
> > command.  Every lvcreate command triggers about 20 multipath
> > invocations)
> >
> >
> I am able to prevent that bug for 8 days (till now) by implementing sleep 5
> seconds then syc then sleep 5 seconds then sync repeating this for 60
> seconds while doing lvm snapshot for 10 domUs.  I mean:
> 
> 1. lvm snapshot domU (lvcreate)
> 2. mount lvm snapsho domUt
> 3. rsync to backup domU
> 4. umount lvm snapshot domU
> 5. remove lvm snapshot domU (lvremove)
> 6. sync (start countdown of 60 seconds and every 5 seconds interval doing
> sync)
> 7. sleep 5
> 8. sync
> 9. sleep 5
> 10. sync
> 11. sleep 5
> 12. sync
> .... until it hits 0 second countdown
> Then next domU repeat the cycle.
> 
> Doing the above I am able to prevent such crash or bug to pop up for 8 days
> (8 such daily LVM snapshot backup for all domUs) which I posted in this
> thread.
> 
> Thanks.
> 
> Kindest regards,
> Giam Teck Choon

> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel


-- 
----------------------------------------------------------------------
  ,''`.       [benco] | mailto: benco@acid.sk | silc: /msg benco
 : :' :  -------------------------------------------------------------
 `. `'           GPG publickey: http://www.acid.sk/pubkey.asc
   `-      KF  =  0DF6 0592 74D2 F17A DACF  A5C3 1720 CB7C F54C F429

Re: kernel BUG at arch/x86/xen/mmu.c:1860!

[Konrad Rzeszutek Wilk]

> (without strace, the bug is very common, about every third "lvcreate"
> command.  Every lvcreate command triggers about 20 multipath

Something must be busted with your udev rules for this to go in effect.
You should have multipathd process those and not run multipath for
every device mapper call.

Re: kernel BUG at arch/x86/xen/mmu.c:1860!

[Christophe Saout]

Hi Konrad,

> > (without strace, the bug is very common, about every third "lvcreate"
> > command.  Every lvcreate command triggers about 20 multipath
> 
> Something must be busted with your udev rules for this to go in effect.
> You should have multipathd process those and not run multipath for
> every device mapper call.

This is a standard Debian squeeze.  I think udev is just calling those
to figure out if a device should be handled by multipath or not (part of
the device identification). I'd rather not fiddle with it, as it "works"
even if Debian is shipping suboptimal rules. Anyhow, changing this would
simply reduce the risk of running into the kernel issue, not remove it
entirely, so I prefer the actual fix. :)

	Christophe

Re: kernel BUG at arch/x86/xen/mmu.c:1860!

[Christophe Saout]

Hi,

> > >      > While doing LVM snapshot for migration and get the following:
> > >      >
> > >      > Dec 26 15:58:29 xen01 kernel: ------------[ cut here ]------------
> > >      > Dec 26 15:58:29 xen01 kernel: kernel BUG at arch/x86/xen/mmu.c:1860!
> > >      > Dec 26 15:58:29 xen01 kernel: invalid opcode: 0000 [#1] SMP
> > >      > Dec 26 15:58:29 xen01 kernel: last sysfs file: /sys/block/dm-26/dev
> > >      > Dec 26 15:58:29 xen01 kernel: CPU 0
> > >      > Dec 26 15:58:29 xen01 kernel: Modules linked in: ipt_MASQUERADE
>
> [...]
> [<ffffffff810052e2>] pin_pagetable_pfn+0x52/0x60    
> [<ffffffff81006f5c>] xen_alloc_ptpage+0x9c/0xa0
> [<ffffffff81006f8e>] xen_alloc_pte+0xe/0x10
> [<ffffffff810decde>] __pte_alloc+0x7e/0xf0
> [<ffffffff810e15c5>] handle_mm_fault+0x855/0x930
> [<ffffffff8102dd9e>] ? pvclock_clocksource_read+0x4e/0x100
> [<ffffffff810e734c>] ? do_mmap_pgoff+0x33c/0x380
> [<ffffffff81452b96>] do_page_fault+0x116/0x3e0
> [<ffffffff8144ff65>] page_fault+0x25/0x30
> [...]
> (XEN) mm.c:2364:d0 Bad type (saw 7400000000000001 != exp 1000000000000000) for mfn 41114f (pfn d514f)
> (XEN) mm.c:2733:d0 Error while pinning mfn 41114f

Looking into the code, the Dom0 code ist attempting to pin what it thins
is a "PGT_l1_page_table", however the hypervisor returns -EINVAL because
it actually is a "PGT_writable_page".

After a few hours I managed to catch the crash while the offending
process is being straced.  However the results where totally
inconclusive, because the last lines before the crash are:

16576 open("/lib/multipath/libcheckdirectio.so", O_RDONLY) = 4
16576 read(4, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0P\v\0\0\0\0\0\0"..., 832) = 832
16576 fstat(4, {st_mode=S_IFREG|0644, st_size=9344, ...}) = 0
16576 mmap(NULL, 2104672, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 4, 0) = 0x7fa6b36f6000
16576 mprotect(0x7fa6b36f8000, 2093056, PROT_NONE) = 0
16576 mmap(0x7fa6b38f7000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 4, 0x1000) = 0x7fa6b38f7000
16576 close(4)                          = 0

A non-crashing execution would have continued with:

16667 open("/etc/ld.so.cache", O_RDONLY) = 4
16667 fstat(4, {st_mode=S_IFREG|0644, st_size=21739, ...}) = 0
16667 mmap(NULL, 21739, PROT_READ, MAP_PRIVATE, 4, 0) = 0x7f237de56000
16667 close(4)                          = 0
16667 access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
16667 open("/lib/libaio.so.1", O_RDONLY) = 4
[...]

Which means that it crashed during the dynamic loading of a plugin
shared library and not while interacting with the device mapper.
(also, the device being investigated was /dev/sde and not some dm
device)

This leads me to believe that some device-mapper shared library has a
particular memory layout that tends to trigger this crash and it has
nothing to do with any device-mapper code at all.  Also, the crash seems
to be timing-sensitive, so it might also be a race condition of some
sort. (on a side-note: this is a 24-core machine (!) and the kernel has
happens to have full preemption enabled).

I am trying to understand the code a bit.  Can someone explain to me
what xen_alloc_ptpage is doing.

> /* This needs to make sure the new pte page is pinned iff its being
>   attached to a pinned pagetable. */
> [...]
> if (PagePinned(virt_to_page(mm->pgd))) {
>     [...]
>     pin_pagetable_pfn(MMUEXT_PIN_L1_TABLE, pfn);

I must admit I don't know very much about memory handling in linux (so
please excuse me if I am interpreting total nonsense into this here,
still I'm intigued and would like to understand it a bit better), but
isn't `mm->pgd' supposed to point to the L1 page table and `pfn', being
a pte page a 3rd/4th level page?  Is this a code path that is exercised
a lot?

Thanks,
	Christophe

Re: kernel BUG at arch/x86/xen/mmu.c:1860!

[Pasi Kärkkäinen]

On Tue, Jan 04, 2011 at 04:10:17PM +0100, Christophe Saout wrote:
> Hello thread,
> 
> hijacking this thread since I am running into the same issue on a new
> machine.
> 
> > >      > While doing LVM snapshot for migration and get the following:
> > >      >
> > >      > Dec 26 15:58:29 xen01 kernel: ------------[ cut here ]------------
> > >      > Dec 26 15:58:29 xen01 kernel: kernel BUG at arch/x86/xen/mmu.c:1860!
> > >      > Dec 26 15:58:29 xen01 kernel: invalid opcode: 0000 [#1] SMP
> > >      > Dec 26 15:58:29 xen01 kernel: last sysfs file: /sys/block/dm-26/dev
> > >      > Dec 26 15:58:29 xen01 kernel: CPU 0
> > >      > Dec 26 15:58:29 xen01 kernel: Modules linked in: ipt_MASQUERADE
> >
> > It would be very good to track this down and get it fixed.. 
> > hopefully you're able to help a bit and try some things to debug it.
> > 
> > Konrad maybe has some ideas to try.. 
> 
> I am seeing this with an lvcreate here, so I guess it's somehow related
> to device-mapper stuff in general.
> 

Sorry if this was already stated earlier..
what are the exact steps to reproduce? I could try reproducing it at some point..

-- Pasi

> It doesn't look like this has been resolved yet.  Somewhere I saw a
> request for the hypervisor message related to the pinning failure.
> 
> Here it is:
> 
> (XEN) mm.c:2364:d0 Bad type (saw 7400000000000001 != exp 1000000000000000) for mfn 41114f (pfn d514f)
> (XEN) mm.c:2733:d0 Error while pinning mfn 41114f
> 
> I have a bit of experience in debugging things, so if I can help someone
> with more information...
> 
> Cheers,
> 	Christophe
> 
> 

Re: kernel BUG at arch/x86/xen/mmu.c:1860!

[Teck Choon Giam]

[-- Attachment #1.1: Type: text/plain, Size: 8774 bytes --]

On Wed, Jan 5, 2011 at 6:51 PM, Pasi Kärkkäinen <pasik@iki.fi> wrote:

> On Tue, Jan 04, 2011 at 04:10:17PM +0100, Christophe Saout wrote:
> > Hello thread,
> >
> > hijacking this thread since I am running into the same issue on a new
> > machine.
> >
> > > >      > While doing LVM snapshot for migration and get the following:
> > > >      >
> > > >      > Dec 26 15:58:29 xen01 kernel: ------------[ cut here
> ]------------
> > > >      > Dec 26 15:58:29 xen01 kernel: kernel BUG at
> arch/x86/xen/mmu.c:1860!
> > > >      > Dec 26 15:58:29 xen01 kernel: invalid opcode: 0000 [#1] SMP
> > > >      > Dec 26 15:58:29 xen01 kernel: last sysfs file:
> /sys/block/dm-26/dev
> > > >      > Dec 26 15:58:29 xen01 kernel: CPU 0
> > > >      > Dec 26 15:58:29 xen01 kernel: Modules linked in:
> ipt_MASQUERADE
> > >
> > > It would be very good to track this down and get it fixed..
> > > hopefully you're able to help a bit and try some things to debug it.
> > >
> > > Konrad maybe has some ideas to try..
> >
> > I am seeing this with an lvcreate here, so I guess it's somehow related
> > to device-mapper stuff in general.
> >
>
> Sorry if this was already stated earlier..
> what are the exact steps to reproduce? I could try reproducing it at some
> point..
>
> -- Pasi
>


Did you try my posted script?  Provided you have existing LV for domUs in
your VG which can be easily created if not there.

The idea is to create snapshot, mount it, umount it, remove snapshot and
repeat this cycle in loop will catch this BUG!!!

Here are the latest crash with serial console output as it doesn't take long
with sh test.sh loop 100 to produce this:

EXT3-fs warning: maximal mount count reached, running e2fsck is recommended
EXT3-fs warning: maximal mount count reached, running e2fsck is recommended
EXT3-fs warning: maximal mount count reached, running e2fsck is recommended
EXT3-fs warning: maximal mount count reached, running e2fsck is recommended
(XEN) mm.c:2364:d0 Bad type (saw 7400000000000001 != exp 1000000000000000)
for mfn 1f7e29 (pfn 25cb4)
(XEN) mm.c:2733:d0 Error while pinning mfn 1f7e29
------------[ cut here ]------------
kernel BUG at arch/x86/xen/mmu.c:1860!
invalid opcode: 0000 [#1] SMP
last sysfs file:
/sys/devices/pci0000:00/0000:00:1e.0/0000:05:05.0/local_cpus
CPU 0
Modules linked in: ext4 jbd2 crc16 gfs2 dlm configfs xt_physdev
iptable_filter ip_tables x_tables bridge stp be2iscsi iscsi_]
Pid: 6526, comm: dmsetup Not tainted 2.6.32.27-0.xen.pvops.choon.centos5 #1
PowerEdge 860
RIP: e030:[<ffffffff8100cb5b>]  [<ffffffff8100cb5b>]
pin_pagetable_pfn+0x53/0x59
RSP: e02b:ffff88001d8dfdc8  EFLAGS: 00010282
RAX: 00000000ffffffea RBX: 0000000000025cb4 RCX: 000000000000012e
RDX: 00000000deadbeef RSI: 00000000deadbeef RDI: 00000000deadbeef
RBP: ffff88001d8dfde8 R08: 00000000000005a0 R09: ffff880000000000
R10: 00000000deadbeef R11: 0000003db6814e00 R12: 0000000000000003
R13: 0000000000025cb4 R14: ffff88002ffe8440 R15: 0000003db7616250
FS:  00007fb54068b710(0000) GS:ffff88002804f000(0000) knlGS:0000000000000000
CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000003db7616250 CR3: 000000002ff88000 CR4: 0000000000002660
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process dmsetup (pid: 6526, threadinfo ffff88001d8de000, task
ffff88002ffe8440)
Stack:
 0000000000000000 00000000001f7e29 000000013f009e18 0000000000025cb4
<0> ffff88001d8dfe08 ffffffff8100e07c ffff88001dc3d040 ffff88001dc6cdd8
<0> ffff88001d8dfe18 ffffffff8100e0af ffff88001d8dfe58 ffffffff810a402f
Call Trace:
 [<ffffffff8100e07c>] xen_alloc_ptpage+0x64/0x69
 [<ffffffff8100e0af>] xen_alloc_pte+0xe/0x10
 [<ffffffff810a402f>] __pte_alloc+0x70/0xce
 [<ffffffff810a41cd>] handle_mm_fault+0x140/0x8b9
 [<ffffffff8131be4d>] do_page_fault+0x252/0x2e2
 [<ffffffff81319dd5>] page_fault+0x25/0x30
Code: 48 b8 ff ff ff ff ff ff ff 7f 48 21 c2 48 89 55 e8 48 8d 7d e0 be 01
00 00 00 31 d2 41 ba f0 7f 00 00 e8 e9 c7 ff ff 8
RIP  [<ffffffff8100cb5b>] pin_pagetable_pfn+0x53/0x59
 RSP <ffff88001d8dfdc8>
---[ end trace b0a2643219f652eb ]---
BUG: soft lockup - CPU#0 stuck for 61s! [dmsetup:6526]
Modules linked in: ext4 jbd2 crc16 gfs2 dlm configfs xt_physdev
iptable_filter ip_tables x_tables bridge stp be2iscsi iscsi_]
CPU 0:
Modules linked in: ext4 jbd2 crc16 gfs2 dlm configfs xt_physdev
iptable_filter ip_tables x_tables bridge stp be2iscsi iscsi_]
Pid: 6526, comm: dmsetup Tainted: G      D
2.6.32.27-0.xen.pvops.choon.centos5 #1 PowerEdge 860
RIP: e030:[<ffffffff813199d3>]  [<ffffffff813199d3>] _spin_lock+0x19/0x20
RSP: e02b:ffff88001d8dfa68  EFLAGS: 00000297
RAX: 0000000000000023 RBX: 0000000025f91000 RCX: 0000000000000004
RDX: 0000000000000022 RSI: 0000000000000004 RDI: ffff88001dc3d0c0
RBP: ffff88001d8dfa68 R08: 0000000000000000 R09: ffffffff816dd100
R10: ffff88003e7424c8 R11: 0000000000000020 R12: ffff88001dc3d040
R13: 0000000000000004 R14: ffff88001dc3d0a0 R15: ffffffff816dd100
FS:  00007fb54068b710(0000) GS:ffff88002804f000(0000) knlGS:0000000000000000
CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000003db7616250 CR3: 0000000001001000 CR4: 0000000000002660
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Call Trace:
 [<ffffffff811660dd>] ? free_cpumask_var+0x9/0xb
 [<ffffffff8100dde1>] xen_exit_mmap+0x199/0x1d7
 [<ffffffff810a8137>] exit_mmap+0x5f/0x14b
 [<ffffffff81048648>] mmput+0x46/0xb2
 [<ffffffff8104c552>] exit_mm+0xfd/0x108
 [<ffffffff8100f799>] ? xen_irq_enable_direct_end+0x0/0x7
 [<ffffffff8104d7ee>] do_exit+0x1f3/0x67b
 [<ffffffff8131a908>] oops_end+0xba/0xc2
 [<ffffffff810163a1>] die+0x55/0x5e
 [<ffffffff8131a192>] do_trap+0x110/0x11f
 [<ffffffff810142c8>] do_invalid_op+0x97/0xa0
 [<ffffffff8100cb5b>] ? pin_pagetable_pfn+0x53/0x59
 [<ffffffff810138bb>] invalid_op+0x1b/0x20
 [<ffffffff8100cb5b>] ? pin_pagetable_pfn+0x53/0x59
 [<ffffffff8100cb57>] ? pin_pagetable_pfn+0x4f/0x59
 [<ffffffff8100e07c>] xen_alloc_ptpage+0x64/0x69
 [<ffffffff8100e0af>] xen_alloc_pte+0xe/0x10
 [<ffffffff810a402f>] __pte_alloc+0x70/0xce
 [<ffffffff810a41cd>] handle_mm_fault+0x140/0x8b9
 [<ffffffff8131be4d>] do_page_fault+0x252/0x2e2
 [<ffffffff81319dd5>] page_fault+0x25/0x30
Kernel panic - not syncing: softlockup: hung tasks
Pid: 6526, comm: dmsetup Tainted: G      D
2.6.32.27-0.xen.pvops.choon.centos5 #1
Call Trace:
 <IRQ>  [<ffffffff8104aa97>] panic+0xa0/0x15f
 [<ffffffff81319dd5>] ? page_fault+0x25/0x30
 [<ffffffff8101640f>] ? show_trace_log_lvl+0x4c/0x58
 [<ffffffff8101642b>] ? show_trace+0x10/0x12
 [<ffffffff81011755>] ? show_regs+0x44/0x48
 [<ffffffff8107f202>] softlockup_tick+0x173/0x182
 [<ffffffff810539bf>] run_local_timers+0x18/0x1a
 [<ffffffff81053bde>] update_process_times+0x30/0x54
 [<ffffffff81068821>] tick_sched_timer+0x70/0x99
 [<ffffffff8105f52e>] __run_hrtimer+0x53/0xb3
 [<ffffffff8105f772>] hrtimer_interrupt+0xae/0x192
 [<ffffffff8100f3a3>] xen_timer_interrupt+0x37/0x181
 [<ffffffff81082898>] ? check_for_new_grace_period+0x97/0xa5
 [<ffffffff811c870f>] ? unmask_evtchn+0x34/0xd6
 [<ffffffff8108318c>] ? __rcu_process_callbacks+0xf2/0x2ae
 [<ffffffff8107f708>] handle_IRQ_event+0x2d/0xb7
 [<ffffffff81081079>] handle_percpu_irq+0x3c/0x69
 [<ffffffff811c8640>] __xen_evtchn_do_upcall+0xe1/0x168
 [<ffffffff811c92d1>] xen_evtchn_do_upcall+0x2e/0x41
 [<ffffffff81013c7e>] xen_do_hypervisor_callback+0x1e/0x30
 <EOI>  [<ffffffff813199d3>] ? _spin_lock+0x19/0x20
 [<ffffffff811660dd>] ? free_cpumask_var+0x9/0xb
 [<ffffffff8100dde1>] ? xen_exit_mmap+0x199/0x1d7
 [<ffffffff810a8137>] ? exit_mmap+0x5f/0x14b
 [<ffffffff81048648>] ? mmput+0x46/0xb2
 [<ffffffff8104c552>] ? exit_mm+0xfd/0x108
 [<ffffffff8100f799>] ? xen_irq_enable_direct_end+0x0/0x7
 [<ffffffff8104d7ee>] ? do_exit+0x1f3/0x67b
 [<ffffffff8131a908>] ? oops_end+0xba/0xc2
 [<ffffffff810163a1>] ? die+0x55/0x5e
 [<ffffffff8131a192>] ? do_trap+0x110/0x11f
 [<ffffffff810142c8>] ? do_invalid_op+0x97/0xa0
 [<ffffffff8100cb5b>] ? pin_pagetable_pfn+0x53/0x59
 [<ffffffff810138bb>] ? invalid_op+0x1b/0x20
 [<ffffffff8100cb5b>] ? pin_pagetable_pfn+0x53/0x59
 [<ffffffff8100cb57>] ? pin_pagetable_pfn+0x4f/0x59
 [<ffffffff8100e07c>] ? xen_alloc_ptpage+0x64/0x69
 [<ffffffff8100e0af>] ? xen_alloc_pte+0xe/0x10
 [<ffffffff810a402f>] ? __pte_alloc+0x70/0xce
 [<ffffffff810a41cd>] ? handle_mm_fault+0x140/0x8b9
 [<ffffffff8131be4d>] ? do_page_fault+0x252/0x2e2
 [<ffffffff81319dd5>] ? page_fault+0x25/0x30
(XEN) Domain 0 crashed: rebooting machine in 5 seconds.


Thanks.

Kindest regards,
Giam Teck Choon

[-- Attachment #1.2: Type: text/html, Size: 10087 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: kernel BUG at arch/x86/xen/mmu.c:1860!

[Konrad Rzeszutek Wilk]

> The idea is to create snapshot, mount it, umount it, remove snapshot and
> repeat this cycle in loop will catch this BUG!!!

What is the hypervisor version? Is it xen-unstable.hg?

Re: kernel BUG at arch/x86/xen/mmu.c:1860!

[Teck Choon Giam]

[-- Attachment #1.1: Type: text/plain, Size: 403 bytes --]

On Fri, Jan 14, 2011 at 11:24 PM, Konrad Rzeszutek Wilk <
konrad.wilk@oracle.com> wrote:

> > The idea is to create snapshot, mount it, umount it, remove snapshot and
> > repeat this cycle in loop will catch this BUG!!!
>
> What is the hypervisor version? Is it xen-unstable.hg?
>

xen-4.0-testing changeset 214xx even with 21436

PVOPS latest stable/2.6.32.x

Thanks.

Kindest regards,
Giam Teck Choon

[-- Attachment #1.2: Type: text/html, Size: 744 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: kernel BUG at arch/x86/xen/mmu.c:1860!

[Ian Campbell]

On Sun, 2010-12-26 at 08:16 +0000, Teck Choon Giam wrote:
> 
> Triggered BUG() in line 1860:
> 
> static void pin_pagetable_pfn(unsigned cmd, unsigned long pfn)
> {
>         struct mmuext_op op;
>         op.cmd = cmd;
>         op.arg1.mfn = pfn_to_mfn(pfn);
>            if (HYPERVISOR_mmuext_op(&op, 1, NULL, DOMID_SELF))
>                    BUG(); <<THIS ONE?
> } 

A failure to pin/unpin is usually associated with a log message from the
hypervisor. Please can you attempt to capture the full host log, e.g.
using serial console.

See http://wiki.xen.org/xenwiki/XenParavirtOps under "Are there more
debugging options I could enable to troubleshoot booting problems?" for
some details.

Ian.

Re: kernel BUG at arch/x86/xen/mmu.c:1860!

[Teck Choon Giam]

[-- Attachment #1.1: Type: text/plain, Size: 962 bytes --]

On Tue, Jan 4, 2011 at 9:48 PM, Ian Campbell <Ian.Campbell@citrix.com>wrote:

> On Sun, 2010-12-26 at 08:16 +0000, Teck Choon Giam wrote:
> >
> > Triggered BUG() in line 1860:
> >
> > static void pin_pagetable_pfn(unsigned cmd, unsigned long pfn)
> > {
> >         struct mmuext_op op;
> >         op.cmd = cmd;
> >         op.arg1.mfn = pfn_to_mfn(pfn);
> >            if (HYPERVISOR_mmuext_op(&op, 1, NULL, DOMID_SELF))
> >                    BUG(); <<THIS ONE?
> > }
>
> A failure to pin/unpin is usually associated with a log message from the
> hypervisor. Please can you attempt to capture the full host log, e.g.
> using serial console.
>
> See http://wiki.xen.org/xenwiki/XenParavirtOps under "Are there more
> debugging options I could enable to troubleshoot booting problems?" for
> some details.
>

I will once I got the serial console cable and have a system to catch the
log during my next visit to the DC.

Thanks.

Kindest regards,
Giam Teck Choon

[-- Attachment #1.2: Type: text/html, Size: 1462 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: kernel BUG at arch/x86/xen/mmu.c:1860!

[Teck Choon Giam]

[-- Attachment #1.1: Type: text/plain, Size: 11155 bytes --]

On Wed, Jan 5, 2011 at 3:24 AM, Teck Choon Giam <giamteckchoon@gmail.com>wrote:

>
>
> On Tue, Jan 4, 2011 at 9:48 PM, Ian Campbell <Ian.Campbell@citrix.com>wrote:
>
>> On Sun, 2010-12-26 at 08:16 +0000, Teck Choon Giam wrote:
>> >
>> > Triggered BUG() in line 1860:
>> >
>> > static void pin_pagetable_pfn(unsigned cmd, unsigned long pfn)
>> > {
>> >         struct mmuext_op op;
>> >         op.cmd = cmd;
>> >         op.arg1.mfn = pfn_to_mfn(pfn);
>> >            if (HYPERVISOR_mmuext_op(&op, 1, NULL, DOMID_SELF))
>> >                    BUG(); <<THIS ONE?
>> > }
>>
>> A failure to pin/unpin is usually associated with a log message from the
>> hypervisor. Please can you attempt to capture the full host log, e.g.
>> using serial console.
>>
>> See http://wiki.xen.org/xenwiki/XenParavirtOps under "Are there more
>> debugging options I could enable to troubleshoot booting problems?" for
>> some details.
>>
>
> I will once I got the serial console cable and have a system to catch the
> log during my next visit to the DC.
>
>
Hi Ian,

Here is another console output besides the other one which I posted:

EXT3-fs warning: maximal mount count reached, running e2fsck is
recommended
EXT3-fs warning: maximal mount count reached, running e2fsck is
recommended
EXT3-fs warning: maximal mount count reached, running e2fsck is
recommended
EXT3-fs warning: maximal mount count reached, running e2fsck is
recommended
EXT3-fs warning: maximal mount count reached, running e2fsck is
recommended
EXT3-fs warning: maximal mount count reached, running e2fsck is
recommended
EXT3-fs warning: maximal mount count reached, running e2fsck is
recommended
EXT3-fs warning: maximal mount count reached, running e2fsck is
recommended
INIT: Id "s1" respawning too fast: disabled for 5
minutes
EXT3-fs warning: maximal mount count reached, running e2fsck is
recommended
EXT3-fs warning: maximal mount count reached, running e2fsck is
recommended
EXT3-fs warning: maximal mount count reached, running e2fsck is
recommended
EXT3-fs warning: maximal mount count reached, running e2fsck is
recommended
EXT3-fs warning: maximal mount count reached, running e2fsck is recommended
EXT3-fs warning: maximal mount count reached, running e2fsck is recommended
EXT3-fs warning: maximal mount count reached, running e2fsck is recommended
EXT3-fs warning: maximal mount count reached, running e2fsck is recommended
EXT3-fs warning: maximal mount count reached, running e2fsck is recommended
EXT3-fs warning: maximal mount count reached, running e2fsck is recommended
EXT3-fs warning: maximal mount count reached, running e2fsck is recommended
EXT3-fs warning: maximal mount count reached, running e2fsck is recommended
EXT3-fs warning: maximal mount count reached, running e2fsck is recommended
EXT3-fs warning: maximal mount count reached, running e2fsck is recommended
EXT3-fs warning: maximal mount count reached, running e2fsck is recommended
EXT3-fs warning: maximal mount count reached, running e2fsck is recommended
EXT3-fs warning: maximal mount count reached, running e2fsck is recommended
EXT3-fs warning: maximal mount count reached, running e2fsck is recommended
EXT3-fs warning: maximal mount count reached, running e2fsck is recommended
EXT3-fs warning: maximal mount count reached, running e2fsck is recommended
EXT3-fs warning: maximal mount count reached, running e2fsck is recommended
EXT3-fs warning: maximal mount count reached, running e2fsck is recommended
EXT3-fs warning: maximal mount count reached, running e2fsck is recommended
EXT3-fs warning: maximal mount count reached, running e2fsck is recommended
hrtimer: interrupt took 3096797 ns
EXT3-fs warning: maximal mount count reached, running e2fsck is recommended
EXT3-fs warning: maximal mount count reached, running e2fsck is recommended
EXT3-fs warning: maximal mount count reached, running e2fsck is recommended
EXT3-fs warning: maximal mount count reached, running e2fsck is recommended
EXT3-fs warning: maximal mount count reached, running e2fsck is recommended
EXT3-fs warning: maximal mount count reached, running e2fsck is recommended
EXT3-fs warning: maximal mount count reached, running e2fsck is recommended
EXT3-fs warning: maximal mount count reached, running e2fsck is recommended
EXT3-fs warning: maximal mount count reached, running e2fsck is recommended
EXT3-fs warning: maximal mount count reached, running e2fsck is recommended
EXT3-fs warning: maximal mount count reached, running e2fsck is recommended
EXT3-fs warning: maximal mount count reached, running e2fsck is recommended
(XEN) mm.c:2364:d0 Bad type (saw 7400000000000001 != exp 1000000000000000)
for mfn 1ee744 (pfn 1c399)
(XEN) mm.c:2733:d0 Error while pinning mfn 1ee744
------------[ cut here ]------------
kernel BUG at arch/x86/xen/mmu.c:1860!
invalid opcode: 0000 [#1] SMP
last sysfs file: /sys/block/dm-17/dev
CPU 1
Modules linked in: ext4 jbd2 crc16 gfs2 dlm configfs xt_physdev
iptable_filter ip_tables x_tables bridge stp be2iscsi iscsi_]
Pid: 19758, comm: dmsetup Not tainted 2.6.32.27-0.xen.pvops.choon.centos5 #1
PowerEdge 860
RIP: e030:[<ffffffff8100cb5b>]  [<ffffffff8100cb5b>]
pin_pagetable_pfn+0x53/0x59
RSP: e02b:ffff88003a615dc8  EFLAGS: 00010282
RAX: 00000000ffffffea RBX: 000000000001c399 RCX: 00000000000000e1
RDX: 00000000deadbeef RSI: 00000000deadbeef RDI: 00000000deadbeef
RBP: ffff88003a615de8 R08: 0000000000000cc8 R09: ffff880000000000
R10: 00000000deadbeef R11: 0000000000000246 R12: 0000000000000003
R13: 000000000001c399 R14: ffff88001c1d86c0 R15: 0000003db7400258
FS:  00007f6cbd5b6710(0000) GS:ffff88002806c000(0000) knlGS:0000000000000000
CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000003db7400258 CR3: 000000003a5ed000 CR4: 0000000000002660
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process dmsetup (pid: 19758, threadinfo ffff88003a614000, task
ffff88001c1d86c0)
Stack:
 0000000000000000 00000000001ee744 000000013e64d518 000000000001c399
<0> ffff88003a615e08 ffffffff8100e07c ffff880027a2c580 ffff88003a56fdd0
<0> ffff88003a615e18 ffffffff8100e0af ffff88003a615e58 ffffffff810a402f
Call Trace:
 [<ffffffff8100e07c>] xen_alloc_ptpage+0x64/0x69
 [<ffffffff8100e0af>] xen_alloc_pte+0xe/0x10
 [<ffffffff810a402f>] __pte_alloc+0x70/0xce
 [<ffffffff810a41cd>] handle_mm_fault+0x140/0x8b9
 [<ffffffff8131be4d>] do_page_fault+0x252/0x2e2
 [<ffffffff81319dd5>] page_fault+0x25/0x30
Code: 48 b8 ff ff ff ff ff ff ff 7f 48 21 c2 48 89 55 e8 48 8d 7d e0 be 01
00 00 00 31 d2 41 ba f0 7f 00 00 e8 e9 c7 ff ff 8
RIP  [<ffffffff8100cb5b>] pin_pagetable_pfn+0x53/0x59
 RSP <ffff88003a615dc8>
---[ end trace 63676fea977b3461 ]---
BUG: soft lockup - CPU#1 stuck for 61s! [dmsetup:19758]
Modules linked in: ext4 jbd2 crc16 gfs2 dlm configfs xt_physdev
iptable_filter ip_tables x_tables bridge stp be2iscsi iscsi_]
CPU 1:
Modules linked in: ext4 jbd2 crc16 gfs2 dlm configfs xt_physdev
iptable_filter ip_tables x_tables bridge stp be2iscsi iscsi_]
Pid: 19758, comm: dmsetup Tainted: G      D
2.6.32.27-0.xen.pvops.choon.centos5 #1 PowerEdge 860
RIP: e030:[<ffffffff813199d3>]  [<ffffffff813199d3>] _spin_lock+0x19/0x20
RSP: e02b:ffff88003a615a68  EFLAGS: 00000297
RAX: 0000000000000025 RBX: 000000003d2fd000 RCX: 0000000000000004
RDX: 0000000000000024 RSI: 0000000000000004 RDI: ffff880027a2c600
RBP: ffff88003a615a68 R08: 0000000000000000 R09: ffffffff816dd100
R10: 3030303030303030 R11: 0000000000000120 R12: ffff880027a2c580
R13: 0000000000000004 R14: ffff880027a2c5e0 R15: ffffffff816dd100
FS:  00007f720a0f36e0(0000) GS:ffff88002806c000(0000) knlGS:0000000000000000
CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007f7209c9c898 CR3: 0000000001001000 CR4: 0000000000002660
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Call Trace:
 [<ffffffff811660dd>] ? free_cpumask_var+0x9/0xb
 [<ffffffff8100dde1>] xen_exit_mmap+0x199/0x1d7
 [<ffffffff810a8137>] exit_mmap+0x5f/0x14b
 [<ffffffff81048648>] mmput+0x46/0xb2
 [<ffffffff8104c552>] exit_mm+0xfd/0x108
 [<ffffffff8100f799>] ? xen_irq_enable_direct_end+0x0/0x7
 [<ffffffff8104d7ee>] do_exit+0x1f3/0x67b
 [<ffffffff8131a908>] oops_end+0xba/0xc2
 [<ffffffff810163a1>] die+0x55/0x5e
 [<ffffffff8131a192>] do_trap+0x110/0x11f
 [<ffffffff810142c8>] do_invalid_op+0x97/0xa0
 [<ffffffff8100cb5b>] ? pin_pagetable_pfn+0x53/0x59
 [<ffffffff810138bb>] invalid_op+0x1b/0x20
 [<ffffffff8100cb5b>] ? pin_pagetable_pfn+0x53/0x59
 [<ffffffff8100cb57>] ? pin_pagetable_pfn+0x4f/0x59
 [<ffffffff8100e07c>] xen_alloc_ptpage+0x64/0x69
 [<ffffffff8100e0af>] xen_alloc_pte+0xe/0x10
 [<ffffffff810a402f>] __pte_alloc+0x70/0xce
 [<ffffffff810a41cd>] handle_mm_fault+0x140/0x8b9
 [<ffffffff8131be4d>] do_page_fault+0x252/0x2e2
 [<ffffffff81319dd5>] page_fault+0x25/0x30
Kernel panic - not syncing: softlockup: hung tasks
Pid: 19758, comm: dmsetup Tainted: G      D
2.6.32.27-0.xen.pvops.choon.centos5 #1
Call Trace:
 <IRQ>  [<ffffffff8104aa97>] panic+0xa0/0x15f
 [<ffffffff81319dd5>] ? page_fault+0x25/0x30
 [<ffffffff8101640f>] ? show_trace_log_lvl+0x4c/0x58
 [<ffffffff8101642b>] ? show_trace+0x10/0x12
 [<ffffffff81011755>] ? show_regs+0x44/0x48
 [<ffffffff8107f202>] softlockup_tick+0x173/0x182
 [<ffffffff810539bf>] run_local_timers+0x18/0x1a
 [<ffffffff81053bde>] update_process_times+0x30/0x54
 [<ffffffff81068821>] tick_sched_timer+0x70/0x99
 [<ffffffff8105f52e>] __run_hrtimer+0x53/0xb3
 [<ffffffff8105f772>] hrtimer_interrupt+0xae/0x192
 [<ffffffff8100f3a3>] xen_timer_interrupt+0x37/0x181
 [<ffffffff81082898>] ? check_for_new_grace_period+0x97/0xa5
 [<ffffffff811c870f>] ? unmask_evtchn+0x34/0xd6
 [<ffffffff8108318c>] ? __rcu_process_callbacks+0xf2/0x2ae
 [<ffffffff8107f708>] handle_IRQ_event+0x2d/0xb7
 [<ffffffff81081079>] handle_percpu_irq+0x3c/0x69
 [<ffffffff811c8640>] __xen_evtchn_do_upcall+0xe1/0x168
 [<ffffffff811c92d1>] xen_evtchn_do_upcall+0x2e/0x41
 [<ffffffff81013c7e>] xen_do_hypervisor_callback+0x1e/0x30
 <EOI>  [<ffffffff813199d3>] ? _spin_lock+0x19/0x20
 [<ffffffff811660dd>] ? free_cpumask_var+0x9/0xb
 [<ffffffff8100dde1>] ? xen_exit_mmap+0x199/0x1d7
 [<ffffffff810a8137>] ? exit_mmap+0x5f/0x14b
 [<ffffffff81048648>] ? mmput+0x46/0xb2
 [<ffffffff8104c552>] ? exit_mm+0xfd/0x108
 [<ffffffff8100f799>] ? xen_irq_enable_direct_end+0x0/0x7
 [<ffffffff8104d7ee>] ? do_exit+0x1f3/0x67b
 [<ffffffff8131a908>] ? oops_end+0xba/0xc2
 [<ffffffff810163a1>] ? die+0x55/0x5e
 [<ffffffff8131a192>] ? do_trap+0x110/0x11f
 [<ffffffff810142c8>] ? do_invalid_op+0x97/0xa0
 [<ffffffff8100cb5b>] ? pin_pagetable_pfn+0x53/0x59
 [<ffffffff810138bb>] ? invalid_op+0x1b/0x20
 [<ffffffff8100cb5b>] ? pin_pagetable_pfn+0x53/0x59
 [<ffffffff8100cb57>] ? pin_pagetable_pfn+0x4f/0x59
 [<ffffffff8100e07c>] ? xen_alloc_ptpage+0x64/0x69
 [<ffffffff8100e0af>] ? xen_alloc_pte+0xe/0x10
 [<ffffffff810a402f>] ? __pte_alloc+0x70/0xce
 [<ffffffff810a41cd>] ? handle_mm_fault+0x140/0x8b9
 [<ffffffff8131be4d>] ? do_page_fault+0x252/0x2e2
 [<ffffffff81319dd5>] ? page_fault+0x25/0x30
(XEN) Domain 0 crashed: rebooting machine in 5 seconds.

Thanks.

Kindest regards,
Giam Teck Choon

[-- Attachment #1.2: Type: text/html, Size: 13622 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: kernel BUG at arch/x86/xen/mmu.c:1860!

[tjaouen]

Same issue:

Backup DomU with lvsnapshot ... 

Linux xencave 2.6.32-5-xen-amd64 #1 SMP Fri Dec 10 17:41:50 UTC 2010 x86_64
GNU/Linux

Jan 13 04:01:51 xencave kernel: [223872.758977] ------------[ cut here
]------------
Jan 13 04:01:51 xencave kernel: [223872.759006] kernel BUG at
/build/buildd-linux-2.6_2.6.32-29-amd64-xcs37n/linux-2.6-2.6.32/debian/build/source_amd64_xen/arch/x86/xen/mmu.c:1649!
Jan 13 04:01:51 xencave kernel: [223872.759047] invalid opcode: 0000 [#2]
SMP 
Jan 13 04:01:51 xencave kernel: [223872.759076] last sysfs file:
/sys/devices/virtual/block/dm-30/dm/suspended
Jan 13 04:01:51 xencave kernel: [223872.759099] CPU 1 
Jan 13 04:01:51 xencave kernel: [223872.759121] Modules linked in:
dm_snapshot nfs lockd fscache nfs_acl auth_rpcgss sunrpc nf_conntrack_ipv4
nf_defrag_ipv4 xt_state nf_conntrack xt_physdev iptable_filter ip_tables
x_tables xen_evtchn xenfs bridge stp coretemp it87 hwmon_vid fuse loop i915
drm_kms_helper processor acpi_processor drm i2c_algo_bit button video
snd_pcsp rng_core evdev i2c_i801 serio_raw snd_pcm snd_timer output i2c_core
snd soundcore snd_page_alloc usbhid hid ext3 jbd mbcache dm_mod raid1 md_mod
sg sr_mod cdrom sd_mod crc_t10dif ata_generic ata_piix libata scsi_mod
ehci_hcd uhci_hcd r8169 mii usbcore nls_base thermal fan thermal_sys [last
unloaded: scsi_wait_scan]
Jan 13 04:01:51 xencave kernel: [223872.759599] Pid: 9244, comm: dmsetup_env
Tainted: G      D    2.6.32-5-xen-amd64 #1 945GM-S2
Jan 13 04:01:51 xencave kernel: [223872.759636] RIP:
e030:[<ffffffff8100c694>]  [<ffffffff8100c694>] pin_pagetable_pfn+0x2d/0x36
Jan 13 04:01:51 xencave kernel: [223872.759683] RSP: e02b:ffff88001d695e08 
EFLAGS: 00010282
Jan 13 04:01:51 xencave kernel: [223872.759707] RAX: 00000000ffffffea RBX:
000000000001e9df RCX: 0000000000000001
Jan 13 04:01:51 xencave kernel: [223872.759742] RDX: 0000000000000000 RSI:
0000000000000001 RDI: ffff88001d695e08
Jan 13 04:01:51 xencave kernel: [223872.759777] RBP: ffff88001f2b4700 R08:
0000000000000ef8 R09: ffffea00006b28c8
Jan 13 04:01:51 xencave kernel: [223872.759812] R10: 0000000000007ff0 R11:
ffffea00005858c8 R12: ffff88001a3ee010
Jan 13 04:01:51 xencave kernel: [223872.759846] R13: ffff8800020a8000 R14:
ffff880002089530 R15: ffff88001a3ee010
Jan 13 04:01:51 xencave kernel: [223872.759887] FS:  00007f81d5709700(0000)
GS:ffff88000308c000(0000) knlGS:0000000000000000
Jan 13 04:01:51 xencave kernel: [223872.759925] CS:  e033 DS: 0000 ES: 0000
CR0: 000000008005003b
Jan 13 04:01:51 xencave kernel: [223872.759948] CR2: 000000000043c56a CR3:
000000001a38c000 CR4: 0000000000002660
Jan 13 04:01:51 xencave kernel: [223872.759985] DR0: 0000000000000000 DR1:
0000000000000000 DR2: 0000000000000000
Jan 13 04:01:51 xencave kernel: [223872.760022] DR3: 0000000000000000 DR6:
00000000ffff0ff0 DR7: 0000000000000400
Jan 13 04:01:51 xencave kernel: [223872.760057] Process dmsetup_env (pid:
9244, threadinfo ffff88001d694000, task ffff880002089530)
Jan 13 04:01:51 xencave kernel: [223872.760094] Stack:
Jan 13 04:01:51 xencave kernel: [223872.760112]  0000000000000000
000000000009e90d ffffea00006b28c8 000000000001e9df
Jan 13 04:01:51 xencave kernel: [223872.760149] <0> ffff88001f2b4700
ffffffff810cd4a1 ffff88001a3dc000 000000000043c56a
Jan 13 04:01:51 xencave kernel: [223872.760203] <0> 000000001a3dc000
ffffffff810cb34c ffff8800020a8000 000000000043c56a
Jan 13 04:01:51 xencave kernel: [223872.760270] Call Trace:
Jan 13 04:01:51 xencave kernel: [223872.760295]  [<ffffffff810cd4a1>] ?
__pte_alloc+0x6b/0xc6
Jan 13 04:01:51 xencave kernel: [223872.760320]  [<ffffffff810cb34c>] ?
pmd_alloc+0x28/0x5b
Jan 13 04:01:51 xencave kernel: [223872.760344]  [<ffffffff810cd5ca>] ?
handle_mm_fault+0xce/0x80f
Jan 13 04:01:51 xencave kernel: [223872.760372]  [<ffffffff8130cc35>] ?
page_fault+0x25/0x30
Jan 13 04:01:51 xencave kernel: [223872.760397]  [<ffffffff8130ed96>] ?
do_page_fault+0x2e0/0x2fc
Jan 13 04:01:51 xencave kernel: [223872.760422]  [<ffffffff8130cc35>] ?
page_fault+0x25/0x30
Jan 13 04:01:51 xencave kernel: [223872.760443] Code: ec 28 89 3c 24 48 89
f7 e8 a2 fd ff ff 48 89 e7 48 89 44 24 08 be 01 00 00 00 31 d2 41 ba f0 7f
00 00 e8 b0 cc ff ff 85 c0 74 04 <0f> 0b eb fe 48 83 c4 28 c3 55 49 89 ca 48
89 d5 40 88 f1 48 89 
Jan 13 04:01:51 xencave kernel: [223872.760761] RIP  [<ffffffff8100c694>]
pin_pagetable_pfn+0x2d/0x36
Jan 13 04:01:51 xencave kernel: [223872.760790]  RSP <ffff88001d695e08>
Jan 13 04:01:51 xencave kernel: [223872.761008] ---[ end trace
b143b5bdb343412e ]---


help!
-- 
View this message in context: http://xen.1045712.n5.nabble.com/kernel-BUG-at-arch-x86-xen-mmu-c-1860-tp3318567p3339844.html
Sent from the Xen - Dev mailing list archive at Nabble.com.