* pvops domu soft lockup under load
@ 2010-04-09 13:22 Pim van Riezen
2010-04-09 16:50 ` pvops domu soft lockup under load (more logs) Pim van Riezen
0 siblings, 1 reply; 17+ messages in thread
From: Pim van Riezen @ 2010-04-09 13:22 UTC (permalink / raw)
To: xen-devel
Good day,
I'm getting lots of these from a couple of pvops DomU guests that are pretty heavy in their i/o and cpu-load. The issue reproduces on pvops 2.6.31 and 2.6.32 trees under Xen 3.3 as well as 3.4.1 (in the process of trying out 3.4.2 but I'm not holding my breath). Is there anything we can do be more helpful towards solving this issue?
BUG: soft lockup - CPU#0 stuck for 561s! [swapper:0]
Modules linked in:
CPU 0:
Modules linked in:
Pid: 0, comm: swapper Not tainted 2.6.32.9xls-domU #2
RIP: e030:[<ffffffff810093aa>] [<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
RSP: e02b:ffffffff81691f70 EFLAGS: 00000246
RAX: 0000000000000000 RBX: ffffffff81690000 RCX: ffffffff810093aa
RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000001
RBP: ffffffff81896d30 R08: 0000000000000000 R09: ffffffff8100e3b2
R10: 0000000000000001 R11: 0000000000000246 R12: ffffffffffffffff
R13: ffffffff818ebf20 R14: ffffffff818eec70 R15: 0000000000000000
FS: 00007f8ac7a9c6e0(0000) GS:ffff8800022ac000(0000) knlGS:0000000000000000
CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007f291b4c9000 CR3: 000000007d8c1000 CR4: 0000000000002660
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff8100ddb7>] ? xen_safe_halt+0xc/0x15
[<ffffffff8100bdcf>] ? xen_idle+0x37/0x40
[<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82
[<ffffffff818b6c42>] ? start_kernel+0x353/0x35f
Kind regards,
Pim van Riezen
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: pvops domu soft lockup under load (more logs)
2010-04-09 13:22 pvops domu soft lockup under load Pim van Riezen
@ 2010-04-09 16:50 ` Pim van Riezen
2010-04-09 18:43 ` Jeremy Fitzhardinge
0 siblings, 1 reply; 17+ messages in thread
From: Pim van Riezen @ 2010-04-09 16:50 UTC (permalink / raw)
To: xen-devel
Some more logs of this bug in action. Stack seems to be all over the place.
> Apr 9 08:21:01 host kernel: BUG: soft lockup - CPU#1 stuck for 628s! [gzip:27658]
> Apr 9 08:21:01 host kernel: Modules linked in:
> Apr 9 08:21:01 host kernel: CPU 1:
> Apr 9 08:21:01 host kernel: Modules linked in:
> Apr 9 08:21:01 host kernel: Pid: 27658, comm: gzip Not tainted 2.6.32.9xls-domU #2
> Apr 9 08:21:01 host kernel: RIP: e033:[<0000000000401958>] [<0000000000401958>] 0x401958
> Apr 9 08:21:01 host kernel: RSP: e02b:00007fff026d7e38 EFLAGS: 00000297
> Apr 9 08:21:01 host kernel: RAX: 0000000000000020 RBX: 0000000000000002 RCX: 0000000000657ce8
> Apr 9 08:21:01 host kernel: RDX: 0000000000000048 RSI: 0000000000000002 RDI: 000000000000da48
> Apr 9 08:21:01 host kernel: RBP: 0000000000006a6c R08: 0000000000000034 R09: 000000000000da48
> Apr 9 08:21:01 host kernel: R10: 0000000000000048 R11: 0000000000000080 R12: 0000000000658c06
> Apr 9 08:21:01 host kernel: R13: 0000000000658d08 R14: 0000000000658c06 R15: 0000000000000080
> Apr 9 08:21:01 host kernel: FS: 00007fa6088946e0(0000) GS:ffff8800022c7000(0000) knlGS:0000000000000000
> Apr 9 08:21:01 host kernel: CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b
> Apr 9 08:21:01 host kernel: CR2: 00007f7addede020 CR3: 00000000267db000 CR4: 0000000000002660
> Apr 9 08:21:01 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> Apr 9 08:21:02 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> Apr 9 08:21:02 host kernel: Call Trace:
> Apr 9 08:29:39 host kernel: BUG: soft lockup - CPU#1 stuck for 483s! [swapper:0]
> Apr 9 08:29:39 host kernel: Modules linked in:
> Apr 9 08:29:39 host kernel: CPU 1:
> Apr 9 08:29:39 host kernel: Modules linked in:
> Apr 9 08:29:39 host kernel: Pid: 0, comm: swapper Not tainted 2.6.32.9xls-domU #2
> Apr 9 08:29:39 host kernel: RIP: e030:[<ffffffff810093aa>] [<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
> Apr 9 08:29:39 host kernel: RSP: e02b:ffff88007f8a1f08 EFLAGS: 00000246
> Apr 9 08:29:39 host kernel: RAX: 0000000000000000 RBX: ffff88007f8a0000 RCX: ffffffff810093aa
> Apr 9 08:29:39 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000001
> Apr 9 08:29:39 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000 R09: ffffffff8100e3b2
> Apr 9 08:29:39 host kernel: R10: 0000000000000001 R11: 0000000000000246 R12: 0000000000000000
> Apr 9 08:29:39 host kernel: R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
> Apr 9 08:29:39 host kernel: FS: 00007fa913f42720(0000) GS:ffff8800022c7000(0000) knlGS:0000000000000000
> Apr 9 08:29:39 host kernel: CS: e033 DS: 002b ES: 002b CR0: 000000008005003b
> Apr 9 08:29:39 host kernel: CR2: 00007ff39862b1f0 CR3: 0000000052c5e000 CR4: 0000000000002660
> Apr 9 08:29:39 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> Apr 9 08:29:39 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> Apr 9 08:29:39 host kernel: Call Trace:
> Apr 9 08:29:39 host kernel: [<ffffffff8100ddb7>] ? xen_safe_halt+0xc/0x15
> Apr 9 08:29:39 host kernel: [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40
> Apr 9 08:29:39 host kernel: [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82
> Apr 9 08:29:39 host kernel: BUG: soft lockup - CPU#0 stuck for 483s! [swapper:0]
> Apr 9 08:29:39 host kernel: Modules linked in:
> Apr 9 08:29:39 host kernel: CPU 0:
> Apr 9 08:29:40 host kernel: Modules linked in:
> Apr 9 08:29:40 host kernel: Pid: 0, comm: swapper Not tainted 2.6.32.9xls-domU #2
> Apr 9 08:29:40 host kernel: RIP: e030:[<ffffffff810093aa>] [<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
> Apr 9 08:29:40 host kernel: RSP: e02b:ffffffff81691f70 EFLAGS: 00000246
> Apr 9 08:29:40 host kernel: RAX: 0000000000000000 RBX: ffffffff81690000 RCX: ffffffff810093aa
> Apr 9 08:29:40 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000001
> Apr 9 08:29:40 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000 R09: ffffffff8100e3b2
> Apr 9 08:29:40 host kernel: R10: 0000000000000001 R11: 0000000000000246 R12: ffffffffffffffff
> Apr 9 08:29:40 host kernel: R13: ffffffff818ebf20 R14: ffffffff818eec70 R15: 0000000000000000
> Apr 9 08:29:40 host kernel: FS: 00007f07704fe6e0(0000) GS:ffff8800022ac000(0000) knlGS:0000000000000000
> Apr 9 08:29:40 host kernel: CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b
> Apr 9 08:29:40 host kernel: CR2: 00007ff39e4a8000 CR3: 000000007ea99000 CR4: 0000000000002660
> Apr 9 08:29:40 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> Apr 9 08:29:40 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> Apr 9 08:29:40 host kernel: Call Trace:
> Apr 9 08:29:40 host kernel: [<ffffffff8100ddb7>] ? xen_safe_halt+0xc/0x15
> Apr 9 08:29:40 host kernel: [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40
> Apr 9 08:29:40 host kernel: [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82
> Apr 9 08:29:40 host kernel: [<ffffffff818b6c42>] ? start_kernel+0x353/0x35f
> Apr 9 08:34:34 host kernel: BUG: soft lockup - CPU#1 stuck for 275s! [ssh:2324]
> Apr 9 08:34:34 host kernel: Modules linked in:
> Apr 9 08:34:34 host kernel: CPU 1:
> Apr 9 08:34:34 host kernel: Modules linked in:
> Apr 9 08:34:34 host kernel: Pid: 2324, comm: ssh Not tainted 2.6.32.9xls-domU #2
> Apr 9 08:34:34 host kernel: RIP: e033:[<00007f1b0fcac76a>] [<00007f1b0fcac76a>] 0x7f1b0fcac76a
> Apr 9 08:34:34 host kernel: RSP: e02b:00007fffa45a7480 EFLAGS: 00010202
> Apr 9 08:34:34 host kernel: RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000181
> Apr 9 08:34:34 host kernel: RDX: 00007f1b0fcdca28 RSI: 0000000000000013 RDI: 000000000000000a
> Apr 9 08:34:34 host kernel: RBP: 0000000000000010 R08: 0000000000000004 R09: 0101010101010101
> Apr 9 08:34:34 host kernel: R10: 737361702e707466 R11: 00007f1b0fcac690 R12: 0000000000000010
> Apr 9 08:34:34 host kernel: R13: 00007f1b11f825e0 R14: 0000000000000001 R15: 0000000000000004
> Apr 9 08:34:34 host kernel: FS: 00007f1b10528710(0000) GS:ffff8800022c7000(0000) knlGS:0000000000000000
> Apr 9 08:34:34 host kernel: CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b
> Apr 9 08:34:34 host kernel: CR2: 00007fa913f54000 CR3: 000000006ad44000 CR4: 0000000000002660
> Apr 9 08:34:34 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> Apr 9 08:34:34 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> Apr 9 08:34:34 host kernel: Call Trace:
> Apr 9 08:34:34 host kernel: BUG: soft lockup - CPU#0 stuck for 275s! [swapper:0]
> Apr 9 08:34:34 host kernel: Modules linked in:
> Apr 9 08:34:34 host kernel: CPU 0:
> Apr 9 08:34:34 host kernel: Modules linked in:
> Apr 9 08:34:34 host kernel: Pid: 0, comm: swapper Not tainted 2.6.32.9xls-domU #2
> Apr 9 08:34:34 host kernel: RIP: e030:[<ffffffff810093aa>] [<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
> Apr 9 08:34:34 host kernel: RSP: e02b:ffffffff81691f70 EFLAGS: 00000246
> Apr 9 08:34:34 host kernel: RAX: 0000000000000000 RBX: ffffffff81690000 RCX: ffffffff810093aa
> Apr 9 08:34:34 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000001
> Apr 9 08:34:34 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000 R09: ffffffff8100e3b2
> Apr 9 08:34:34 host kernel: R10: 0000000000000001 R11: 0000000000000246 R12: ffffffffffffffff
> Apr 9 08:34:34 host kernel: R13: ffffffff818ebf20 R14: ffffffff818eec70 R15: 0000000000000000
> Apr 9 08:34:34 host kernel: FS: 00007fa913f42720(0000) GS:ffff8800022ac000(0000) knlGS:0000000000000000
> Apr 9 08:34:34 host kernel: CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b
> Apr 9 08:34:34 host kernel: CR2: 00007fa913f54000 CR3: 000000006acb1000 CR4: 0000000000002660
> Apr 9 08:34:34 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> Apr 9 08:34:34 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> Apr 9 08:34:34 host kernel: Call Trace:
> Apr 9 08:34:34 host kernel: [<ffffffff8100ddb7>] ? xen_safe_halt+0xc/0x15
> Apr 9 08:34:34 host kernel: [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40
> Apr 9 08:34:34 host kernel: [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82
> Apr 9 08:34:34 host kernel: [<ffffffff818b6c42>] ? start_kernel+0x353/0x35f
> Apr 9 08:39:49 host kernel: BUG: soft lockup - CPU#1 stuck for 293s! [gzip:30581]
> Apr 9 08:39:49 host kernel: Modules linked in:
> Apr 9 08:39:49 host kernel: CPU 1:
> Apr 9 08:39:49 host kernel: Modules linked in:
> Apr 9 08:39:49 host kernel: Pid: 30581, comm: gzip Not tainted 2.6.32.9xls-domU #2
> Apr 9 08:39:49 host kernel: RIP: e030:[<ffffffff811ede38>] [<ffffffff811ede38>] fuse_request_send+0x60/0x22b
> Apr 9 08:39:49 host kernel: RSP: e02b:ffff88007952bc28 EFLAGS: 00000202
> Apr 9 08:39:49 host kernel: RAX: ffff880079652df0 RBX: ffff88007e07d2c0 RCX: fffffffffffdfb5e
> Apr 9 08:39:49 host kernel: RDX: 0000000000020001 RSI: 000000000000001d RDI: ffff88007eb1f710
> Apr 9 08:39:49 host kernel: RBP: ffff88007e07d2c0 R08: 000000037ede1c40 R09: ffff8800593170d8
> Apr 9 08:39:49 host kernel: R10: 0000000000000000 R11: 000000000000000f R12: ffff88007eb1f400
> Apr 9 08:39:49 host kernel: R13: ffff88007e07d2e0 R14: 0000000000001000 R15: 0000000000001000
> Apr 9 08:39:49 host kernel: FS: 00007f7b12f606e0(0000) GS:ffff8800022c7000(0000) knlGS:0000000000000000
> Apr 9 08:39:49 host kernel: CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b
> Apr 9 08:39:49 host kernel: CR2: 00000000006d9138 CR3: 00000000264f7000 CR4: 0000000000002660
> Apr 9 08:39:49 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> Apr 9 08:39:49 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> Apr 9 08:39:49 host kernel: Call Trace:
> Apr 9 08:39:49 host kernel: [<ffffffff811ede34>] ? fuse_request_send+0x5c/0x22b
> Apr 9 08:39:49 host kernel: [<ffffffff811f1655>] ? fuse_wait_on_page_writeback+0x31/0x168
> Apr 9 08:39:49 host kernel: [<ffffffff810904b4>] ? add_to_page_cache_lru+0x41/0x58
> Apr 9 08:39:49 host kernel: [<ffffffff811f186a>] ? fuse_send_write+0xb8/0xc2
> Apr 9 08:39:49 host kernel: [<ffffffff811f29d7>] ? fuse_file_aio_write+0x497/0x62f
> Apr 9 08:39:49 host kernel: [<ffffffff810bab7e>] ? do_sync_write+0xce/0x113
> Apr 9 08:39:49 host kernel: [<ffffffff8100f6af>] ? __switch_to+0x1ad/0x297
> Apr 9 08:39:49 host kernel: [<ffffffff8100e3b2>] ? check_events+0x12/0x20
> Apr 9 08:39:49 host kernel: [<ffffffff810561b8>] ? autoremove_wake_function+0x0/0x2e
> Apr 9 08:39:49 host kernel: [<ffffffff8103b070>] ? finish_task_switch+0x27/0x6e
> Apr 9 08:39:49 host kernel: [<ffffffff814d6a77>] ? thread_return+0x3e/0x109
> Apr 9 08:39:49 host kernel: [<ffffffff810bb4b2>] ? vfs_write+0xad/0x149
> Apr 9 08:39:49 host kernel: [<ffffffff810bb60a>] ? sys_write+0x45/0x6e
> Apr 9 08:39:49 host kernel: [<ffffffff81010ac2>] ? system_call_fastpath+0x16/0x1b
> Apr 9 08:39:49 host kernel: BUG: soft lockup - CPU#0 stuck for 293s! [sshfs:2332]
> Apr 9 08:39:49 host kernel: Modules linked in:
> Apr 9 08:39:49 host kernel: CPU 0:
> Apr 9 08:39:49 host kernel: Modules linked in:
> Apr 9 08:39:49 host kernel: Pid: 2332, comm: sshfs Not tainted 2.6.32.9xls-domU #2
> Apr 9 08:39:49 host kernel: RIP: e030:[<ffffffff813b93bd>] [<ffffffff813b93bd>] sk_reset_timer+0x1/0x1d
> Apr 9 08:39:49 host kernel: RSP: e02b:ffff8800022afbc0 EFLAGS: 00000206
> Apr 9 08:39:49 host kernel: RAX: 0000000101b312c6 RBX: 0000000000000080 RCX: 0000000101b312ce
> Apr 9 08:39:49 host kernel: RDX: 0000000101b312ce RSI: ffff88007d1dc4d0 RDI: ffff88007d1dc1c0
> Apr 9 08:39:49 host kernel: RBP: ffff88007d1dc1c0 R08: 000000007d1dc10f R09: 0000000101b312ce
> Apr 9 08:39:49 host kernel: R10: ffff880068b8f1c0 R11: ffffffff8126167a R12: ffff880068b8f1c0
> Apr 9 08:39:49 host kernel: R13: 0000000000000000 R14: 0000000000000020 R15: ffff880068b8f1f8
> Apr 9 08:39:49 host kernel: FS: 00007f1b10528710(0063) GS:ffff8800022ac000(0000) knlGS:0000000000000000
> Apr 9 08:39:49 host kernel: CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b
> Apr 9 08:39:49 host kernel: CR2: 00007f66a4ede000 CR3: 000000006acb1000 CR4: 0000000000002660
> Apr 9 08:39:49 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> Apr 9 08:39:49 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> Apr 9 08:39:49 host kernel: Call Trace:
> Apr 9 08:39:49 host kernel: <IRQ> [<ffffffff81419140>] ? tcp_rcv_established+0x587/0x87b
> Apr 9 08:39:49 host kernel: [<ffffffff8141f7f1>] ? tcp_v4_do_rcv+0x2c/0x1c9
> Apr 9 08:39:49 host kernel: [<ffffffff81046156>] ? local_bh_enable+0x22/0x8f
> Apr 9 08:39:49 host kernel: [<ffffffff8141fe05>] ? tcp_v4_rcv+0x477/0x6f8
> Apr 9 08:39:49 host kernel: [<ffffffff814064f7>] ? ip_local_deliver_finish+0xf2/0x1c7
> Apr 9 08:39:49 host kernel: [<ffffffff81406126>] ? ip_rcv_finish+0x326/0x340
> Apr 9 08:39:49 host kernel: [<ffffffff814063c0>] ? ip_rcv+0x280/0x2c5
> Apr 9 08:39:49 host kernel: [<ffffffff8135d39d>] ? xennet_poll+0xb06/0xc68
> Apr 9 08:39:49 host kernel: [<ffffffff8100e39f>] ? xen_restore_fl_direct_end+0x0/0x1
> Apr 9 08:39:49 host kernel: [<ffffffff81334afc>] ? do_blkif_request+0x341/0x350
> Apr 9 08:39:49 host kernel: [<ffffffff8100e3b2>] ? check_events+0x12/0x20
> Apr 9 08:39:49 host kernel: [<ffffffff813c469b>] ? net_rx_action+0x77/0x179
> Apr 9 08:39:49 host kernel: [<ffffffff810462b4>] ? __do_softirq+0xdd/0x1a1
> Apr 9 08:39:49 host kernel: [<ffffffff81011bec>] ? call_softirq+0x1c/0x30
> Apr 9 08:39:49 host kernel: [<ffffffff81012edf>] ? do_softirq+0x3f/0x7c
> Apr 9 08:39:49 host kernel: [<ffffffff812ff975>] ? xen_evtchn_do_upcall+0x173/0x190
> Apr 9 08:39:49 host kernel: [<ffffffff81011c3e>] ? xen_do_hypervisor_callback+0x1e/0x30
> Apr 9 08:39:49 host kernel: <EOI> [<ffffffff8100e3b2>] ? check_events+0x12/0x20
> Apr 9 08:39:49 host kernel: [<ffffffff8100922a>] ? hypercall_page+0x22a/0x1001
> Apr 9 08:39:49 host kernel: [<ffffffff8100922a>] ? hypercall_page+0x22a/0x1001
> Apr 9 08:39:49 host kernel: [<ffffffff8100dd39>] ? xen_force_evtchn_callback+0x9/0xa
> Apr 9 08:39:49 host kernel: [<ffffffff8100e3b2>] ? check_events+0x12/0x20
> Apr 9 08:39:49 host kernel: [<ffffffff8100e3b2>] ? check_events+0x12/0x20
> Apr 9 08:39:49 host kernel: [<ffffffff8100e359>] ? xen_irq_enable_direct_end+0x0/0x7
> Apr 9 08:39:49 host kernel: [<ffffffff8103b070>] ? finish_task_switch+0x27/0x6e
> Apr 9 08:39:49 host kernel: [<ffffffff814d6a77>] ? thread_return+0x3e/0x109
> Apr 9 08:39:49 host kernel: [<ffffffff8103d336>] ? default_wake_function+0x0/0x9
> Apr 9 08:39:49 host kernel: [<ffffffff81261668>] ? cap_socket_recvmsg+0x0/0x3
> Apr 9 08:39:49 host kernel: [<ffffffff81261668>] ? cap_socket_recvmsg+0x0/0x3
> Apr 9 08:39:49 host kernel: [<ffffffff814d6e59>] ? schedule_timeout+0x23/0x1d9
> Apr 9 08:39:49 host kernel: [<ffffffff8100dd39>] ? xen_force_evtchn_callback+0x9/0xa
> Apr 9 08:39:49 host kernel: [<ffffffff8100e3b2>] ? check_events+0x12/0x20
> Apr 9 08:39:49 host kernel: [<ffffffff81261668>] ? cap_socket_recvmsg+0x0/0x3
> Apr 9 08:39:49 host kernel: [<ffffffff814d824c>] ? _spin_unlock_irqrestore+0xc/0xd
> Apr 9 08:39:49 host kernel: [<ffffffff81456ebe>] ? unix_stream_recvmsg+0x2c4/0x644
> Apr 9 08:39:49 host kernel: [<ffffffff810561b8>] ? autoremove_wake_function+0x0/0x2e
> Apr 9 08:39:49 host kernel: [<ffffffff81457231>] ? unix_stream_recvmsg+0x637/0x644
> Apr 9 08:39:49 host kernel: [<ffffffff813b6303>] ? sock_aio_read+0x103/0x117
> Apr 9 08:39:49 host kernel: [<ffffffff810bac91>] ? do_sync_read+0xce/0x113
> Apr 9 08:39:49 host kernel: [<ffffffff81011c1b>] ? xen_hypervisor_callback+0x1b/0x20
> Apr 9 08:39:49 host kernel: [<ffffffff810561b8>] ? autoremove_wake_function+0x0/0x2e
> Apr 9 08:39:49 host kernel: [<ffffffff81261668>] ? cap_socket_recvmsg+0x0/0x3
> Apr 9 08:39:49 host kernel: [<ffffffff814d824c>] ? _spin_unlock_irqrestore+0xc/0xd
> Apr 9 08:39:49 host kernel: [<ffffffff81011c1b>] ? xen_hypervisor_callback+0x1b/0x20
> Apr 9 08:39:49 host kernel: [<ffffffff810bb6f0>] ? vfs_read+0xbd/0x146
> Apr 9 08:39:49 host kernel: [<ffffffff8100922a>] ? hypercall_page+0x22a/0x1001
> Apr 9 08:39:49 host kernel: [<ffffffff810bb835>] ? sys_read+0x45/0x6e
> Apr 9 08:39:49 host kernel: [<ffffffff81010ac2>] ? system_call_fastpath+0x16/0x1b
> Apr 9 09:11:36 host kernel: BUG: soft lockup - CPU#1 stuck for 1776s! [swapper:0]
> Apr 9 09:11:36 host kernel: Modules linked in:
> Apr 9 09:11:36 host kernel: CPU 1:
> Apr 9 09:11:36 host kernel: Modules linked in:
> Apr 9 09:11:36 host kernel: Pid: 0, comm: swapper Not tainted 2.6.32.9xls-domU #2
> Apr 9 09:11:36 host kernel: RIP: e030:[<ffffffff810093aa>] [<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
> Apr 9 09:11:36 host kernel: RSP: e02b:ffff88007f8a1f08 EFLAGS: 00000246
> Apr 9 09:11:36 host kernel: RAX: 0000000000000000 RBX: ffff88007f8a0000 RCX: ffffffff810093aa
> Apr 9 09:11:36 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000001
> Apr 9 09:11:36 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000 R09: ffffffff8100e3b2
> Apr 9 09:11:36 host kernel: R10: 0000000000000001 R11: 0000000000000246 R12: 0000000000000000
> Apr 9 09:11:36 host kernel: R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
> Apr 9 09:11:36 host kernel: FS: 00007f44bc51f710(0000) GS:ffff8800022c7000(0000) knlGS:0000000000000000
> Apr 9 09:11:36 host kernel: CS: e033 DS: 002b ES: 002b CR0: 000000008005003b
> Apr 9 09:11:36 host kernel: CR2: 00007f44bc52c000 CR3: 0000000071c86000 CR4: 0000000000002660
> Apr 9 09:11:36 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> Apr 9 09:11:36 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> Apr 9 09:11:36 host kernel: Call Trace:
> Apr 9 09:11:36 host kernel: [<ffffffff8100ddb7>] ? xen_safe_halt+0xc/0x15
> Apr 9 09:11:36 host kernel: [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40
> Apr 9 09:11:36 host kernel: [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82
> Apr 9 09:11:36 host kernel: BUG: soft lockup - CPU#0 stuck for 1776s! [swapper:0]
> Apr 9 09:11:36 host kernel: Modules linked in:
> Apr 9 09:11:36 host kernel: CPU 0:
> Apr 9 09:11:36 host kernel: Modules linked in:
> Apr 9 09:11:36 host kernel: Pid: 0, comm: swapper Not tainted 2.6.32.9xls-domU #2
> Apr 9 09:11:36 host kernel: RIP: e030:[<ffffffff810093aa>] [<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
> Apr 9 09:11:36 host kernel: RSP: e02b:ffffffff81691f70 EFLAGS: 00000246
> Apr 9 09:11:36 host kernel: RAX: 0000000000000000 RBX: ffffffff81690000 RCX: ffffffff810093aa
> Apr 9 09:11:36 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000001
> Apr 9 09:11:36 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000 R09: ffffffff8100e3b2
> Apr 9 09:11:36 host kernel: R10: 0000000000000001 R11: 0000000000000246 R12: ffffffffffffffff
> Apr 9 09:11:36 host kernel: R13: ffffffff818ebf20 R14: ffffffff818eec70 R15: 0000000000000000
> Apr 9 09:11:36 host kernel: FS: 00007f07704fe6e0(0000) GS:ffff8800022ac000(0000) knlGS:0000000000000000
> Apr 9 09:11:36 host kernel: CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b
> Apr 9 09:11:36 host kernel: CR2: 00000000004884b2 CR3: 0000000071c86000 CR4: 0000000000002660
> Apr 9 09:11:36 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> Apr 9 09:11:36 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> Apr 9 09:11:36 host kernel: Call Trace:
> Apr 9 09:11:36 host kernel: [<ffffffff8100ddb7>] ? xen_safe_halt+0xc/0x15
> Apr 9 09:11:36 host kernel: [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40
> Apr 9 09:11:36 host kernel: [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82
> Apr 9 09:11:36 host kernel: [<ffffffff818b6c42>] ? start_kernel+0x353/0x35f
> Apr 9 09:20:24 host kernel: BUG: soft lockup - CPU#1 stuck for 492s! [ssh:2324]
> Apr 9 09:20:24 host kernel: Modules linked in:
> Apr 9 09:20:24 host kernel: CPU 1:
> Apr 9 09:20:24 host kernel: Modules linked in:
> Apr 9 09:20:24 host kernel: Pid: 2324, comm: ssh Not tainted 2.6.32.9xls-domU #2
> Apr 9 09:20:24 host kernel: RIP: e033:[<00007f1b0e259f20>] [<00007f1b0e259f20>] 0x7f1b0e259f20
> Apr 9 09:20:24 host kernel: RSP: e02b:00007fffa45a7338 EFLAGS: 00000206
> Apr 9 09:20:24 host kernel: RAX: 00007f1b11f32b20 RBX: 00007f1b0e5399c0 RCX: 00000000000000a0
> Apr 9 09:20:24 host kernel: RDX: 00007f1b11f32b80 RSI: 0000000000000004 RDI: 00007f1b0e539c30
> Apr 9 09:20:24 host kernel: RBP: 0000000000000064 R08: 0000000000000001 R09: 0000000000000001
> Apr 9 09:20:24 host kernel: R10: 00007f1b0e539a30 R11: 00007f1b0e25bfb0 R12: 0000000000000000
> Apr 9 09:20:24 host kernel: R13: 00007f1b0fcd2aec R14: 0000000000000111 R15: 0000000000000071
> Apr 9 09:20:24 host kernel: FS: 00007f1b10528710(0000) GS:ffff8800022c7000(0000) knlGS:0000000000000000
> Apr 9 09:20:24 host kernel: CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b
> Apr 9 09:20:24 host kernel: CR2: 00007fa913f53000 CR3: 000000006ad44000 CR4: 0000000000002660
> Apr 9 09:20:24 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> Apr 9 09:20:24 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> Apr 9 09:20:24 host kernel: Call Trace:
> Apr 9 09:20:24 host kernel: BUG: soft lockup - CPU#0 stuck for 492s! [swapper:0]
> Apr 9 09:20:24 host kernel: Modules linked in:
> Apr 9 09:20:24 host kernel: CPU 0:
> Apr 9 09:20:24 host kernel: Modules linked in:
> Apr 9 09:20:24 host kernel: Pid: 0, comm: swapper Not tainted 2.6.32.9xls-domU #2
> Apr 9 09:20:24 host kernel: RIP: e030:[<ffffffff810093aa>] [<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
> Apr 9 09:20:24 host kernel: RSP: e02b:ffffffff81691f70 EFLAGS: 00000246
> Apr 9 09:20:24 host kernel: RAX: 0000000000000000 RBX: ffffffff81690000 RCX: ffffffff810093aa
> Apr 9 09:20:24 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000001
> Apr 9 09:20:24 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000 R09: ffffffff8100e3b2
> Apr 9 09:20:24 host kernel: R10: 0000000000000001 R11: 0000000000000246 R12: ffffffffffffffff
> Apr 9 09:20:24 host kernel: R13: ffffffff818ebf20 R14: ffffffff818eec70 R15: 0000000000000000
> Apr 9 09:20:24 host kernel: FS: 00007ff39e495710(0000) GS:ffff8800022ac000(0000) knlGS:0000000000000000
> Apr 9 09:20:24 host kernel: CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b
> Apr 9 09:20:24 host kernel: CR2: 00007fff892a1e30 CR3: 000000006acb1000 CR4: 0000000000002660
> Apr 9 09:20:24 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> Apr 9 09:20:24 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> Apr 9 09:20:24 host kernel: Call Trace:
> Apr 9 09:20:24 host kernel: [<ffffffff8100ddb7>] ? xen_safe_halt+0xc/0x15
> Apr 9 09:20:24 host kernel: [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40
> Apr 9 09:20:24 host kernel: [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82
> Apr 9 09:20:24 host kernel: [<ffffffff818b6c42>] ? start_kernel+0x353/0x35f
> Apr 9 09:24:36 host kernel: BUG: soft lockup - CPU#1 stuck for 234s! [gzip:4496]
> Apr 9 09:24:36 host kernel: Modules linked in:
> Apr 9 09:24:36 host kernel: CPU 1:
> Apr 9 09:24:36 host kernel: Modules linked in:
> Apr 9 09:24:36 host kernel: Pid: 4496, comm: gzip Not tainted 2.6.32.9xls-domU #2
> Apr 9 09:24:36 host kernel: RIP: e033:[<0000000000409804>] [<0000000000409804>] 0x409804
> Apr 9 09:24:36 host kernel: RSP: e02b:00007fff87e009f8 EFLAGS: 00000202
> Apr 9 09:24:36 host kernel: RAX: 00000000f6019e6d RBX: 0000000000008000 RCX: 00007f529598c610
> Apr 9 09:24:36 host kernel: RDX: 0000000000f6019e RSI: 00000000000045a6 RDI: 00000000006534fb
> Apr 9 09:24:36 host kernel: RBP: 0000000000005800 R08: 0000000000004000 R09: 7464697720223333
> Apr 9 09:24:36 host kernel: R10: 6c61202236223d68 R11: 0000000000000246 R12: 00000000006522a0
> Apr 9 09:24:36 host kernel: R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
> Apr 9 09:24:36 host kernel: FS: 00007f5295e2a6e0(0000) GS:ffff8800022c7000(0000) knlGS:0000000000000000
> Apr 9 09:24:36 host kernel: CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b
> Apr 9 09:24:36 host kernel: CR2: 0000000000d550c8 CR3: 0000000026474000 CR4: 0000000000002660
> Apr 9 09:24:37 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> Apr 9 09:24:37 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> Apr 9 09:24:37 host kernel: Call Trace:
> Apr 9 09:24:37 host kernel: BUG: soft lockup - CPU#0 stuck for 234s! [swapper:0]
> Apr 9 09:24:37 host kernel: Modules linked in:
> Apr 9 09:24:37 host kernel: CPU 0:
> Apr 9 09:24:37 host kernel: Modules linked in:
> Apr 9 09:24:37 host kernel: Pid: 0, comm: swapper Not tainted 2.6.32.9xls-domU #2
> Apr 9 09:24:37 host kernel: RIP: e030:[<ffffffff810093aa>] [<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
> Apr 9 09:24:38 host kernel: RSP: e02b:ffffffff81691f70 EFLAGS: 00000246
> Apr 9 09:24:38 host kernel: RAX: 0000000000000000 RBX: ffffffff81690000 RCX: ffffffff810093aa
> Apr 9 09:24:38 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000001
> Apr 9 09:24:38 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000 R09: ffffffff8100e3b2
> Apr 9 09:24:38 host kernel: R10: 0000000000000001 R11: 0000000000000246 R12: ffffffffffffffff
> Apr 9 09:24:38 host kernel: R13: ffffffff818ebf20 R14: ffffffff818eec70 R15: 0000000000000000
> Apr 9 09:24:38 host kernel: FS: 00007f2f6a31a710(0000) GS:ffff8800022ac000(0000) knlGS:0000000000000000
> Apr 9 09:24:38 host kernel: CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b
> Apr 9 09:24:38 host kernel: CR2: 00007f2f6a327000 CR3: 000000003f62e000 CR4: 0000000000002660
> Apr 9 09:24:38 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> Apr 9 09:24:38 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> Apr 9 09:24:38 host kernel: Call Trace:
> Apr 9 09:24:38 host kernel: [<ffffffff8100ddb7>] ? xen_safe_halt+0xc/0x15
> Apr 9 09:24:38 host kernel: [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40
> Apr 9 09:24:38 host kernel: [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82
> Apr 9 09:24:38 host kernel: [<ffffffff818b6c42>] ? start_kernel+0x353/0x35f
> Apr 9 10:05:51 host kernel: BUG: soft lockup - CPU#1 stuck for 2305s! [swapper:0]
> Apr 9 10:05:51 host kernel: Modules linked in:
> Apr 9 10:05:51 host kernel: CPU 1:
> Apr 9 10:05:51 host kernel: Modules linked in:
> Apr 9 10:05:51 host kernel: Pid: 0, comm: swapper Not tainted 2.6.32.9xls-domU #2
> Apr 9 10:05:51 host kernel: RIP: e030:[<ffffffff810093aa>] [<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
> Apr 9 10:05:51 host kernel: RSP: e02b:ffff88007f8a1f08 EFLAGS: 00000246
> Apr 9 10:05:51 host kernel: RAX: 0000000000000000 RBX: ffff88007f8a0000 RCX: ffffffff810093aa
> Apr 9 10:05:51 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000001
> Apr 9 10:05:51 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000 R09: ffffffff8100e3b2
> Apr 9 10:05:51 host kernel: R10: 0000000000000001 R11: 0000000000000246 R12: 0000000000000000
> Apr 9 10:05:51 host kernel: R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
> Apr 9 10:05:51 host kernel: FS: 00007ff39e495710(0000) GS:ffff8800022c7000(0000) knlGS:0000000000000000
> Apr 9 10:05:51 host kernel: CS: e033 DS: 002b ES: 002b CR0: 000000008005003b
> Apr 9 10:05:51 host kernel: CR2: 0000000004786158 CR3: 000000007e2a6000 CR4: 0000000000002660
> Apr 9 10:05:51 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> Apr 9 10:05:51 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> Apr 9 10:05:51 host kernel: Call Trace:
> Apr 9 10:05:51 host kernel: [<ffffffff8100ddb7>] ? xen_safe_halt+0xc/0x15
> Apr 9 10:05:51 host kernel: [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40
> Apr 9 10:05:51 host kernel: [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82
> Apr 9 10:05:51 host kernel: BUG: soft lockup - CPU#0 stuck for 2305s! [httpd:9922]
> Apr 9 10:05:51 host kernel: Modules linked in:
> Apr 9 10:05:51 host kernel: CPU 0:
> Apr 9 10:05:51 host kernel: Modules linked in:
> Apr 9 10:05:51 host kernel: Pid: 9922, comm: httpd Not tainted 2.6.32.9xls-domU #2
> Apr 9 10:05:51 host kernel: RIP: e033:[<00007ff39a44a36d>] [<00007ff39a44a36d>] 0x7ff39a44a36d
> Apr 9 10:05:51 host kernel: RSP: e02b:00007fff4b9ad118 EFLAGS: 00000a86
> Apr 9 10:05:51 host kernel: RAX: c0c176e403831574 RBX: 0000000003643b60 RCX: c0c176e403831574
> Apr 9 10:05:51 host kernel: RDX: 0000000000000000 RSI: 0000000000000004 RDI: 000000000420e43b
> Apr 9 10:05:51 host kernel: RBP: 0000000003643b60 R08: 0000000000000004 R09: 00007fff4b9ad270
> Apr 9 10:05:51 host kernel: R10: 00007ff39aaa9fa0 R11: 00007fff4b9ad188 R12: 000000000320e6d0
> Apr 9 10:05:51 host kernel: R13: 0000000000000000 R14: 00007fff4b9ad368 R15: 0000000000000000
> Apr 9 10:05:51 host kernel: FS: 00007ff39e495710(0000) GS:ffff8800022ac000(0000) knlGS:0000000000000000
> Apr 9 10:05:51 host kernel: CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b
> Apr 9 10:05:51 host kernel: CR2: 00000000047a4018 CR3: 000000007972c000 CR4: 0000000000002660
> Apr 9 10:05:51 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> Apr 9 10:05:51 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> Apr 9 10:05:51 host kernel: Call Trace:
> Apr 9 10:50:04 host kernel: BUG: soft lockup - CPU#1 stuck for 2471s! [swapper:0]
> Apr 9 10:50:04 host kernel: Modules linked in:
> Apr 9 10:50:04 host kernel: CPU 1:
> Apr 9 10:50:04 host kernel: Modules linked in:
> Apr 9 10:50:04 host kernel: Pid: 0, comm: swapper Not tainted 2.6.32.9xls-domU #2
> Apr 9 10:50:04 host kernel: RIP: e030:[<ffffffff810093aa>] [<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
> Apr 9 10:50:04 host kernel: RSP: e02b:ffff88007f8a1f08 EFLAGS: 00000246
> Apr 9 10:50:04 host kernel: RAX: 0000000000000000 RBX: ffff88007f8a0000 RCX: ffffffff810093aa
> Apr 9 10:50:04 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000001
> Apr 9 10:50:04 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000 R09: ffffffff8100e3b2
> Apr 9 10:50:04 host kernel: R10: 0000000000000001 R11: 0000000000000246 R12: 0000000000000000
> Apr 9 10:50:04 host kernel: R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
> Apr 9 10:50:04 host kernel: FS: 00007ff39e495710(0000) GS:ffff8800022c7000(0000) knlGS:0000000000000000
> Apr 9 10:50:04 host kernel: CS: e033 DS: 002b ES: 002b CR0: 000000008005003b
> Apr 9 10:50:04 host kernel: CR2: 00007ff398a0ce08 CR3: 000000004ddb4000 CR4: 0000000000002660
> Apr 9 10:50:04 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> Apr 9 10:50:04 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> Apr 9 10:50:04 host kernel: Call Trace:
> Apr 9 10:50:04 host kernel: [<ffffffff8100ddb7>] ? xen_safe_halt+0xc/0x15
> Apr 9 10:50:04 host kernel: [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40
> Apr 9 10:50:04 host kernel: [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82
> Apr 9 10:50:04 host kernel: BUG: soft lockup - CPU#0 stuck for 2471s! [httpd:16797]
> Apr 9 10:50:04 host kernel: Modules linked in:
> Apr 9 10:50:04 host kernel: CPU 0:
> Apr 9 10:50:04 host kernel: Modules linked in:
> Apr 9 10:50:04 host kernel: Pid: 16797, comm: httpd Not tainted 2.6.32.9xls-domU #2
> Apr 9 10:50:04 host kernel: RIP: e033:[<00007ff39a49ae55>] [<00007ff39a49ae55>] 0x7ff39a49ae55
> Apr 9 10:50:04 host kernel: RSP: e02b:00007fff4b9b10b8 EFLAGS: 00000246
> Apr 9 10:50:04 host kernel: RAX: 000000000368dd40 RBX: 00000000036868d0 RCX: 222879726575713e
> Apr 9 10:50:05 host kernel: RDX: 000000000000000c RSI: 0000000003686a6c RDI: 00007fff4b9b1d00
> Apr 9 10:50:05 host kernel: RBP: 00007fff4b9b1da0 R08: 0000000000000001 R09: 0101010101010101
> Apr 9 10:50:05 host kernel: R10: 0000000000000001 R11: 00007ff39c4ac0be R12: 00007fff4b9b1d00
> Apr 9 10:50:05 host kernel: R13: 00007ff39aaa9fa0 R14: 00007ff39aaa9fa0 R15: 0000000003695df0
> Apr 9 10:50:05 host kernel: FS: 00007ff39e495710(0000) GS:ffff8800022ac000(0000) knlGS:0000000000000000
> Apr 9 10:50:05 host kernel: CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b
> Apr 9 10:50:05 host kernel: CR2: 00000000032068c0 CR3: 000000004ddb4000 CR4: 0000000000002660
> Apr 9 10:50:05 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> Apr 9 10:50:05 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> Apr 9 10:50:05 host kernel: Call Trace:
> Apr 9 10:57:01 host kernel: BUG: soft lockup - CPU#1 stuck for 389s! [exim:17977]
> Apr 9 10:57:01 host kernel: Modules linked in:
> Apr 9 10:57:01 host kernel: CPU 1:
> Apr 9 10:57:01 host kernel: Modules linked in:
> Apr 9 10:57:01 host kernel: Pid: 17977, comm: exim Not tainted 2.6.32.9xls-domU #2
> Apr 9 10:57:01 host kernel: RIP: e030:[<ffffffff8100922a>] [<ffffffff8100922a>] hypercall_page+0x22a/0x1001
> Apr 9 10:57:01 host kernel: RSP: e02b:ffff880056611ea0 EFLAGS: 00000246
> Apr 9 10:57:01 host kernel: RAX: 0000000000030004 RBX: 0000000000000000 RCX: ffffffff8100922a
> Apr 9 10:57:01 host kernel: RDX: 00007fd75a6d614e RSI: 0000000000000000 RDI: 0000000000000000
> Apr 9 10:57:01 host kernel: RBP: 00007fd75a6d614e R08: 00007fd75c048710 R09: 0000000000000003
> Apr 9 10:57:01 host kernel: R10: 00007fd75a909a30 R11: 0000000000000246 R12: 0000000000000004
> Apr 9 10:57:01 host kernel: R13: ffff880056611f58 R14: ffff88005aa996d0 R15: ffff88005a51b3c0
> Apr 9 10:57:01 host kernel: FS: 00007fd75c048710(0000) GS:ffff8800022c7000(0000) knlGS:0000000000000000
> Apr 9 10:57:01 host kernel: CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b
> Apr 9 10:57:01 host kernel: CR2: 00007fd75a6d614e CR3: 000000007eb9a000 CR4: 0000000000002660
> Apr 9 10:57:01 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> Apr 9 10:57:01 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> Apr 9 10:57:01 host kernel: Call Trace:
> Apr 9 10:57:01 host kernel: [<ffffffff8100dd39>] ? xen_force_evtchn_callback+0x9/0xa
> Apr 9 10:57:01 host kernel: [<ffffffff8100e3b2>] ? check_events+0x12/0x20
> Apr 9 10:57:01 host kernel: [<ffffffff8100e359>] ? xen_irq_enable_direct_end+0x0/0x7
> Apr 9 10:57:01 host kernel: [<ffffffff814da477>] ? do_page_fault+0xfb/0x28d
> Apr 9 10:57:01 host kernel: [<ffffffff814d85b5>] ? page_fault+0x25/0x30
> Apr 9 10:57:01 host kernel: BUG: soft lockup - CPU#0 stuck for 389s! [netstat:17966]
> Apr 9 10:57:01 host kernel: Modules linked in:
> Apr 9 10:57:01 host kernel: CPU 0:
> Apr 9 10:57:01 host kernel: Modules linked in:
> Apr 9 10:57:01 host kernel: Pid: 17966, comm: netstat Not tainted 2.6.32.9xls-domU #2
> Apr 9 10:57:01 host kernel: RIP: e030:[<ffffffff814d8226>] [<ffffffff814d8226>] _spin_lock+0xa/0x15
> Apr 9 10:57:01 host kernel: RSP: e02b:ffff88006ad4dce0 EFLAGS: 00000202
> Apr 9 10:57:01 host kernel: RAX: 0000000000001c1c RBX: ffff88007ff77180 RCX: 0000000000000012
> Apr 9 10:57:01 host kernel: RDX: 018721dfffc35c9f RSI: ffff88006ad4ddc8 RDI: ffff88007ff77188
> Apr 9 10:57:01 host kernel: RBP: ffff88007ff77198 R08: 0000000000000000 R09: 0000000000300000
> Apr 9 10:57:01 host kernel: R10: 00000000fffffff2 R11: ffffffff8126158e R12: ffff88000ff859c0
> Apr 9 10:57:01 host kernel: R13: ffff88006ad4ddc8 R14: 0000000000034b38 R15: 0000000000000002
> Apr 9 10:57:01 host kernel: FS: 00007fb435545710(0000) GS:ffff8800022ac000(0000) knlGS:0000000000000000
> Apr 9 10:57:01 host kernel: CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b
> Apr 9 10:57:01 host kernel: CR2: 00007f0e429d6000 CR3: 000000007ee3d000 CR4: 0000000000002660
> Apr 9 10:57:01 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> Apr 9 10:57:01 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> Apr 9 10:57:01 host kernel: Call Trace:
> Apr 9 10:57:01 host kernel: [<ffffffff810cb01b>] ? __d_lookup+0x7a/0x105
> Apr 9 10:57:01 host kernel: [<ffffffff810c251e>] ? do_lookup+0x2a/0x1b3
> Apr 9 10:57:01 host kernel: [<ffffffff810c4498>] ? __link_path_walk+0x348/0xdcd
> Apr 9 10:57:01 host kernel: [<ffffffff8100dd39>] ? xen_force_evtchn_callback+0x9/0xa
> Apr 9 10:57:01 host kernel: [<ffffffff8100e3b2>] ? check_events+0x12/0x20
> Apr 9 10:57:01 host kernel: [<ffffffff810c514b>] ? path_walk+0x66/0xca
> Apr 9 10:57:01 host kernel: [<ffffffff810c51cf>] ? do_path_lookup+0x20/0x77
> Apr 9 10:57:01 host kernel: [<ffffffff810c5ae9>] ? user_path_at+0x48/0x79
> Apr 9 10:57:01 host kernel: [<ffffffff8100e39f>] ? xen_restore_fl_direct_end+0x0/0x1
> Apr 9 10:57:01 host kernel: [<ffffffff810cfd6b>] ? mntput_no_expire+0x23/0xed
> Apr 9 10:57:01 host kernel: [<ffffffff8129b703>] ? _atomic_dec_and_lock+0x33/0x50
> Apr 9 10:57:01 host kernel: [<ffffffff810cfd6b>] ? mntput_no_expire+0x23/0xed
> Apr 9 10:57:01 host kernel: [<ffffffff810bdf4d>] ? sys_readlinkat+0x25/0x8d
> Apr 9 10:57:01 host kernel: [<ffffffff81010ac2>] ? system_call_fastpath+0x16/0x1b
> Apr 9 11:00:36 host kernel: BUG: soft lockup - CPU#1 stuck for 200s! [swapper:0]
> Apr 9 11:00:36 host kernel: Modules linked in:
> Apr 9 11:00:36 host kernel: CPU 1:
> Apr 9 11:00:36 host kernel: Modules linked in:
> Apr 9 11:00:36 host kernel: Pid: 0, comm: swapper Not tainted 2.6.32.9xls-domU #2
> Apr 9 11:00:36 host kernel: RIP: e030:[<ffffffff810093aa>] [<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
> Apr 9 11:00:36 host kernel: RSP: e02b:ffff88007f8a1f08 EFLAGS: 00000246
> Apr 9 11:00:36 host kernel: RAX: 0000000000000000 RBX: ffff88007f8a0000 RCX: ffffffff810093aa
> Apr 9 11:00:36 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000001
> Apr 9 11:00:36 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000 R09: ffffffff8100e3b2
> Apr 9 11:00:36 host kernel: R10: 0000000000000001 R11: 0000000000000246 R12: 0000000000000000
> Apr 9 11:00:36 host kernel: R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
> Apr 9 11:00:36 host kernel: FS: 00007f719f5fa6e0(0000) GS:ffff8800022c7000(0000) knlGS:0000000000000000
> Apr 9 11:00:36 host kernel: CS: e033 DS: 002b ES: 002b CR0: 000000008005003b
> Apr 9 11:00:36 host kernel: CR2: 00007ff39c43cfa8 CR3: 000000007d0db000 CR4: 0000000000002660
> Apr 9 11:00:36 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> Apr 9 11:00:36 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> Apr 9 11:00:37 host kernel: Call Trace:
> Apr 9 11:00:37 host kernel: [<ffffffff8100ddb7>] ? xen_safe_halt+0xc/0x15
> Apr 9 11:00:37 host kernel: [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40
> Apr 9 11:00:37 host kernel: [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82
> Apr 9 11:00:37 host kernel: BUG: soft lockup - CPU#0 stuck for 200s! [swapper:0]
> Apr 9 11:00:37 host kernel: Modules linked in:
> Apr 9 11:00:37 host kernel: CPU 0:
> Apr 9 11:00:37 host kernel: Modules linked in:
> Apr 9 11:00:37 host kernel: Pid: 0, comm: swapper Not tainted 2.6.32.9xls-domU #2
> Apr 9 11:00:37 host kernel: RIP: e030:[<ffffffff810093aa>] [<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
> Apr 9 11:00:37 host kernel: RSP: e02b:ffffffff81691f70 EFLAGS: 00000246
> Apr 9 11:00:37 host kernel: RAX: 0000000000000000 RBX: ffffffff81690000 RCX: ffffffff810093aa
> Apr 9 11:00:37 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000001
> Apr 9 11:00:37 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000 R09: ffffffff8100e3b2
> Apr 9 11:00:37 host kernel: R10: 0000000000000001 R11: 0000000000000246 R12: ffffffffffffffff
> Apr 9 11:00:37 host kernel: R13: ffffffff818ebf20 R14: ffffffff818eec70 R15: 0000000000000000
> Apr 9 11:00:37 host kernel: FS: 00007f07704fe6e0(0000) GS:ffff8800022ac000(0000) knlGS:0000000000000000
> Apr 9 11:00:37 host kernel: CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b
> Apr 9 11:00:37 host kernel: CR2: 00000000035750c8 CR3: 000000007ea99000 CR4: 0000000000002660
> Apr 9 11:00:37 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> Apr 9 11:00:37 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> Apr 9 11:00:37 host kernel: Call Trace:
> Apr 9 11:00:37 host kernel: [<ffffffff8100ddb7>] ? xen_safe_halt+0xc/0x15
> Apr 9 11:00:37 host kernel: [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40
> Apr 9 11:00:37 host kernel: [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82
> Apr 9 11:00:37 host kernel: [<ffffffff818b6c42>] ? start_kernel+0x353/0x35f
> Apr 9 11:08:50 host kernel: BUG: soft lockup - CPU#0 stuck for 460s! [httpd:19360]
> Apr 9 11:08:50 host kernel: Modules linked in:
> Apr 9 11:08:50 host kernel: CPU 0:
> Apr 9 11:08:50 host kernel: Modules linked in:
> Apr 9 11:08:50 host kernel: Pid: 19360, comm: httpd Not tainted 2.6.32.9xls-domU #2
> Apr 9 11:08:50 host kernel: RIP: e030:[<ffffffff813b995e>] [<ffffffff813b995e>] sock_alloc_send_skb+0x3/0xc
> Apr 9 11:08:50 host kernel: RSP: e02b:ffff88005675dc80 EFLAGS: 00000246
> Apr 9 11:08:50 host kernel: RAX: 0000000000003e00 RBX: ffff88005675dcb8 RCX: ffff88005675dce4
> Apr 9 11:08:50 host kernel: RDX: 0000000000000000 RSI: 0000000000000057 RDI: ffff88002678c900
> Apr 9 11:08:50 host kernel: RBP: ffff88007f740b80 R08: ffff88005675dce4 R09: 75646f72705f6d76
> Apr 9 11:08:50 host kernel: R10: 5245485720607463 R11: ffffffff81261683 R12: 0000000000000057
> Apr 9 11:08:50 host kernel: R13: 0000000000000000 R14: ffff88006add3c80 R15: ffff88005675dd60
> Apr 9 11:08:50 host kernel: FS: 00007ff39e495710(0000) GS:ffff8800022ac000(0000) knlGS:0000000000000000
> Apr 9 11:08:50 host kernel: CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b
> Apr 9 11:08:50 host kernel: CR2: 00000000045cf000 CR3: 000000007f97d000 CR4: 0000000000002660
> Apr 9 11:08:50 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> Apr 9 11:08:50 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> Apr 9 11:08:50 host kernel: Call Trace:
> Apr 9 11:08:50 host kernel: [<ffffffff814573c8>] ? unix_stream_sendmsg+0x18a/0x31f
> Apr 9 11:08:50 host kernel: [<ffffffff813b61ec>] ? sock_aio_write+0xf6/0x10a
> Apr 9 11:08:50 host kernel: [<ffffffff8100dd39>] ? xen_force_evtchn_callback+0x9/0xa
> Apr 9 11:08:50 host kernel: [<ffffffff8100e39f>] ? xen_restore_fl_direct_end+0x0/0x1
> Apr 9 11:08:50 host kernel: [<ffffffff810bab7e>] ? do_sync_write+0xce/0x113
> Apr 9 11:08:50 host kernel: [<ffffffff810c5af3>] ? user_path_at+0x52/0x79
> Apr 9 11:08:50 host kernel: [<ffffffff810be0b0>] ? cp_new_stat+0xe9/0xfc
> Apr 9 11:08:50 host kernel: [<ffffffff810561b8>] ? autoremove_wake_function+0x0/0x2e
> Apr 9 11:08:50 host kernel: [<ffffffff8100e3b2>] ? check_events+0x12/0x20
> Apr 9 11:08:50 host kernel: [<ffffffff8100dd39>] ? xen_force_evtchn_callback+0x9/0xa
> Apr 9 11:08:50 host kernel: [<ffffffff8100e3b2>] ? check_events+0x12/0x20
> Apr 9 11:08:50 host kernel: [<ffffffff810bb4c5>] ? vfs_write+0xc0/0x149
> Apr 9 11:08:50 host kernel: [<ffffffff810bb60a>] ? sys_write+0x45/0x6e
> Apr 9 11:08:50 host kernel: [<ffffffff81010ac2>] ? system_call_fastpath+0x16/0x1b
> Apr 9 11:08:50 host kernel: BUG: soft lockup - CPU#1 stuck for 460s! [swapper:0]
> Apr 9 11:08:50 host kernel: Modules linked in:
> Apr 9 11:08:50 host kernel: CPU 1:
> Apr 9 11:08:50 host kernel: Modules linked in:
> Apr 9 11:08:50 host kernel: Pid: 0, comm: swapper Not tainted 2.6.32.9xls-domU #2
> Apr 9 11:08:50 host kernel: RIP: e030:[<ffffffff810093aa>] [<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
> Apr 9 11:08:50 host kernel: RSP: e02b:ffff88007f8a1f08 EFLAGS: 00000246
> Apr 9 11:08:50 host kernel: RAX: 0000000000000000 RBX: ffff88007f8a0000 RCX: ffffffff810093aa
> Apr 9 11:08:50 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000001
> Apr 9 11:08:50 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000 R09: ffffffff8100e3b2
> Apr 9 11:08:50 host kernel: R10: 0000000000000001 R11: 0000000000000246 R12: 0000000000000000
> Apr 9 11:08:50 host kernel: R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
> Apr 9 11:08:50 host kernel: FS: 00007f0e429cb6e0(0000) GS:ffff8800022c7000(0000) knlGS:0000000000000000
> Apr 9 11:08:50 host kernel: CS: e033 DS: 002b ES: 002b CR0: 000000008005003b
> Apr 9 11:08:50 host kernel: CR2: 0000000004598000 CR3: 000000007e2a6000 CR4: 0000000000002660
> Apr 9 11:08:50 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> Apr 9 11:08:50 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> Apr 9 11:08:50 host kernel: Call Trace:
> Apr 9 11:08:50 host kernel: [<ffffffff8100ddb7>] ? xen_safe_halt+0xc/0x15
> Apr 9 11:08:50 host kernel: [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40
> Apr 9 11:08:50 host kernel: [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82
> Apr 9 12:00:40 host kernel: BUG: soft lockup - CPU#1 stuck for 2897s! [httpd:26903]
> Apr 9 12:00:40 host kernel: Modules linked in:
> Apr 9 12:00:40 host kernel: CPU 1:
> Apr 9 12:00:40 host kernel: Modules linked in:
> Apr 9 12:00:40 host kernel: Pid: 26903, comm: httpd Not tainted 2.6.32.9xls-domU #2
> Apr 9 12:00:40 host kernel: RIP: e033:[<00007ff39a427cd2>] [<00007ff39a427cd2>] 0x7ff39a427cd2
> Apr 9 12:00:40 host kernel: RSP: e02b:00007fff4b9afa20 EFLAGS: 00000202
> Apr 9 12:00:40 host kernel: RAX: 0000000000000020 RBX: 00007fff4b9aff30 RCX: 0000000000000013
> Apr 9 12:00:40 host kernel: RDX: 00007ff39a77be20 RSI: 00007fff4b9aff50 RDI: 0000000003bd6418
> Apr 9 12:00:40 host kernel: RBP: 0000000000000000 R08: 00007ff39a782e60 R09: 00007ff39a783d80
> Apr 9 12:00:40 host kernel: R10: 0000000000000000 R11: 00007ff39c4ac0a2 R12: 00007ff39aaaa3e0
> Apr 9 12:00:40 host kernel: R13: 00007fff4b9aff50 R14: 00007fff4b9aff50 R15: 000000000000015e
> Apr 9 12:00:40 host kernel: FS: 00007ff39e495710(0000) GS:ffff8800022c7000(0000) knlGS:0000000000000000
> Apr 9 12:00:40 host kernel: CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b
> Apr 9 12:00:40 host kernel: CR2: 0000000003bdc000 CR3: 0000000056566000 CR4: 0000000000002660
> Apr 9 12:00:40 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> Apr 9 12:00:40 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> Apr 9 12:00:40 host kernel: Call Trace:
> Apr 9 12:00:40 host kernel: BUG: soft lockup - CPU#0 stuck for 2897s! [swapper:0]
> Apr 9 12:00:40 host kernel: Modules linked in:
> Apr 9 12:00:40 host kernel: CPU 0:
> Apr 9 12:00:40 host kernel: Modules linked in:
> Apr 9 12:00:40 host kernel: Pid: 0, comm: swapper Not tainted 2.6.32.9xls-domU #2
> Apr 9 12:00:40 host kernel: RIP: e030:[<ffffffff810093aa>] [<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
> Apr 9 12:00:40 host kernel: RSP: e02b:ffffffff81691f70 EFLAGS: 00000246
> Apr 9 12:00:40 host kernel: RAX: 0000000000000000 RBX: ffffffff81690000 RCX: ffffffff810093aa
> Apr 9 12:00:40 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000001
> Apr 9 12:00:40 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000 R09: ffffffff8100e3b2
> Apr 9 12:00:40 host kernel: R10: 0000000000000001 R11: 0000000000000246 R12: ffffffffffffffff
> Apr 9 12:00:40 host kernel: R13: ffffffff818ebf20 R14: ffffffff818eec70 R15: 0000000000000000
> Apr 9 12:00:40 host kernel: FS: 00007f74fde746e0(0000) GS:ffff8800022ac000(0000) knlGS:0000000000000000
> Apr 9 12:00:40 host kernel: CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b
> Apr 9 12:00:40 host kernel: CR2: 00007ff39aec11a7 CR3: 000000007e2a6000 CR4: 0000000000002660
> Apr 9 12:00:40 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> Apr 9 12:00:40 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> Apr 9 12:00:40 host kernel: Call Trace:
> Apr 9 12:00:40 host kernel: [<ffffffff8100ddb7>] ? xen_safe_halt+0xc/0x15
> Apr 9 12:00:40 host kernel: [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40
> Apr 9 12:00:40 host kernel: [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82
> Apr 9 12:00:40 host kernel: [<ffffffff818b6c42>] ? start_kernel+0x353/0x35f
> Apr 9 12:19:45 host kernel: BUG: soft lockup - CPU#1 stuck for 1065s! [swapper:0]
> Apr 9 12:19:45 host kernel: Modules linked in:
> Apr 9 12:19:45 host kernel: CPU 1:
> Apr 9 12:19:45 host kernel: Modules linked in:
> Apr 9 12:19:45 host kernel: Pid: 0, comm: swapper Not tainted 2.6.32.9xls-domU #2
> Apr 9 12:19:45 host kernel: RIP: e030:[<ffffffff810093aa>] [<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
> Apr 9 12:19:45 host kernel: RSP: e02b:ffff88007f8a1f08 EFLAGS: 00000246
> Apr 9 12:19:45 host kernel: RAX: 0000000000000000 RBX: ffff88007f8a0000 RCX: ffffffff810093aa
> Apr 9 12:19:45 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000001
> Apr 9 12:19:45 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000 R09: ffffffff8100e3b2
> Apr 9 12:19:45 host kernel: R10: 0000000000000001 R11: 0000000000000246 R12: 0000000000000000
> Apr 9 12:19:45 host kernel: R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
> Apr 9 12:19:45 host kernel: FS: 00007ff39e495710(0000) GS:ffff8800022c7000(0000) knlGS:0000000000000000
> Apr 9 12:19:45 host kernel: CS: e033 DS: 002b ES: 002b CR0: 000000008005003b
> Apr 9 12:19:45 host kernel: CR2: 00007ff39c5002b0 CR3: 0000000079588000 CR4: 0000000000002660
> Apr 9 12:19:45 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> Apr 9 12:19:45 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> Apr 9 12:19:45 host kernel: Call Trace:
> Apr 9 12:19:45 host kernel: [<ffffffff8100ddb7>] ? xen_safe_halt+0xc/0x15
> Apr 9 12:19:45 host kernel: [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40
> Apr 9 12:19:45 host kernel: [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82
> Apr 9 12:19:45 host kernel: BUG: soft lockup - CPU#0 stuck for 1065s! [swapper:0]
> Apr 9 12:19:45 host kernel: Modules linked in:
> Apr 9 12:19:45 host kernel: CPU 0:
> Apr 9 12:19:45 host kernel: Modules linked in:
> Apr 9 12:19:45 host kernel: Pid: 0, comm: swapper Not tainted 2.6.32.9xls-domU #2
> Apr 9 12:19:45 host kernel: RIP: e030:[<ffffffff810093aa>] [<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
> Apr 9 12:19:45 host kernel: RSP: e02b:ffffffff81691f70 EFLAGS: 00000246
> Apr 9 12:19:45 host kernel: RAX: 0000000000000000 RBX: ffffffff81690000 RCX: ffffffff810093aa
> Apr 9 12:19:45 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000001
> Apr 9 12:19:45 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000 R09: ffffffff8100e3b2
> Apr 9 12:19:45 host kernel: R10: 0000000000000001 R11: 0000000000000246 R12: ffffffffffffffff
> Apr 9 12:19:45 host kernel: R13: ffffffff818ebf20 R14: ffffffff818eec70 R15: 0000000000000000
> Apr 9 12:19:45 host kernel: FS: 00007ff39e495710(0000) GS:ffff8800022ac000(0000) knlGS:0000000000000000
> Apr 9 12:19:45 host kernel: CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b
> Apr 9 12:19:45 host kernel: CR2: 00007ff39e404000 CR3: 000000007e2a6000 CR4: 0000000000002660
> Apr 9 12:19:45 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> Apr 9 12:19:45 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> Apr 9 12:19:45 host kernel: Call Trace:
> Apr 9 12:19:45 host kernel: [<ffffffff8100ddb7>] ? xen_safe_halt+0xc/0x15
> Apr 9 12:19:45 host kernel: [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40
> Apr 9 12:19:45 host kernel: [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82
> Apr 9 12:19:45 host kernel: [<ffffffff818b6c42>] ? start_kernel+0x353/0x35f
> Apr 9 12:37:26 host kernel: BUG: soft lockup - CPU#1 stuck for 989s! [swapper:0]
> Apr 9 12:37:26 host kernel: Modules linked in:
> Apr 9 12:37:26 host kernel: CPU 1:
> Apr 9 12:37:26 host kernel: Modules linked in:
> Apr 9 12:37:26 host kernel: Pid: 0, comm: swapper Not tainted 2.6.32.9xls-domU #2
> Apr 9 12:37:26 host kernel: RIP: e030:[<ffffffff810093aa>] [<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
> Apr 9 12:37:26 host kernel: RSP: e02b:ffff88007f8a1f08 EFLAGS: 00000246
> Apr 9 12:37:26 host kernel: RAX: 0000000000000000 RBX: ffff88007f8a0000 RCX: ffffffff810093aa
> Apr 9 12:37:26 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000001
> Apr 9 12:37:26 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000 R09: ffffffff8100e3b2
> Apr 9 12:37:26 host kernel: R10: 0000000000000001 R11: 0000000000000246 R12: 0000000000000000
> Apr 9 12:37:26 host kernel: R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
> Apr 9 12:37:26 host kernel: FS: 00007ff39e495710(0000) GS:ffff8800022c7000(0000) knlGS:0000000000000000
> Apr 9 12:37:26 host kernel: CS: e033 DS: 002b ES: 002b CR0: 000000008005003b
> Apr 9 12:37:26 host kernel: CR2: 00000000039f7330 CR3: 000000007e2a6000 CR4: 0000000000002660
> Apr 9 12:37:26 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> Apr 9 12:37:26 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> Apr 9 12:37:26 host kernel: Call Trace:
> Apr 9 12:37:26 host kernel: [<ffffffff8100ddb7>] ? xen_safe_halt+0xc/0x15
> Apr 9 12:37:26 host kernel: [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40
> Apr 9 12:37:26 host kernel: [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82
> Apr 9 12:37:26 host kernel: BUG: soft lockup - CPU#0 stuck for 989s! [httpd:31684]
> Apr 9 12:37:26 host kernel: Modules linked in:
> Apr 9 12:37:26 host kernel: CPU 0:
> Apr 9 12:37:26 host kernel: Modules linked in:
> Apr 9 12:37:26 host kernel: Pid: 31684, comm: httpd Not tainted 2.6.32.9xls-domU #2
> Apr 9 12:37:26 host kernel: RIP: e033:[<00007ff39a41f7d3>] [<00007ff39a41f7d3>] 0x7ff39a41f7d3
> Apr 9 12:37:26 host kernel: RSP: e02b:00007fff4b9ad330 EFLAGS: 00000246
> Apr 9 12:37:26 host kernel: RAX: 00007ff39a41f7ac RBX: 00007ff39aaaa3e0 RCX: 0000000000000001
> Apr 9 12:37:26 host kernel: RDX: 00007ff39a781bdc RSI: 000000000000003e RDI: 0000000000000000
> Apr 9 12:37:26 host kernel: RBP: 00007ff39aaa9e60 R08: 00007ff39a782e60 R09: 0000000000000003
> Apr 9 12:37:26 host kernel: R10: 0000000000000000 R11: 00007ff39c4ac0a2 R12: 0000000003a2b297
> Apr 9 12:37:26 host kernel: R13: 00007ff39aaaa3e0 R14: 00007ff39a784da0 R15: 00007ff39a783e80
> Apr 9 12:37:26 host kernel: FS: 00007ff39e495710(0000) GS:ffff8800022ac000(0000) knlGS:0000000000000000
> Apr 9 12:37:26 host kernel: CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b
> Apr 9 12:37:26 host kernel: CR2: 0000000003aa1000 CR3: 000000004debb000 CR4: 0000000000002660
> Apr 9 12:37:26 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> Apr 9 12:37:26 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> Apr 9 12:37:26 host kernel: Call Trace:
> Apr 9 13:44:52 host kernel: BUG: soft lockup - CPU#1 stuck for 3768s! [swapper:0]
> Apr 9 13:44:52 host kernel: Modules linked in:
> Apr 9 13:44:52 host kernel: CPU 1:
> Apr 9 13:44:52 host kernel: Modules linked in:
> Apr 9 13:44:52 host kernel: Pid: 0, comm: swapper Not tainted 2.6.32.9xls-domU #2
> Apr 9 13:44:52 host kernel: RIP: e030:[<ffffffff810093aa>] [<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
> Apr 9 13:44:52 host kernel: RSP: e02b:ffff88007f8a1f08 EFLAGS: 00000246
> Apr 9 13:44:52 host kernel: RAX: 0000000000000000 RBX: ffff88007f8a0000 RCX: ffffffff810093aa
> Apr 9 13:44:52 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000001
> Apr 9 13:44:52 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000 R09: ffffffff8100e3b2
> Apr 9 13:44:52 host kernel: R10: 0000000000000001 R11: 0000000000000246 R12: 0000000000000000
> Apr 9 13:44:52 host kernel: R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
> Apr 9 13:44:52 host kernel: FS: 00007fd75c048710(0000) GS:ffff8800022c7000(0000) knlGS:0000000000000000
> Apr 9 13:44:52 host kernel: CS: e033 DS: 002b ES: 002b CR0: 000000008005003b
> Apr 9 13:44:52 host kernel: CR2: 00007fff892a1e30 CR3: 0000000071c86000 CR4: 0000000000002660
> Apr 9 13:44:52 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> Apr 9 13:44:52 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> Apr 9 13:44:52 host kernel: Call Trace:
> Apr 9 13:44:52 host kernel: [<ffffffff8100ddb7>] ? xen_safe_halt+0xc/0x15
> Apr 9 13:44:52 host kernel: [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40
> Apr 9 13:44:52 host kernel: [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82
> Apr 9 13:44:52 host kernel: BUG: soft lockup - CPU#0 stuck for 3768s! [swapper:0]
> Apr 9 13:44:52 host kernel: Modules linked in:
> Apr 9 13:44:52 host kernel: CPU 0:
> Apr 9 13:44:52 host kernel: Modules linked in:
> Apr 9 13:44:52 host kernel: Pid: 0, comm: swapper Not tainted 2.6.32.9xls-domU #2
> Apr 9 13:44:52 host kernel: RIP: e030:[<ffffffff810093aa>] [<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
> Apr 9 13:44:52 host kernel: RSP: e02b:ffffffff81691f70 EFLAGS: 00000246
> Apr 9 13:44:52 host kernel: RAX: 0000000000000000 RBX: ffffffff81690000 RCX: ffffffff810093aa
> Apr 9 13:44:52 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000001
> Apr 9 13:44:52 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000 R09: ffffffff8100e3b2
> Apr 9 13:44:52 host kernel: R10: 0000000000000001 R11: 0000000000000246 R12: ffffffffffffffff
> Apr 9 13:44:52 host kernel: R13: ffffffff818ebf20 R14: ffffffff818eec70 R15: 0000000000000000
> Apr 9 13:44:52 host kernel: FS: 00007f0e429cb6e0(0000) GS:ffff8800022ac000(0000) knlGS:0000000000000000
> Apr 9 13:44:52 host kernel: CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b
> Apr 9 13:44:52 host kernel: CR2: 00007f0e429d6000 CR3: 000000007d0de000 CR4: 0000000000002660
> Apr 9 13:44:52 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> Apr 9 13:44:52 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> Apr 9 13:44:52 host kernel: Call Trace:
> Apr 9 13:44:52 host kernel: [<ffffffff8100ddb7>] ? xen_safe_halt+0xc/0x15
> Apr 9 13:44:52 host kernel: [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40
> Apr 9 13:44:52 host kernel: [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82
> Apr 9 13:44:52 host kernel: [<ffffffff818b6c42>] ? start_kernel+0x353/0x35f
> Apr 9 13:55:43 host kernel: BUG: soft lockup - CPU#1 stuck for 607s! [swapper:0]
> Apr 9 13:55:43 host kernel: Modules linked in:
> Apr 9 13:55:43 host kernel: CPU 1:
> Apr 9 13:55:43 host kernel: Modules linked in:
> Apr 9 13:55:43 host kernel: Pid: 0, comm: swapper Not tainted 2.6.32.9xls-domU #2
> Apr 9 13:55:43 host kernel: RIP: e030:[<ffffffff810093aa>] [<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
> Apr 9 13:55:43 host kernel: RSP: e02b:ffff88007f8a1f08 EFLAGS: 00000246
> Apr 9 13:55:43 host kernel: RAX: 0000000000000000 RBX: ffff88007f8a0000 RCX: ffffffff810093aa
> Apr 9 13:55:43 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000001
> Apr 9 13:55:43 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000 R09: ffffffff8100e3b2
> Apr 9 13:55:43 host kernel: R10: 0000000000000001 R11: 0000000000000246 R12: 0000000000000000
> Apr 9 13:55:43 host kernel: R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
> Apr 9 13:55:43 host kernel: FS: 00007ff39e495710(0000) GS:ffff8800022c7000(0000) knlGS:0000000000000000
> Apr 9 13:55:43 host kernel: CS: e033 DS: 002b ES: 002b CR0: 000000008005003b
> Apr 9 13:55:43 host kernel: CR2: 0000000003c7b27f CR3: 00000000266e8000 CR4: 0000000000002660
> Apr 9 13:55:43 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> Apr 9 13:55:43 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> Apr 9 13:55:43 host kernel: Call Trace:
> Apr 9 13:55:43 host kernel: [<ffffffff8100ddb7>] ? xen_safe_halt+0xc/0x15
> Apr 9 13:55:43 host kernel: [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40
> Apr 9 13:55:43 host kernel: [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82
> Apr 9 13:55:43 host kernel: BUG: soft lockup - CPU#0 stuck for 607s! [swapper:0]
> Apr 9 13:55:43 host kernel: Modules linked in:
> Apr 9 13:55:43 host kernel: CPU 0:
> Apr 9 13:55:43 host kernel: Modules linked in:
> Apr 9 13:55:43 host kernel: Pid: 0, comm: swapper Not tainted 2.6.32.9xls-domU #2
> Apr 9 13:55:43 host kernel: RIP: e030:[<ffffffff810093aa>] [<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
> Apr 9 13:55:43 host kernel: RSP: e02b:ffffffff81691f70 EFLAGS: 00000246
> Apr 9 13:55:43 host kernel: RAX: 0000000000000000 RBX: ffffffff81690000 RCX: ffffffff810093aa
> Apr 9 13:55:43 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000001
> Apr 9 13:55:43 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000 R09: ffffffff8100e3b2
> Apr 9 13:55:43 host kernel: R10: 0000000000000001 R11: 0000000000000246 R12: ffffffffffffffff
> Apr 9 13:55:43 host kernel: R13: ffffffff818ebf20 R14: ffffffff818eec70 R15: 0000000000000000
> Apr 9 13:55:43 host kernel: FS: 00007ff39e495710(0000) GS:ffff8800022ac000(0000) knlGS:0000000000000000
> Apr 9 13:55:43 host kernel: CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b
> Apr 9 13:55:43 host kernel: CR2: 00007f2f28bf62e8 CR3: 000000007e2a6000 CR4: 0000000000002660
> Apr 9 13:55:43 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> Apr 9 13:55:43 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> Apr 9 13:55:43 host kernel: Call Trace:
> Apr 9 13:55:43 host kernel: [<ffffffff8100ddb7>] ? xen_safe_halt+0xc/0x15
> Apr 9 13:55:43 host kernel: [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40
> Apr 9 13:55:43 host kernel: [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82
> Apr 9 13:55:43 host kernel: [<ffffffff818b6c42>] ? start_kernel+0x353/0x35f
> Apr 9 14:16:21 host kernel: BUG: soft lockup - CPU#1 stuck for 1153s! [swapper:0]
> Apr 9 14:16:21 host kernel: Modules linked in:
> Apr 9 14:16:21 host kernel: CPU 1:
> Apr 9 14:16:21 host kernel: Modules linked in:
> Apr 9 14:16:21 host kernel: Pid: 0, comm: swapper Not tainted 2.6.32.9xls-domU #2
> Apr 9 14:16:21 host kernel: RIP: e030:[<ffffffff8100922a>] [<ffffffff8100922a>] hypercall_page+0x22a/0x1001
> Apr 9 14:16:21 host kernel: RSP: e02b:ffff88007f8a1dd8 EFLAGS: 00000246
> Apr 9 14:16:21 host kernel: RAX: 0000000000030004 RBX: ffff88007db55810 RCX: ffffffff8100922a
> Apr 9 14:16:21 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
> Apr 9 14:16:21 host kernel: RBP: 0000000000000001 R08: ffff88007f8a0000 R09: ffffffff8100e3b2
> Apr 9 14:16:21 host kernel: R10: 0000000000000001 R11: 0000000000000246 R12: 0000000000000000
> Apr 9 14:16:21 host kernel: R13: ffff88007e0040c0 R14: 0000000000000000 R15: 0000000000001000
> Apr 9 14:16:21 host kernel: FS: 00007ff39e495710(0000) GS:ffff8800022c7000(0000) knlGS:0000000000000000
> Apr 9 14:16:21 host kernel: CS: e033 DS: 002b ES: 002b CR0: 000000008005003b
> Apr 9 14:16:21 host kernel: CR2: 00007ff39e4a7000 CR3: 000000007e2a6000 CR4: 0000000000002660
> Apr 9 14:16:21 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> Apr 9 14:16:21 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> Apr 9 14:16:21 host kernel: Call Trace:
> Apr 9 14:16:21 host kernel: [<ffffffff8100dd39>] ? xen_force_evtchn_callback+0x9/0xa
> Apr 9 14:16:21 host kernel: [<ffffffff8100e3b2>] ? check_events+0x12/0x20
> Apr 9 14:16:21 host kernel: [<ffffffff8100e3b2>] ? check_events+0x12/0x20
> Apr 9 14:16:21 host kernel: [<ffffffff8100e359>] ? xen_irq_enable_direct_end+0x0/0x7
> Apr 9 14:16:21 host kernel: [<ffffffff8103b070>] ? finish_task_switch+0x27/0x6e
> Apr 9 14:16:21 host kernel: [<ffffffff814d6a77>] ? thread_return+0x3e/0x109
> Apr 9 14:16:21 host kernel: [<ffffffff814d87ba>] ? error_exit+0x2a/0x60
> Apr 9 14:16:21 host kernel: [<ffffffff8101149d>] ? retint_restore_args+0x5/0x6
> Apr 9 14:16:21 host kernel: [<ffffffff8100fe5f>] ? cpu_idle+0x80/0x82
> Apr 9 14:16:21 host kernel: BUG: soft lockup - CPU#0 stuck for 1153s! [swapper:0]
> Apr 9 14:16:21 host kernel: Modules linked in:
> Apr 9 14:16:21 host kernel: CPU 0:
> Apr 9 14:16:21 host kernel: Modules linked in:
> Apr 9 14:16:21 host kernel: Pid: 0, comm: swapper Not tainted 2.6.32.9xls-domU #2
> Apr 9 14:16:21 host kernel: RIP: e030:[<ffffffff810093aa>] [<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
> Apr 9 14:16:21 host kernel: RSP: e02b:ffffffff81691f70 EFLAGS: 00000246
> Apr 9 14:16:21 host kernel: RAX: 0000000000000000 RBX: ffffffff81690000 RCX: ffffffff810093aa
> Apr 9 14:16:21 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000001
> Apr 9 14:16:21 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000 R09: ffffffff8100e3b2
> Apr 9 14:16:21 host kernel: R10: 0000000000000001 R11: 0000000000000246 R12: ffffffffffffffff
> Apr 9 14:16:21 host kernel: R13: ffffffff818ebf20 R14: ffffffff818eec70 R15: 0000000000000000
> Apr 9 14:16:21 host kernel: FS: 00007ff39e495710(0000) GS:ffff8800022ac000(0000) knlGS:0000000000000000
> Apr 9 14:16:21 host kernel: CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b
> Apr 9 14:16:21 host kernel: CR2: 00007ff39e4a7000 CR3: 000000007e2a6000 CR4: 0000000000002660
> Apr 9 14:16:21 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> Apr 9 14:16:21 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> Apr 9 14:16:21 host kernel: Call Trace:
> Apr 9 14:16:21 host kernel: [<ffffffff8100ddb7>] ? xen_safe_halt+0xc/0x15
> Apr 9 14:16:21 host kernel: [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40
> Apr 9 14:16:21 host kernel: [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82
> Apr 9 14:16:21 host kernel: [<ffffffff818b6c42>] ? start_kernel+0x353/0x35f
> Apr 9 15:40:21 host kernel: BUG: soft lockup - CPU#1 stuck for 4104s! [swapper:0]
> Apr 9 15:40:21 host kernel: Modules linked in:
> Apr 9 15:40:21 host kernel: CPU 1:
> Apr 9 15:40:21 host kernel: Modules linked in:
> Apr 9 15:40:21 host kernel: Pid: 0, comm: swapper Not tainted 2.6.32.9xls-domU #2
> Apr 9 15:40:21 host kernel: RIP: e030:[<ffffffff810093aa>] [<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
> Apr 9 15:40:21 host kernel: RSP: e02b:ffff88007f8a1f08 EFLAGS: 00000246
> Apr 9 15:40:21 host kernel: RAX: 0000000000000000 RBX: ffff88007f8a0000 RCX: ffffffff810093aa
> Apr 9 15:40:21 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000001
> Apr 9 15:40:21 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000 R09: ffffffff8100e3b2
> Apr 9 15:40:21 host kernel: R10: 0000000000000001 R11: 0000000000000246 R12: 0000000000000000
> Apr 9 15:40:21 host kernel: R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
> Apr 9 15:40:21 host kernel: FS: 00007fd75c048710(0000) GS:ffff8800022c7000(0000) knlGS:0000000000000000
> Apr 9 15:40:21 host kernel: CS: e033 DS: 002b ES: 002b CR0: 000000008005003b
> Apr 9 15:40:21 host kernel: CR2: 000000000040d0e2 CR3: 0000000071c86000 CR4: 0000000000002660
> Apr 9 15:40:21 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> Apr 9 15:40:21 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> Apr 9 15:40:21 host kernel: Call Trace:
> Apr 9 15:40:21 host kernel: [<ffffffff8100ddb7>] ? xen_safe_halt+0xc/0x15
> Apr 9 15:40:21 host kernel: [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40
> Apr 9 15:40:21 host kernel: [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82
> Apr 9 15:40:21 host kernel: BUG: soft lockup - CPU#0 stuck for 4104s! [swapper:0]
> Apr 9 15:40:21 host kernel: Modules linked in:
> Apr 9 15:40:21 host kernel: CPU 0:
> Apr 9 15:40:21 host kernel: Modules linked in:
> Apr 9 15:40:21 host kernel: Pid: 0, comm: swapper Not tainted 2.6.32.9xls-domU #2
> Apr 9 15:40:21 host kernel: RIP: e030:[<ffffffff810093aa>] [<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
> Apr 9 15:40:21 host kernel: RSP: e02b:ffffffff81691f70 EFLAGS: 00000246
> Apr 9 15:40:21 host kernel: RAX: 0000000000000000 RBX: ffffffff81690000 RCX: ffffffff810093aa
> Apr 9 15:40:21 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000001
> Apr 9 15:40:21 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000 R09: ffffffff8100e3b2
> Apr 9 15:40:21 host kernel: R10: 0000000000000001 R11: 0000000000000246 R12: ffffffffffffffff
> Apr 9 15:40:21 host kernel: R13: ffffffff818ebf20 R14: ffffffff818eec70 R15: 0000000000000000
> Apr 9 15:40:21 host kernel: FS: 00007f07704fe6e0(0000) GS:ffff8800022ac000(0000) knlGS:0000000000000000
> Apr 9 15:40:21 host kernel: CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b
> Apr 9 15:40:21 host kernel: CR2: 00000000004c04a8 CR3: 0000000071c86000 CR4: 0000000000002660
> Apr 9 15:40:21 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> Apr 9 15:40:21 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> Apr 9 15:40:21 host kernel: Call Trace:
> Apr 9 15:40:21 host kernel: [<ffffffff8100ddb7>] ? xen_safe_halt+0xc/0x15
> Apr 9 15:40:21 host kernel: [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40
> Apr 9 15:40:21 host kernel: [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82
> Apr 9 15:40:21 host kernel: [<ffffffff818b6c42>] ? start_kernel+0x353/0x35f
> Apr 9 16:06:21 host kernel: BUG: soft lockup - CPU#1 stuck for 1452s! [swapper:0]
> Apr 9 16:06:21 host kernel: Modules linked in:
> Apr 9 16:06:21 host kernel: CPU 1:
> Apr 9 16:06:21 host kernel: Modules linked in:
> Apr 9 16:06:21 host kernel: Pid: 0, comm: swapper Not tainted 2.6.32.9xls-domU #2
> Apr 9 16:06:21 host kernel: RIP: e030:[<ffffffff810093aa>] [<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
> Apr 9 16:06:21 host kernel: RSP: e02b:ffff88007f8a1f08 EFLAGS: 00000246
> Apr 9 16:06:21 host kernel: RAX: 0000000000000000 RBX: ffff88007f8a0000 RCX: ffffffff810093aa
> Apr 9 16:06:21 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000001
> Apr 9 16:06:21 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000 R09: ffffffff8100e3b2
> Apr 9 16:06:21 host kernel: R10: 0000000000000001 R11: 0000000000000246 R12: 0000000000000000
> Apr 9 16:06:21 host kernel: R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
> Apr 9 16:06:21 host kernel: FS: 00007ff39e495710(0000) GS:ffff8800022c7000(0000) knlGS:0000000000000000
> Apr 9 16:06:25 host kernel: CS: e033 DS: 002b ES: 002b CR0: 000000008005003b
> Apr 9 16:06:25 host kernel: CR2: 00007fbd3c012909 CR3: 0000000071c86000 CR4: 0000000000002660
> Apr 9 16:06:26 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> Apr 9 16:06:26 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> Apr 9 16:06:26 host kernel: Call Trace:
> Apr 9 16:06:26 host kernel: [<ffffffff8100ddb7>] ? xen_safe_halt+0xc/0x15
> Apr 9 16:06:26 host kernel: [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40
> Apr 9 16:06:26 host kernel: [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82
> Apr 9 16:06:26 host kernel: BUG: soft lockup - CPU#0 stuck for 1452s! [swapper:0]
> Apr 9 16:06:26 host kernel: Modules linked in:
> Apr 9 16:06:26 host kernel: CPU 0:
> Apr 9 16:06:26 host kernel: Modules linked in:
> Apr 9 16:06:26 host kernel: Pid: 0, comm: swapper Not tainted 2.6.32.9xls-domU #2
> Apr 9 16:06:26 host kernel: RIP: e030:[<ffffffff810093aa>] [<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
> Apr 9 16:06:26 host kernel: RSP: e02b:ffffffff81691f70 EFLAGS: 00000246
> Apr 9 16:06:26 host kernel: RAX: 0000000000000000 RBX: ffffffff81690000 RCX: ffffffff810093aa
> Apr 9 16:06:26 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000001
> Apr 9 16:06:26 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000 R09: ffffffff8100e3b2
> Apr 9 16:06:26 host kernel: R10: 0000000000000001 R11: 0000000000000246 R12: ffffffffffffffff
> Apr 9 16:06:26 host kernel: R13: ffffffff818ebf20 R14: ffffffff818eec70 R15: 0000000000000000
> Apr 9 16:06:26 host kernel: FS: 00007f07704fe6e0(0000) GS:ffff8800022ac000(0000) knlGS:0000000000000000
> Apr 9 16:06:26 host kernel: CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b
> Apr 9 16:06:26 host kernel: CR2: 00007f09a4413000 CR3: 000000007ea99000 CR4: 0000000000002660
> Apr 9 16:06:26 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> Apr 9 16:06:26 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> Apr 9 16:06:26 host kernel: Call Trace:
> Apr 9 16:06:26 host kernel: [<ffffffff8100ddb7>] ? xen_safe_halt+0xc/0x15
> Apr 9 16:06:26 host kernel: [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40
> Apr 9 16:06:26 host kernel: [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82
> Apr 9 16:06:26 host kernel: [<ffffffff818b6c42>] ? start_kernel+0x353/0x35f
> Apr 9 16:17:43 host kernel: BUG: soft lockup - CPU#1 stuck for 637s! [httpd:32342]
> Apr 9 16:17:43 host kernel: Modules linked in:
> Apr 9 16:17:43 host kernel: CPU 1:
> Apr 9 16:17:43 host kernel: Modules linked in:
> Apr 9 16:17:43 host kernel: Pid: 32342, comm: httpd Not tainted 2.6.32.9xls-domU #2
> Apr 9 16:17:43 host kernel: RIP: e030:[<ffffffff8100922a>] [<ffffffff8100922a>] hypercall_page+0x22a/0x1001
> Apr 9 16:17:43 host kernel: RSP: e02b:ffff88007955fea0 EFLAGS: 00000246
> Apr 9 16:17:43 host kernel: RAX: 0000000000030004 RBX: 0000000000000000 RCX: ffffffff8100922a
> Apr 9 16:17:43 host kernel: RDX: 000000000002ba30 RSI: 0000000000000000 RDI: 0000000000000000
> Apr 9 16:17:43 host kernel: RBP: 000000000394ca98 R08: 0000000000033240 R09: 000000310f6de059
> Apr 9 16:17:43 host kernel: R10: 0000000000000001 R11: 0000000000000246 R12: 0000000000000006
> Apr 9 16:17:43 host kernel: R13: ffff88007955ff58 R14: ffff88007f9a8ab0 R15: ffff88007d093b40
> Apr 9 16:17:43 host kernel: FS: 00007ff39e495710(0000) GS:ffff8800022c7000(0000) knlGS:0000000000000000
> Apr 9 16:17:43 host kernel: CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b
> Apr 9 16:17:43 host kernel: CR2: 000000000394ca98 CR3: 000000006adde000 CR4: 0000000000002660
> Apr 9 16:17:43 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> Apr 9 16:17:43 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> Apr 9 16:17:43 host kernel: Call Trace:
> Apr 9 16:17:43 host kernel: [<ffffffff8100dd39>] ? xen_force_evtchn_callback+0x9/0xa
> Apr 9 16:17:43 host kernel: [<ffffffff8100e3b2>] ? check_events+0x12/0x20
> Apr 9 16:17:43 host kernel: [<ffffffff8100e359>] ? xen_irq_enable_direct_end+0x0/0x7
> Apr 9 16:17:43 host kernel: [<ffffffff814da477>] ? do_page_fault+0xfb/0x28d
> Apr 9 16:17:43 host kernel: [<ffffffff814d85b5>] ? page_fault+0x25/0x30
> Apr 9 16:17:43 host kernel: BUG: soft lockup - CPU#0 stuck for 637s! [swapper:0]
> Apr 9 16:17:43 host kernel: Modules linked in:
> Apr 9 16:17:43 host kernel: CPU 0:
> Apr 9 16:17:43 host kernel: Modules linked in:
> Apr 9 16:17:43 host kernel: Pid: 0, comm: swapper Not tainted 2.6.32.9xls-domU #2
> Apr 9 16:17:43 host kernel: RIP: e030:[<ffffffff810093aa>] [<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
> Apr 9 16:17:44 host kernel: RSP: e02b:ffffffff81691f70 EFLAGS: 00000246
> Apr 9 16:17:44 host kernel: RAX: 0000000000000000 RBX: ffffffff81690000 RCX: ffffffff810093aa
> Apr 9 16:17:44 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000001
> Apr 9 16:17:44 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000 R09: ffffffff8100e3b2
> Apr 9 16:17:44 host kernel: R10: 0000000000000001 R11: 0000000000000246 R12: ffffffffffffffff
> Apr 9 16:17:44 host kernel: R13: ffffffff818ebf20 R14: ffffffff818eec70 R15: 0000000000000000
> Apr 9 16:17:44 host kernel: FS: 00007f07704fe6e0(0000) GS:ffff8800022ac000(0000) knlGS:0000000000000000
> Apr 9 16:17:44 host kernel: CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b
> Apr 9 16:17:44 host kernel: CR2: 00000000004505b0 CR3: 000000007ea99000 CR4: 0000000000002660
> Apr 9 16:17:44 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> Apr 9 16:17:44 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> Apr 9 16:17:44 host kernel: Call Trace:
> Apr 9 16:17:44 host kernel: [<ffffffff8100ddb7>] ? xen_safe_halt+0xc/0x15
> Apr 9 16:17:44 host kernel: [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40
> Apr 9 16:17:44 host kernel: [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82
> Apr 9 16:17:44 host kernel: [<ffffffff818b6c42>] ? start_kernel+0x353/0x35f
> Apr 9 16:20:22 host kernel: BUG: soft lockup - CPU#1 stuck for 147s! [swapper:0]
> Apr 9 16:20:22 host kernel: Modules linked in:
> Apr 9 16:20:22 host kernel: CPU 1:
> Apr 9 16:20:22 host kernel: Modules linked in:
> Apr 9 16:20:22 host kernel: Pid: 0, comm: swapper Not tainted 2.6.32.9xls-domU #2
> Apr 9 16:20:22 host kernel: RIP: e030:[<ffffffff810093aa>] [<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
> Apr 9 16:20:22 host kernel: RSP: e02b:ffff88007f8a1f08 EFLAGS: 00000246
> Apr 9 16:20:22 host kernel: RAX: 0000000000000000 RBX: ffff88007f8a0000 RCX: ffffffff810093aa
> Apr 9 16:20:22 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000001
> Apr 9 16:20:22 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000 R09: ffffffff8100e3b2
> Apr 9 16:20:22 host kernel: R10: 0000000000000001 R11: 0000000000000246 R12: 0000000000000000
> Apr 9 16:20:22 host kernel: R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
> Apr 9 16:20:22 host kernel: FS: 00007ff39e495710(0000) GS:ffff8800022c7000(0000) knlGS:0000000000000000
> Apr 9 16:20:22 host kernel: CS: e033 DS: 002b ES: 002b CR0: 000000008005003b
> Apr 9 16:20:22 host kernel: CR2: 00007ff39aec11a7 CR3: 0000000071c86000 CR4: 0000000000002660
> Apr 9 16:20:22 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> Apr 9 16:20:22 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> Apr 9 16:20:22 host kernel: Call Trace:
> Apr 9 16:20:22 host kernel: [<ffffffff8100ddb7>] ? xen_safe_halt+0xc/0x15
> Apr 9 16:20:22 host kernel: [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40
> Apr 9 16:20:22 host kernel: [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82
> Apr 9 16:20:22 host kernel: BUG: soft lockup - CPU#0 stuck for 147s! [swapper:0]
> Apr 9 16:20:22 host kernel: Modules linked in:
> Apr 9 16:20:22 host kernel: CPU 0:
> Apr 9 16:20:22 host kernel: Modules linked in:
> Apr 9 16:20:22 host kernel: Pid: 0, comm: swapper Not tainted 2.6.32.9xls-domU #2
> Apr 9 16:20:22 host kernel: RIP: e030:[<ffffffff810093aa>] [<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
> Apr 9 16:20:22 host kernel: RSP: e02b:ffffffff81691f70 EFLAGS: 00000246
> Apr 9 16:20:22 host kernel: RAX: 0000000000000000 RBX: ffffffff81690000 RCX: ffffffff810093aa
> Apr 9 16:20:22 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000001
> Apr 9 16:20:22 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000 R09: ffffffff8100e3b2
> Apr 9 16:20:22 host kernel: R10: 0000000000000001 R11: 0000000000000246 R12: ffffffffffffffff
> Apr 9 16:20:22 host kernel: R13: ffffffff818ebf20 R14: ffffffff818eec70 R15: 0000000000000000
> Apr 9 16:20:22 host kernel: FS: 00007f07704fe6e0(0000) GS:ffff8800022ac000(0000) knlGS:0000000000000000
> Apr 9 16:20:22 host kernel: CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b
> Apr 9 16:20:22 host kernel: CR2: 00000000006ddc20 CR3: 0000000071c86000 CR4: 0000000000002660
> Apr 9 16:20:22 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> Apr 9 16:20:22 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> Apr 9 16:20:22 host kernel: Call Trace:
> Apr 9 16:20:22 host kernel: [<ffffffff8100ddb7>] ? xen_safe_halt+0xc/0x15
> Apr 9 16:20:22 host kernel: [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40
> Apr 9 16:20:22 host kernel: [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82
> Apr 9 16:20:22 host kernel: [<ffffffff818b6c42>] ? start_kernel+0x353/0x35f
> Apr 9 16:22:29 host kernel: BUG: soft lockup - CPU#1 stuck for 119s! [swapper:0]
> Apr 9 16:22:29 host kernel: Modules linked in:
> Apr 9 16:22:29 host kernel: CPU 1:
> Apr 9 16:22:29 host kernel: Modules linked in:
> Apr 9 16:22:29 host kernel: Pid: 0, comm: swapper Not tainted 2.6.32.9xls-domU #2
> Apr 9 16:22:29 host kernel: RIP: e030:[<ffffffff810093aa>] [<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
> Apr 9 16:22:29 host kernel: RSP: e02b:ffff88007f8a1f08 EFLAGS: 00000246
> Apr 9 16:22:29 host kernel: RAX: 0000000000000000 RBX: ffff88007f8a0000 RCX: ffffffff810093aa
> Apr 9 16:22:29 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000001
> Apr 9 16:22:29 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000 R09: ffffffff8100e3b2
> Apr 9 16:22:29 host kernel: R10: 0000000000000001 R11: 0000000000000246 R12: 0000000000000000
> Apr 9 16:22:29 host kernel: R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
> Apr 9 16:22:29 host kernel: FS: 00007ff39e495710(0000) GS:ffff8800022c7000(0000) knlGS:0000000000000000
> Apr 9 16:22:29 host kernel: CS: e033 DS: 002b ES: 002b CR0: 000000008005003b
> Apr 9 16:22:29 host kernel: CR2: 00007ff39c506460 CR3: 0000000026469000 CR4: 0000000000002660
> Apr 9 16:22:29 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> Apr 9 16:22:29 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> Apr 9 16:22:29 host kernel: Call Trace:
> Apr 9 16:22:29 host kernel: [<ffffffff8100ddb7>] ? xen_safe_halt+0xc/0x15
> Apr 9 16:22:29 host kernel: [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40
> Apr 9 16:22:29 host kernel: [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82
> Apr 9 16:22:29 host kernel: BUG: soft lockup - CPU#0 stuck for 119s! [syslogd:1848]
> Apr 9 16:22:29 host kernel: Modules linked in:
> Apr 9 16:22:29 host kernel: CPU 0:
> Apr 9 16:22:29 host kernel: Modules linked in:
> Apr 9 16:22:29 host kernel: Pid: 1848, comm: syslogd Not tainted 2.6.32.9xls-domU #2
> Apr 9 16:22:29 host kernel: RIP: 0033:[<00007f74fd9e4ad5>] [<00007f74fd9e4ad5>] 0x7f74fd9e4ad5
> Apr 9 16:22:29 host kernel: RSP: 002b:00007fff22fa00d0 EFLAGS: 00000246
> Apr 9 16:22:29 host kernel: RAX: 000000000000002e RBX: 0000000000000000 RCX: ffffffff810092eb
> Apr 9 16:22:29 host kernel: RDX: 00000000000003fe RSI: 00007fff22fa01b0 RDI: 0000000000000000
> Apr 9 16:22:30 host kernel: RBP: 00000000000003fe R08: 0000000000000000 R09: 0000000000000000
> Apr 9 16:22:30 host kernel: R10: 0000000000000000 R11: 0000000000000246 R12: 00007fff22fa01b0
> Apr 9 16:22:30 host kernel: R13: 0000000000000000 R14: 00007fff22fa0120 R15: 00007f74fde74690
> Apr 9 16:22:30 host kernel: FS: 00007f74fde746e0(0000) GS:ffff8800022ac000(0000) knlGS:0000000000000000
> Apr 9 16:22:30 host kernel: CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b
> Apr 9 16:22:30 host kernel: CR2: 00007f0e429d6000 CR3: 000000007d03a000 CR4: 0000000000002660
> Apr 9 16:22:30 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> Apr 9 16:22:30 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> Apr 9 16:22:30 host kernel: Call Trace:
On Apr 9, 2010, at 15:22 , Pim van Riezen wrote:
> Good day,
>
> I'm getting lots of these from a couple of pvops DomU guests that are pretty heavy in their i/o and cpu-load. The issue reproduces on pvops 2.6.31 and 2.6.32 trees under Xen 3.3 as well as 3.4.1 (in the process of trying out 3.4.2 but I'm not holding my breath). Is there anything we can do be more helpful towards solving this issue?
>
> BUG: soft lockup - CPU#0 stuck for 561s! [swapper:0]
> Modules linked in:
> CPU 0:
> Modules linked in:
> Pid: 0, comm: swapper Not tainted 2.6.32.9xls-domU #2
> RIP: e030:[<ffffffff810093aa>] [<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
> RSP: e02b:ffffffff81691f70 EFLAGS: 00000246
> RAX: 0000000000000000 RBX: ffffffff81690000 RCX: ffffffff810093aa
> RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000001
> RBP: ffffffff81896d30 R08: 0000000000000000 R09: ffffffff8100e3b2
> R10: 0000000000000001 R11: 0000000000000246 R12: ffffffffffffffff
> R13: ffffffff818ebf20 R14: ffffffff818eec70 R15: 0000000000000000
> FS: 00007f8ac7a9c6e0(0000) GS:ffff8800022ac000(0000) knlGS:0000000000000000
> CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b
> CR2: 00007f291b4c9000 CR3: 000000007d8c1000 CR4: 0000000000002660
> DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> Call Trace:
> [<ffffffff8100ddb7>] ? xen_safe_halt+0xc/0x15
> [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40
> [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82
> [<ffffffff818b6c42>] ? start_kernel+0x353/0x35f
>
> Kind regards,
> Pim van Riezen
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: pvops domu soft lockup under load (more logs)
2010-04-09 16:50 ` pvops domu soft lockup under load (more logs) Pim van Riezen
@ 2010-04-09 18:43 ` Jeremy Fitzhardinge
2010-04-10 7:12 ` Pim van Riezen
0 siblings, 1 reply; 17+ messages in thread
From: Jeremy Fitzhardinge @ 2010-04-09 18:43 UTC (permalink / raw)
To: Pim van Riezen; +Cc: xen-devel
On 04/09/2010 09:50 AM, Pim van Riezen wrote:
> Some more logs of this bug in action. Stack seems to be all over the place.
>
What kernel is this? Did you build it, or does it come from a distro?
What devices are you using within the domain? What storage backend are
you using?
Could you include the full kernel output?
When you're getting lockups, could you capture the output of sysrq-t (xm
sysrq <domain> t, or "Ctrl-O t" on the domain's console)?
Thanks,
J
>
>> Apr 9 08:21:01 host kernel: BUG: soft lockup - CPU#1 stuck for 628s! [gzip:27658]
>> Apr 9 08:21:01 host kernel: Modules linked in:
>> Apr 9 08:21:01 host kernel: CPU 1:
>> Apr 9 08:21:01 host kernel: Modules linked in:
>> Apr 9 08:21:01 host kernel: Pid: 27658, comm: gzip Not tainted 2.6.32.9xls-domU #2
>> Apr 9 08:21:01 host kernel: RIP: e033:[<0000000000401958>] [<0000000000401958>] 0x401958
>> Apr 9 08:21:01 host kernel: RSP: e02b:00007fff026d7e38 EFLAGS: 00000297
>> Apr 9 08:21:01 host kernel: RAX: 0000000000000020 RBX: 0000000000000002 RCX: 0000000000657ce8
>> Apr 9 08:21:01 host kernel: RDX: 0000000000000048 RSI: 0000000000000002 RDI: 000000000000da48
>> Apr 9 08:21:01 host kernel: RBP: 0000000000006a6c R08: 0000000000000034 R09: 000000000000da48
>> Apr 9 08:21:01 host kernel: R10: 0000000000000048 R11: 0000000000000080 R12: 0000000000658c06
>> Apr 9 08:21:01 host kernel: R13: 0000000000658d08 R14: 0000000000658c06 R15: 0000000000000080
>> Apr 9 08:21:01 host kernel: FS: 00007fa6088946e0(0000) GS:ffff8800022c7000(0000) knlGS:0000000000000000
>> Apr 9 08:21:01 host kernel: CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b
>> Apr 9 08:21:01 host kernel: CR2: 00007f7addede020 CR3: 00000000267db000 CR4: 0000000000002660
>> Apr 9 08:21:01 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
>> Apr 9 08:21:02 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
>> Apr 9 08:21:02 host kernel: Call Trace:
>> Apr 9 08:29:39 host kernel: BUG: soft lockup - CPU#1 stuck for 483s! [swapper:0]
>> Apr 9 08:29:39 host kernel: Modules linked in:
>> Apr 9 08:29:39 host kernel: CPU 1:
>> Apr 9 08:29:39 host kernel: Modules linked in:
>> Apr 9 08:29:39 host kernel: Pid: 0, comm: swapper Not tainted 2.6.32.9xls-domU #2
>> Apr 9 08:29:39 host kernel: RIP: e030:[<ffffffff810093aa>] [<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
>> Apr 9 08:29:39 host kernel: RSP: e02b:ffff88007f8a1f08 EFLAGS: 00000246
>> Apr 9 08:29:39 host kernel: RAX: 0000000000000000 RBX: ffff88007f8a0000 RCX: ffffffff810093aa
>> Apr 9 08:29:39 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000001
>> Apr 9 08:29:39 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000 R09: ffffffff8100e3b2
>> Apr 9 08:29:39 host kernel: R10: 0000000000000001 R11: 0000000000000246 R12: 0000000000000000
>> Apr 9 08:29:39 host kernel: R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
>> Apr 9 08:29:39 host kernel: FS: 00007fa913f42720(0000) GS:ffff8800022c7000(0000) knlGS:0000000000000000
>> Apr 9 08:29:39 host kernel: CS: e033 DS: 002b ES: 002b CR0: 000000008005003b
>> Apr 9 08:29:39 host kernel: CR2: 00007ff39862b1f0 CR3: 0000000052c5e000 CR4: 0000000000002660
>> Apr 9 08:29:39 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
>> Apr 9 08:29:39 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
>> Apr 9 08:29:39 host kernel: Call Trace:
>> Apr 9 08:29:39 host kernel: [<ffffffff8100ddb7>] ? xen_safe_halt+0xc/0x15
>> Apr 9 08:29:39 host kernel: [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40
>> Apr 9 08:29:39 host kernel: [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82
>> Apr 9 08:29:39 host kernel: BUG: soft lockup - CPU#0 stuck for 483s! [swapper:0]
>> Apr 9 08:29:39 host kernel: Modules linked in:
>> Apr 9 08:29:39 host kernel: CPU 0:
>> Apr 9 08:29:40 host kernel: Modules linked in:
>> Apr 9 08:29:40 host kernel: Pid: 0, comm: swapper Not tainted 2.6.32.9xls-domU #2
>> Apr 9 08:29:40 host kernel: RIP: e030:[<ffffffff810093aa>] [<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
>> Apr 9 08:29:40 host kernel: RSP: e02b:ffffffff81691f70 EFLAGS: 00000246
>> Apr 9 08:29:40 host kernel: RAX: 0000000000000000 RBX: ffffffff81690000 RCX: ffffffff810093aa
>> Apr 9 08:29:40 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000001
>> Apr 9 08:29:40 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000 R09: ffffffff8100e3b2
>> Apr 9 08:29:40 host kernel: R10: 0000000000000001 R11: 0000000000000246 R12: ffffffffffffffff
>> Apr 9 08:29:40 host kernel: R13: ffffffff818ebf20 R14: ffffffff818eec70 R15: 0000000000000000
>> Apr 9 08:29:40 host kernel: FS: 00007f07704fe6e0(0000) GS:ffff8800022ac000(0000) knlGS:0000000000000000
>> Apr 9 08:29:40 host kernel: CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b
>> Apr 9 08:29:40 host kernel: CR2: 00007ff39e4a8000 CR3: 000000007ea99000 CR4: 0000000000002660
>> Apr 9 08:29:40 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
>> Apr 9 08:29:40 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
>> Apr 9 08:29:40 host kernel: Call Trace:
>> Apr 9 08:29:40 host kernel: [<ffffffff8100ddb7>] ? xen_safe_halt+0xc/0x15
>> Apr 9 08:29:40 host kernel: [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40
>> Apr 9 08:29:40 host kernel: [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82
>> Apr 9 08:29:40 host kernel: [<ffffffff818b6c42>] ? start_kernel+0x353/0x35f
>> Apr 9 08:34:34 host kernel: BUG: soft lockup - CPU#1 stuck for 275s! [ssh:2324]
>> Apr 9 08:34:34 host kernel: Modules linked in:
>> Apr 9 08:34:34 host kernel: CPU 1:
>> Apr 9 08:34:34 host kernel: Modules linked in:
>> Apr 9 08:34:34 host kernel: Pid: 2324, comm: ssh Not tainted 2.6.32.9xls-domU #2
>> Apr 9 08:34:34 host kernel: RIP: e033:[<00007f1b0fcac76a>] [<00007f1b0fcac76a>] 0x7f1b0fcac76a
>> Apr 9 08:34:34 host kernel: RSP: e02b:00007fffa45a7480 EFLAGS: 00010202
>> Apr 9 08:34:34 host kernel: RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000181
>> Apr 9 08:34:34 host kernel: RDX: 00007f1b0fcdca28 RSI: 0000000000000013 RDI: 000000000000000a
>> Apr 9 08:34:34 host kernel: RBP: 0000000000000010 R08: 0000000000000004 R09: 0101010101010101
>> Apr 9 08:34:34 host kernel: R10: 737361702e707466 R11: 00007f1b0fcac690 R12: 0000000000000010
>> Apr 9 08:34:34 host kernel: R13: 00007f1b11f825e0 R14: 0000000000000001 R15: 0000000000000004
>> Apr 9 08:34:34 host kernel: FS: 00007f1b10528710(0000) GS:ffff8800022c7000(0000) knlGS:0000000000000000
>> Apr 9 08:34:34 host kernel: CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b
>> Apr 9 08:34:34 host kernel: CR2: 00007fa913f54000 CR3: 000000006ad44000 CR4: 0000000000002660
>> Apr 9 08:34:34 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
>> Apr 9 08:34:34 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
>> Apr 9 08:34:34 host kernel: Call Trace:
>> Apr 9 08:34:34 host kernel: BUG: soft lockup - CPU#0 stuck for 275s! [swapper:0]
>> Apr 9 08:34:34 host kernel: Modules linked in:
>> Apr 9 08:34:34 host kernel: CPU 0:
>> Apr 9 08:34:34 host kernel: Modules linked in:
>> Apr 9 08:34:34 host kernel: Pid: 0, comm: swapper Not tainted 2.6.32.9xls-domU #2
>> Apr 9 08:34:34 host kernel: RIP: e030:[<ffffffff810093aa>] [<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
>> Apr 9 08:34:34 host kernel: RSP: e02b:ffffffff81691f70 EFLAGS: 00000246
>> Apr 9 08:34:34 host kernel: RAX: 0000000000000000 RBX: ffffffff81690000 RCX: ffffffff810093aa
>> Apr 9 08:34:34 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000001
>> Apr 9 08:34:34 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000 R09: ffffffff8100e3b2
>> Apr 9 08:34:34 host kernel: R10: 0000000000000001 R11: 0000000000000246 R12: ffffffffffffffff
>> Apr 9 08:34:34 host kernel: R13: ffffffff818ebf20 R14: ffffffff818eec70 R15: 0000000000000000
>> Apr 9 08:34:34 host kernel: FS: 00007fa913f42720(0000) GS:ffff8800022ac000(0000) knlGS:0000000000000000
>> Apr 9 08:34:34 host kernel: CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b
>> Apr 9 08:34:34 host kernel: CR2: 00007fa913f54000 CR3: 000000006acb1000 CR4: 0000000000002660
>> Apr 9 08:34:34 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
>> Apr 9 08:34:34 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
>> Apr 9 08:34:34 host kernel: Call Trace:
>> Apr 9 08:34:34 host kernel: [<ffffffff8100ddb7>] ? xen_safe_halt+0xc/0x15
>> Apr 9 08:34:34 host kernel: [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40
>> Apr 9 08:34:34 host kernel: [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82
>> Apr 9 08:34:34 host kernel: [<ffffffff818b6c42>] ? start_kernel+0x353/0x35f
>> Apr 9 08:39:49 host kernel: BUG: soft lockup - CPU#1 stuck for 293s! [gzip:30581]
>> Apr 9 08:39:49 host kernel: Modules linked in:
>> Apr 9 08:39:49 host kernel: CPU 1:
>> Apr 9 08:39:49 host kernel: Modules linked in:
>> Apr 9 08:39:49 host kernel: Pid: 30581, comm: gzip Not tainted 2.6.32.9xls-domU #2
>> Apr 9 08:39:49 host kernel: RIP: e030:[<ffffffff811ede38>] [<ffffffff811ede38>] fuse_request_send+0x60/0x22b
>> Apr 9 08:39:49 host kernel: RSP: e02b:ffff88007952bc28 EFLAGS: 00000202
>> Apr 9 08:39:49 host kernel: RAX: ffff880079652df0 RBX: ffff88007e07d2c0 RCX: fffffffffffdfb5e
>> Apr 9 08:39:49 host kernel: RDX: 0000000000020001 RSI: 000000000000001d RDI: ffff88007eb1f710
>> Apr 9 08:39:49 host kernel: RBP: ffff88007e07d2c0 R08: 000000037ede1c40 R09: ffff8800593170d8
>> Apr 9 08:39:49 host kernel: R10: 0000000000000000 R11: 000000000000000f R12: ffff88007eb1f400
>> Apr 9 08:39:49 host kernel: R13: ffff88007e07d2e0 R14: 0000000000001000 R15: 0000000000001000
>> Apr 9 08:39:49 host kernel: FS: 00007f7b12f606e0(0000) GS:ffff8800022c7000(0000) knlGS:0000000000000000
>> Apr 9 08:39:49 host kernel: CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b
>> Apr 9 08:39:49 host kernel: CR2: 00000000006d9138 CR3: 00000000264f7000 CR4: 0000000000002660
>> Apr 9 08:39:49 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
>> Apr 9 08:39:49 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
>> Apr 9 08:39:49 host kernel: Call Trace:
>> Apr 9 08:39:49 host kernel: [<ffffffff811ede34>] ? fuse_request_send+0x5c/0x22b
>> Apr 9 08:39:49 host kernel: [<ffffffff811f1655>] ? fuse_wait_on_page_writeback+0x31/0x168
>> Apr 9 08:39:49 host kernel: [<ffffffff810904b4>] ? add_to_page_cache_lru+0x41/0x58
>> Apr 9 08:39:49 host kernel: [<ffffffff811f186a>] ? fuse_send_write+0xb8/0xc2
>> Apr 9 08:39:49 host kernel: [<ffffffff811f29d7>] ? fuse_file_aio_write+0x497/0x62f
>> Apr 9 08:39:49 host kernel: [<ffffffff810bab7e>] ? do_sync_write+0xce/0x113
>> Apr 9 08:39:49 host kernel: [<ffffffff8100f6af>] ? __switch_to+0x1ad/0x297
>> Apr 9 08:39:49 host kernel: [<ffffffff8100e3b2>] ? check_events+0x12/0x20
>> Apr 9 08:39:49 host kernel: [<ffffffff810561b8>] ? autoremove_wake_function+0x0/0x2e
>> Apr 9 08:39:49 host kernel: [<ffffffff8103b070>] ? finish_task_switch+0x27/0x6e
>> Apr 9 08:39:49 host kernel: [<ffffffff814d6a77>] ? thread_return+0x3e/0x109
>> Apr 9 08:39:49 host kernel: [<ffffffff810bb4b2>] ? vfs_write+0xad/0x149
>> Apr 9 08:39:49 host kernel: [<ffffffff810bb60a>] ? sys_write+0x45/0x6e
>> Apr 9 08:39:49 host kernel: [<ffffffff81010ac2>] ? system_call_fastpath+0x16/0x1b
>> Apr 9 08:39:49 host kernel: BUG: soft lockup - CPU#0 stuck for 293s! [sshfs:2332]
>> Apr 9 08:39:49 host kernel: Modules linked in:
>> Apr 9 08:39:49 host kernel: CPU 0:
>> Apr 9 08:39:49 host kernel: Modules linked in:
>> Apr 9 08:39:49 host kernel: Pid: 2332, comm: sshfs Not tainted 2.6.32.9xls-domU #2
>> Apr 9 08:39:49 host kernel: RIP: e030:[<ffffffff813b93bd>] [<ffffffff813b93bd>] sk_reset_timer+0x1/0x1d
>> Apr 9 08:39:49 host kernel: RSP: e02b:ffff8800022afbc0 EFLAGS: 00000206
>> Apr 9 08:39:49 host kernel: RAX: 0000000101b312c6 RBX: 0000000000000080 RCX: 0000000101b312ce
>> Apr 9 08:39:49 host kernel: RDX: 0000000101b312ce RSI: ffff88007d1dc4d0 RDI: ffff88007d1dc1c0
>> Apr 9 08:39:49 host kernel: RBP: ffff88007d1dc1c0 R08: 000000007d1dc10f R09: 0000000101b312ce
>> Apr 9 08:39:49 host kernel: R10: ffff880068b8f1c0 R11: ffffffff8126167a R12: ffff880068b8f1c0
>> Apr 9 08:39:49 host kernel: R13: 0000000000000000 R14: 0000000000000020 R15: ffff880068b8f1f8
>> Apr 9 08:39:49 host kernel: FS: 00007f1b10528710(0063) GS:ffff8800022ac000(0000) knlGS:0000000000000000
>> Apr 9 08:39:49 host kernel: CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b
>> Apr 9 08:39:49 host kernel: CR2: 00007f66a4ede000 CR3: 000000006acb1000 CR4: 0000000000002660
>> Apr 9 08:39:49 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
>> Apr 9 08:39:49 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
>> Apr 9 08:39:49 host kernel: Call Trace:
>> Apr 9 08:39:49 host kernel: <IRQ> [<ffffffff81419140>] ? tcp_rcv_established+0x587/0x87b
>> Apr 9 08:39:49 host kernel: [<ffffffff8141f7f1>] ? tcp_v4_do_rcv+0x2c/0x1c9
>> Apr 9 08:39:49 host kernel: [<ffffffff81046156>] ? local_bh_enable+0x22/0x8f
>> Apr 9 08:39:49 host kernel: [<ffffffff8141fe05>] ? tcp_v4_rcv+0x477/0x6f8
>> Apr 9 08:39:49 host kernel: [<ffffffff814064f7>] ? ip_local_deliver_finish+0xf2/0x1c7
>> Apr 9 08:39:49 host kernel: [<ffffffff81406126>] ? ip_rcv_finish+0x326/0x340
>> Apr 9 08:39:49 host kernel: [<ffffffff814063c0>] ? ip_rcv+0x280/0x2c5
>> Apr 9 08:39:49 host kernel: [<ffffffff8135d39d>] ? xennet_poll+0xb06/0xc68
>> Apr 9 08:39:49 host kernel: [<ffffffff8100e39f>] ? xen_restore_fl_direct_end+0x0/0x1
>> Apr 9 08:39:49 host kernel: [<ffffffff81334afc>] ? do_blkif_request+0x341/0x350
>> Apr 9 08:39:49 host kernel: [<ffffffff8100e3b2>] ? check_events+0x12/0x20
>> Apr 9 08:39:49 host kernel: [<ffffffff813c469b>] ? net_rx_action+0x77/0x179
>> Apr 9 08:39:49 host kernel: [<ffffffff810462b4>] ? __do_softirq+0xdd/0x1a1
>> Apr 9 08:39:49 host kernel: [<ffffffff81011bec>] ? call_softirq+0x1c/0x30
>> Apr 9 08:39:49 host kernel: [<ffffffff81012edf>] ? do_softirq+0x3f/0x7c
>> Apr 9 08:39:49 host kernel: [<ffffffff812ff975>] ? xen_evtchn_do_upcall+0x173/0x190
>> Apr 9 08:39:49 host kernel: [<ffffffff81011c3e>] ? xen_do_hypervisor_callback+0x1e/0x30
>> Apr 9 08:39:49 host kernel: <EOI> [<ffffffff8100e3b2>] ? check_events+0x12/0x20
>> Apr 9 08:39:49 host kernel: [<ffffffff8100922a>] ? hypercall_page+0x22a/0x1001
>> Apr 9 08:39:49 host kernel: [<ffffffff8100922a>] ? hypercall_page+0x22a/0x1001
>> Apr 9 08:39:49 host kernel: [<ffffffff8100dd39>] ? xen_force_evtchn_callback+0x9/0xa
>> Apr 9 08:39:49 host kernel: [<ffffffff8100e3b2>] ? check_events+0x12/0x20
>> Apr 9 08:39:49 host kernel: [<ffffffff8100e3b2>] ? check_events+0x12/0x20
>> Apr 9 08:39:49 host kernel: [<ffffffff8100e359>] ? xen_irq_enable_direct_end+0x0/0x7
>> Apr 9 08:39:49 host kernel: [<ffffffff8103b070>] ? finish_task_switch+0x27/0x6e
>> Apr 9 08:39:49 host kernel: [<ffffffff814d6a77>] ? thread_return+0x3e/0x109
>> Apr 9 08:39:49 host kernel: [<ffffffff8103d336>] ? default_wake_function+0x0/0x9
>> Apr 9 08:39:49 host kernel: [<ffffffff81261668>] ? cap_socket_recvmsg+0x0/0x3
>> Apr 9 08:39:49 host kernel: [<ffffffff81261668>] ? cap_socket_recvmsg+0x0/0x3
>> Apr 9 08:39:49 host kernel: [<ffffffff814d6e59>] ? schedule_timeout+0x23/0x1d9
>> Apr 9 08:39:49 host kernel: [<ffffffff8100dd39>] ? xen_force_evtchn_callback+0x9/0xa
>> Apr 9 08:39:49 host kernel: [<ffffffff8100e3b2>] ? check_events+0x12/0x20
>> Apr 9 08:39:49 host kernel: [<ffffffff81261668>] ? cap_socket_recvmsg+0x0/0x3
>> Apr 9 08:39:49 host kernel: [<ffffffff814d824c>] ? _spin_unlock_irqrestore+0xc/0xd
>> Apr 9 08:39:49 host kernel: [<ffffffff81456ebe>] ? unix_stream_recvmsg+0x2c4/0x644
>> Apr 9 08:39:49 host kernel: [<ffffffff810561b8>] ? autoremove_wake_function+0x0/0x2e
>> Apr 9 08:39:49 host kernel: [<ffffffff81457231>] ? unix_stream_recvmsg+0x637/0x644
>> Apr 9 08:39:49 host kernel: [<ffffffff813b6303>] ? sock_aio_read+0x103/0x117
>> Apr 9 08:39:49 host kernel: [<ffffffff810bac91>] ? do_sync_read+0xce/0x113
>> Apr 9 08:39:49 host kernel: [<ffffffff81011c1b>] ? xen_hypervisor_callback+0x1b/0x20
>> Apr 9 08:39:49 host kernel: [<ffffffff810561b8>] ? autoremove_wake_function+0x0/0x2e
>> Apr 9 08:39:49 host kernel: [<ffffffff81261668>] ? cap_socket_recvmsg+0x0/0x3
>> Apr 9 08:39:49 host kernel: [<ffffffff814d824c>] ? _spin_unlock_irqrestore+0xc/0xd
>> Apr 9 08:39:49 host kernel: [<ffffffff81011c1b>] ? xen_hypervisor_callback+0x1b/0x20
>> Apr 9 08:39:49 host kernel: [<ffffffff810bb6f0>] ? vfs_read+0xbd/0x146
>> Apr 9 08:39:49 host kernel: [<ffffffff8100922a>] ? hypercall_page+0x22a/0x1001
>> Apr 9 08:39:49 host kernel: [<ffffffff810bb835>] ? sys_read+0x45/0x6e
>> Apr 9 08:39:49 host kernel: [<ffffffff81010ac2>] ? system_call_fastpath+0x16/0x1b
>> Apr 9 09:11:36 host kernel: BUG: soft lockup - CPU#1 stuck for 1776s! [swapper:0]
>> Apr 9 09:11:36 host kernel: Modules linked in:
>> Apr 9 09:11:36 host kernel: CPU 1:
>> Apr 9 09:11:36 host kernel: Modules linked in:
>> Apr 9 09:11:36 host kernel: Pid: 0, comm: swapper Not tainted 2.6.32.9xls-domU #2
>> Apr 9 09:11:36 host kernel: RIP: e030:[<ffffffff810093aa>] [<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
>> Apr 9 09:11:36 host kernel: RSP: e02b:ffff88007f8a1f08 EFLAGS: 00000246
>> Apr 9 09:11:36 host kernel: RAX: 0000000000000000 RBX: ffff88007f8a0000 RCX: ffffffff810093aa
>> Apr 9 09:11:36 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000001
>> Apr 9 09:11:36 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000 R09: ffffffff8100e3b2
>> Apr 9 09:11:36 host kernel: R10: 0000000000000001 R11: 0000000000000246 R12: 0000000000000000
>> Apr 9 09:11:36 host kernel: R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
>> Apr 9 09:11:36 host kernel: FS: 00007f44bc51f710(0000) GS:ffff8800022c7000(0000) knlGS:0000000000000000
>> Apr 9 09:11:36 host kernel: CS: e033 DS: 002b ES: 002b CR0: 000000008005003b
>> Apr 9 09:11:36 host kernel: CR2: 00007f44bc52c000 CR3: 0000000071c86000 CR4: 0000000000002660
>> Apr 9 09:11:36 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
>> Apr 9 09:11:36 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
>> Apr 9 09:11:36 host kernel: Call Trace:
>> Apr 9 09:11:36 host kernel: [<ffffffff8100ddb7>] ? xen_safe_halt+0xc/0x15
>> Apr 9 09:11:36 host kernel: [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40
>> Apr 9 09:11:36 host kernel: [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82
>> Apr 9 09:11:36 host kernel: BUG: soft lockup - CPU#0 stuck for 1776s! [swapper:0]
>> Apr 9 09:11:36 host kernel: Modules linked in:
>> Apr 9 09:11:36 host kernel: CPU 0:
>> Apr 9 09:11:36 host kernel: Modules linked in:
>> Apr 9 09:11:36 host kernel: Pid: 0, comm: swapper Not tainted 2.6.32.9xls-domU #2
>> Apr 9 09:11:36 host kernel: RIP: e030:[<ffffffff810093aa>] [<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
>> Apr 9 09:11:36 host kernel: RSP: e02b:ffffffff81691f70 EFLAGS: 00000246
>> Apr 9 09:11:36 host kernel: RAX: 0000000000000000 RBX: ffffffff81690000 RCX: ffffffff810093aa
>> Apr 9 09:11:36 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000001
>> Apr 9 09:11:36 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000 R09: ffffffff8100e3b2
>> Apr 9 09:11:36 host kernel: R10: 0000000000000001 R11: 0000000000000246 R12: ffffffffffffffff
>> Apr 9 09:11:36 host kernel: R13: ffffffff818ebf20 R14: ffffffff818eec70 R15: 0000000000000000
>> Apr 9 09:11:36 host kernel: FS: 00007f07704fe6e0(0000) GS:ffff8800022ac000(0000) knlGS:0000000000000000
>> Apr 9 09:11:36 host kernel: CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b
>> Apr 9 09:11:36 host kernel: CR2: 00000000004884b2 CR3: 0000000071c86000 CR4: 0000000000002660
>> Apr 9 09:11:36 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
>> Apr 9 09:11:36 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
>> Apr 9 09:11:36 host kernel: Call Trace:
>> Apr 9 09:11:36 host kernel: [<ffffffff8100ddb7>] ? xen_safe_halt+0xc/0x15
>> Apr 9 09:11:36 host kernel: [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40
>> Apr 9 09:11:36 host kernel: [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82
>> Apr 9 09:11:36 host kernel: [<ffffffff818b6c42>] ? start_kernel+0x353/0x35f
>> Apr 9 09:20:24 host kernel: BUG: soft lockup - CPU#1 stuck for 492s! [ssh:2324]
>> Apr 9 09:20:24 host kernel: Modules linked in:
>> Apr 9 09:20:24 host kernel: CPU 1:
>> Apr 9 09:20:24 host kernel: Modules linked in:
>> Apr 9 09:20:24 host kernel: Pid: 2324, comm: ssh Not tainted 2.6.32.9xls-domU #2
>> Apr 9 09:20:24 host kernel: RIP: e033:[<00007f1b0e259f20>] [<00007f1b0e259f20>] 0x7f1b0e259f20
>> Apr 9 09:20:24 host kernel: RSP: e02b:00007fffa45a7338 EFLAGS: 00000206
>> Apr 9 09:20:24 host kernel: RAX: 00007f1b11f32b20 RBX: 00007f1b0e5399c0 RCX: 00000000000000a0
>> Apr 9 09:20:24 host kernel: RDX: 00007f1b11f32b80 RSI: 0000000000000004 RDI: 00007f1b0e539c30
>> Apr 9 09:20:24 host kernel: RBP: 0000000000000064 R08: 0000000000000001 R09: 0000000000000001
>> Apr 9 09:20:24 host kernel: R10: 00007f1b0e539a30 R11: 00007f1b0e25bfb0 R12: 0000000000000000
>> Apr 9 09:20:24 host kernel: R13: 00007f1b0fcd2aec R14: 0000000000000111 R15: 0000000000000071
>> Apr 9 09:20:24 host kernel: FS: 00007f1b10528710(0000) GS:ffff8800022c7000(0000) knlGS:0000000000000000
>> Apr 9 09:20:24 host kernel: CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b
>> Apr 9 09:20:24 host kernel: CR2: 00007fa913f53000 CR3: 000000006ad44000 CR4: 0000000000002660
>> Apr 9 09:20:24 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
>> Apr 9 09:20:24 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
>> Apr 9 09:20:24 host kernel: Call Trace:
>> Apr 9 09:20:24 host kernel: BUG: soft lockup - CPU#0 stuck for 492s! [swapper:0]
>> Apr 9 09:20:24 host kernel: Modules linked in:
>> Apr 9 09:20:24 host kernel: CPU 0:
>> Apr 9 09:20:24 host kernel: Modules linked in:
>> Apr 9 09:20:24 host kernel: Pid: 0, comm: swapper Not tainted 2.6.32.9xls-domU #2
>> Apr 9 09:20:24 host kernel: RIP: e030:[<ffffffff810093aa>] [<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
>> Apr 9 09:20:24 host kernel: RSP: e02b:ffffffff81691f70 EFLAGS: 00000246
>> Apr 9 09:20:24 host kernel: RAX: 0000000000000000 RBX: ffffffff81690000 RCX: ffffffff810093aa
>> Apr 9 09:20:24 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000001
>> Apr 9 09:20:24 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000 R09: ffffffff8100e3b2
>> Apr 9 09:20:24 host kernel: R10: 0000000000000001 R11: 0000000000000246 R12: ffffffffffffffff
>> Apr 9 09:20:24 host kernel: R13: ffffffff818ebf20 R14: ffffffff818eec70 R15: 0000000000000000
>> Apr 9 09:20:24 host kernel: FS: 00007ff39e495710(0000) GS:ffff8800022ac000(0000) knlGS:0000000000000000
>> Apr 9 09:20:24 host kernel: CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b
>> Apr 9 09:20:24 host kernel: CR2: 00007fff892a1e30 CR3: 000000006acb1000 CR4: 0000000000002660
>> Apr 9 09:20:24 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
>> Apr 9 09:20:24 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
>> Apr 9 09:20:24 host kernel: Call Trace:
>> Apr 9 09:20:24 host kernel: [<ffffffff8100ddb7>] ? xen_safe_halt+0xc/0x15
>> Apr 9 09:20:24 host kernel: [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40
>> Apr 9 09:20:24 host kernel: [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82
>> Apr 9 09:20:24 host kernel: [<ffffffff818b6c42>] ? start_kernel+0x353/0x35f
>> Apr 9 09:24:36 host kernel: BUG: soft lockup - CPU#1 stuck for 234s! [gzip:4496]
>> Apr 9 09:24:36 host kernel: Modules linked in:
>> Apr 9 09:24:36 host kernel: CPU 1:
>> Apr 9 09:24:36 host kernel: Modules linked in:
>> Apr 9 09:24:36 host kernel: Pid: 4496, comm: gzip Not tainted 2.6.32.9xls-domU #2
>> Apr 9 09:24:36 host kernel: RIP: e033:[<0000000000409804>] [<0000000000409804>] 0x409804
>> Apr 9 09:24:36 host kernel: RSP: e02b:00007fff87e009f8 EFLAGS: 00000202
>> Apr 9 09:24:36 host kernel: RAX: 00000000f6019e6d RBX: 0000000000008000 RCX: 00007f529598c610
>> Apr 9 09:24:36 host kernel: RDX: 0000000000f6019e RSI: 00000000000045a6 RDI: 00000000006534fb
>> Apr 9 09:24:36 host kernel: RBP: 0000000000005800 R08: 0000000000004000 R09: 7464697720223333
>> Apr 9 09:24:36 host kernel: R10: 6c61202236223d68 R11: 0000000000000246 R12: 00000000006522a0
>> Apr 9 09:24:36 host kernel: R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
>> Apr 9 09:24:36 host kernel: FS: 00007f5295e2a6e0(0000) GS:ffff8800022c7000(0000) knlGS:0000000000000000
>> Apr 9 09:24:36 host kernel: CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b
>> Apr 9 09:24:36 host kernel: CR2: 0000000000d550c8 CR3: 0000000026474000 CR4: 0000000000002660
>> Apr 9 09:24:37 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
>> Apr 9 09:24:37 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
>> Apr 9 09:24:37 host kernel: Call Trace:
>> Apr 9 09:24:37 host kernel: BUG: soft lockup - CPU#0 stuck for 234s! [swapper:0]
>> Apr 9 09:24:37 host kernel: Modules linked in:
>> Apr 9 09:24:37 host kernel: CPU 0:
>> Apr 9 09:24:37 host kernel: Modules linked in:
>> Apr 9 09:24:37 host kernel: Pid: 0, comm: swapper Not tainted 2.6.32.9xls-domU #2
>> Apr 9 09:24:37 host kernel: RIP: e030:[<ffffffff810093aa>] [<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
>> Apr 9 09:24:38 host kernel: RSP: e02b:ffffffff81691f70 EFLAGS: 00000246
>> Apr 9 09:24:38 host kernel: RAX: 0000000000000000 RBX: ffffffff81690000 RCX: ffffffff810093aa
>> Apr 9 09:24:38 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000001
>> Apr 9 09:24:38 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000 R09: ffffffff8100e3b2
>> Apr 9 09:24:38 host kernel: R10: 0000000000000001 R11: 0000000000000246 R12: ffffffffffffffff
>> Apr 9 09:24:38 host kernel: R13: ffffffff818ebf20 R14: ffffffff818eec70 R15: 0000000000000000
>> Apr 9 09:24:38 host kernel: FS: 00007f2f6a31a710(0000) GS:ffff8800022ac000(0000) knlGS:0000000000000000
>> Apr 9 09:24:38 host kernel: CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b
>> Apr 9 09:24:38 host kernel: CR2: 00007f2f6a327000 CR3: 000000003f62e000 CR4: 0000000000002660
>> Apr 9 09:24:38 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
>> Apr 9 09:24:38 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
>> Apr 9 09:24:38 host kernel: Call Trace:
>> Apr 9 09:24:38 host kernel: [<ffffffff8100ddb7>] ? xen_safe_halt+0xc/0x15
>> Apr 9 09:24:38 host kernel: [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40
>> Apr 9 09:24:38 host kernel: [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82
>> Apr 9 09:24:38 host kernel: [<ffffffff818b6c42>] ? start_kernel+0x353/0x35f
>> Apr 9 10:05:51 host kernel: BUG: soft lockup - CPU#1 stuck for 2305s! [swapper:0]
>> Apr 9 10:05:51 host kernel: Modules linked in:
>> Apr 9 10:05:51 host kernel: CPU 1:
>> Apr 9 10:05:51 host kernel: Modules linked in:
>> Apr 9 10:05:51 host kernel: Pid: 0, comm: swapper Not tainted 2.6.32.9xls-domU #2
>> Apr 9 10:05:51 host kernel: RIP: e030:[<ffffffff810093aa>] [<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
>> Apr 9 10:05:51 host kernel: RSP: e02b:ffff88007f8a1f08 EFLAGS: 00000246
>> Apr 9 10:05:51 host kernel: RAX: 0000000000000000 RBX: ffff88007f8a0000 RCX: ffffffff810093aa
>> Apr 9 10:05:51 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000001
>> Apr 9 10:05:51 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000 R09: ffffffff8100e3b2
>> Apr 9 10:05:51 host kernel: R10: 0000000000000001 R11: 0000000000000246 R12: 0000000000000000
>> Apr 9 10:05:51 host kernel: R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
>> Apr 9 10:05:51 host kernel: FS: 00007ff39e495710(0000) GS:ffff8800022c7000(0000) knlGS:0000000000000000
>> Apr 9 10:05:51 host kernel: CS: e033 DS: 002b ES: 002b CR0: 000000008005003b
>> Apr 9 10:05:51 host kernel: CR2: 0000000004786158 CR3: 000000007e2a6000 CR4: 0000000000002660
>> Apr 9 10:05:51 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
>> Apr 9 10:05:51 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
>> Apr 9 10:05:51 host kernel: Call Trace:
>> Apr 9 10:05:51 host kernel: [<ffffffff8100ddb7>] ? xen_safe_halt+0xc/0x15
>> Apr 9 10:05:51 host kernel: [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40
>> Apr 9 10:05:51 host kernel: [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82
>> Apr 9 10:05:51 host kernel: BUG: soft lockup - CPU#0 stuck for 2305s! [httpd:9922]
>> Apr 9 10:05:51 host kernel: Modules linked in:
>> Apr 9 10:05:51 host kernel: CPU 0:
>> Apr 9 10:05:51 host kernel: Modules linked in:
>> Apr 9 10:05:51 host kernel: Pid: 9922, comm: httpd Not tainted 2.6.32.9xls-domU #2
>> Apr 9 10:05:51 host kernel: RIP: e033:[<00007ff39a44a36d>] [<00007ff39a44a36d>] 0x7ff39a44a36d
>> Apr 9 10:05:51 host kernel: RSP: e02b:00007fff4b9ad118 EFLAGS: 00000a86
>> Apr 9 10:05:51 host kernel: RAX: c0c176e403831574 RBX: 0000000003643b60 RCX: c0c176e403831574
>> Apr 9 10:05:51 host kernel: RDX: 0000000000000000 RSI: 0000000000000004 RDI: 000000000420e43b
>> Apr 9 10:05:51 host kernel: RBP: 0000000003643b60 R08: 0000000000000004 R09: 00007fff4b9ad270
>> Apr 9 10:05:51 host kernel: R10: 00007ff39aaa9fa0 R11: 00007fff4b9ad188 R12: 000000000320e6d0
>> Apr 9 10:05:51 host kernel: R13: 0000000000000000 R14: 00007fff4b9ad368 R15: 0000000000000000
>> Apr 9 10:05:51 host kernel: FS: 00007ff39e495710(0000) GS:ffff8800022ac000(0000) knlGS:0000000000000000
>> Apr 9 10:05:51 host kernel: CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b
>> Apr 9 10:05:51 host kernel: CR2: 00000000047a4018 CR3: 000000007972c000 CR4: 0000000000002660
>> Apr 9 10:05:51 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
>> Apr 9 10:05:51 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
>> Apr 9 10:05:51 host kernel: Call Trace:
>> Apr 9 10:50:04 host kernel: BUG: soft lockup - CPU#1 stuck for 2471s! [swapper:0]
>> Apr 9 10:50:04 host kernel: Modules linked in:
>> Apr 9 10:50:04 host kernel: CPU 1:
>> Apr 9 10:50:04 host kernel: Modules linked in:
>> Apr 9 10:50:04 host kernel: Pid: 0, comm: swapper Not tainted 2.6.32.9xls-domU #2
>> Apr 9 10:50:04 host kernel: RIP: e030:[<ffffffff810093aa>] [<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
>> Apr 9 10:50:04 host kernel: RSP: e02b:ffff88007f8a1f08 EFLAGS: 00000246
>> Apr 9 10:50:04 host kernel: RAX: 0000000000000000 RBX: ffff88007f8a0000 RCX: ffffffff810093aa
>> Apr 9 10:50:04 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000001
>> Apr 9 10:50:04 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000 R09: ffffffff8100e3b2
>> Apr 9 10:50:04 host kernel: R10: 0000000000000001 R11: 0000000000000246 R12: 0000000000000000
>> Apr 9 10:50:04 host kernel: R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
>> Apr 9 10:50:04 host kernel: FS: 00007ff39e495710(0000) GS:ffff8800022c7000(0000) knlGS:0000000000000000
>> Apr 9 10:50:04 host kernel: CS: e033 DS: 002b ES: 002b CR0: 000000008005003b
>> Apr 9 10:50:04 host kernel: CR2: 00007ff398a0ce08 CR3: 000000004ddb4000 CR4: 0000000000002660
>> Apr 9 10:50:04 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
>> Apr 9 10:50:04 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
>> Apr 9 10:50:04 host kernel: Call Trace:
>> Apr 9 10:50:04 host kernel: [<ffffffff8100ddb7>] ? xen_safe_halt+0xc/0x15
>> Apr 9 10:50:04 host kernel: [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40
>> Apr 9 10:50:04 host kernel: [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82
>> Apr 9 10:50:04 host kernel: BUG: soft lockup - CPU#0 stuck for 2471s! [httpd:16797]
>> Apr 9 10:50:04 host kernel: Modules linked in:
>> Apr 9 10:50:04 host kernel: CPU 0:
>> Apr 9 10:50:04 host kernel: Modules linked in:
>> Apr 9 10:50:04 host kernel: Pid: 16797, comm: httpd Not tainted 2.6.32.9xls-domU #2
>> Apr 9 10:50:04 host kernel: RIP: e033:[<00007ff39a49ae55>] [<00007ff39a49ae55>] 0x7ff39a49ae55
>> Apr 9 10:50:04 host kernel: RSP: e02b:00007fff4b9b10b8 EFLAGS: 00000246
>> Apr 9 10:50:04 host kernel: RAX: 000000000368dd40 RBX: 00000000036868d0 RCX: 222879726575713e
>> Apr 9 10:50:05 host kernel: RDX: 000000000000000c RSI: 0000000003686a6c RDI: 00007fff4b9b1d00
>> Apr 9 10:50:05 host kernel: RBP: 00007fff4b9b1da0 R08: 0000000000000001 R09: 0101010101010101
>> Apr 9 10:50:05 host kernel: R10: 0000000000000001 R11: 00007ff39c4ac0be R12: 00007fff4b9b1d00
>> Apr 9 10:50:05 host kernel: R13: 00007ff39aaa9fa0 R14: 00007ff39aaa9fa0 R15: 0000000003695df0
>> Apr 9 10:50:05 host kernel: FS: 00007ff39e495710(0000) GS:ffff8800022ac000(0000) knlGS:0000000000000000
>> Apr 9 10:50:05 host kernel: CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b
>> Apr 9 10:50:05 host kernel: CR2: 00000000032068c0 CR3: 000000004ddb4000 CR4: 0000000000002660
>> Apr 9 10:50:05 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
>> Apr 9 10:50:05 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
>> Apr 9 10:50:05 host kernel: Call Trace:
>> Apr 9 10:57:01 host kernel: BUG: soft lockup - CPU#1 stuck for 389s! [exim:17977]
>> Apr 9 10:57:01 host kernel: Modules linked in:
>> Apr 9 10:57:01 host kernel: CPU 1:
>> Apr 9 10:57:01 host kernel: Modules linked in:
>> Apr 9 10:57:01 host kernel: Pid: 17977, comm: exim Not tainted 2.6.32.9xls-domU #2
>> Apr 9 10:57:01 host kernel: RIP: e030:[<ffffffff8100922a>] [<ffffffff8100922a>] hypercall_page+0x22a/0x1001
>> Apr 9 10:57:01 host kernel: RSP: e02b:ffff880056611ea0 EFLAGS: 00000246
>> Apr 9 10:57:01 host kernel: RAX: 0000000000030004 RBX: 0000000000000000 RCX: ffffffff8100922a
>> Apr 9 10:57:01 host kernel: RDX: 00007fd75a6d614e RSI: 0000000000000000 RDI: 0000000000000000
>> Apr 9 10:57:01 host kernel: RBP: 00007fd75a6d614e R08: 00007fd75c048710 R09: 0000000000000003
>> Apr 9 10:57:01 host kernel: R10: 00007fd75a909a30 R11: 0000000000000246 R12: 0000000000000004
>> Apr 9 10:57:01 host kernel: R13: ffff880056611f58 R14: ffff88005aa996d0 R15: ffff88005a51b3c0
>> Apr 9 10:57:01 host kernel: FS: 00007fd75c048710(0000) GS:ffff8800022c7000(0000) knlGS:0000000000000000
>> Apr 9 10:57:01 host kernel: CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b
>> Apr 9 10:57:01 host kernel: CR2: 00007fd75a6d614e CR3: 000000007eb9a000 CR4: 0000000000002660
>> Apr 9 10:57:01 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
>> Apr 9 10:57:01 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
>> Apr 9 10:57:01 host kernel: Call Trace:
>> Apr 9 10:57:01 host kernel: [<ffffffff8100dd39>] ? xen_force_evtchn_callback+0x9/0xa
>> Apr 9 10:57:01 host kernel: [<ffffffff8100e3b2>] ? check_events+0x12/0x20
>> Apr 9 10:57:01 host kernel: [<ffffffff8100e359>] ? xen_irq_enable_direct_end+0x0/0x7
>> Apr 9 10:57:01 host kernel: [<ffffffff814da477>] ? do_page_fault+0xfb/0x28d
>> Apr 9 10:57:01 host kernel: [<ffffffff814d85b5>] ? page_fault+0x25/0x30
>> Apr 9 10:57:01 host kernel: BUG: soft lockup - CPU#0 stuck for 389s! [netstat:17966]
>> Apr 9 10:57:01 host kernel: Modules linked in:
>> Apr 9 10:57:01 host kernel: CPU 0:
>> Apr 9 10:57:01 host kernel: Modules linked in:
>> Apr 9 10:57:01 host kernel: Pid: 17966, comm: netstat Not tainted 2.6.32.9xls-domU #2
>> Apr 9 10:57:01 host kernel: RIP: e030:[<ffffffff814d8226>] [<ffffffff814d8226>] _spin_lock+0xa/0x15
>> Apr 9 10:57:01 host kernel: RSP: e02b:ffff88006ad4dce0 EFLAGS: 00000202
>> Apr 9 10:57:01 host kernel: RAX: 0000000000001c1c RBX: ffff88007ff77180 RCX: 0000000000000012
>> Apr 9 10:57:01 host kernel: RDX: 018721dfffc35c9f RSI: ffff88006ad4ddc8 RDI: ffff88007ff77188
>> Apr 9 10:57:01 host kernel: RBP: ffff88007ff77198 R08: 0000000000000000 R09: 0000000000300000
>> Apr 9 10:57:01 host kernel: R10: 00000000fffffff2 R11: ffffffff8126158e R12: ffff88000ff859c0
>> Apr 9 10:57:01 host kernel: R13: ffff88006ad4ddc8 R14: 0000000000034b38 R15: 0000000000000002
>> Apr 9 10:57:01 host kernel: FS: 00007fb435545710(0000) GS:ffff8800022ac000(0000) knlGS:0000000000000000
>> Apr 9 10:57:01 host kernel: CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b
>> Apr 9 10:57:01 host kernel: CR2: 00007f0e429d6000 CR3: 000000007ee3d000 CR4: 0000000000002660
>> Apr 9 10:57:01 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
>> Apr 9 10:57:01 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
>> Apr 9 10:57:01 host kernel: Call Trace:
>> Apr 9 10:57:01 host kernel: [<ffffffff810cb01b>] ? __d_lookup+0x7a/0x105
>> Apr 9 10:57:01 host kernel: [<ffffffff810c251e>] ? do_lookup+0x2a/0x1b3
>> Apr 9 10:57:01 host kernel: [<ffffffff810c4498>] ? __link_path_walk+0x348/0xdcd
>> Apr 9 10:57:01 host kernel: [<ffffffff8100dd39>] ? xen_force_evtchn_callback+0x9/0xa
>> Apr 9 10:57:01 host kernel: [<ffffffff8100e3b2>] ? check_events+0x12/0x20
>> Apr 9 10:57:01 host kernel: [<ffffffff810c514b>] ? path_walk+0x66/0xca
>> Apr 9 10:57:01 host kernel: [<ffffffff810c51cf>] ? do_path_lookup+0x20/0x77
>> Apr 9 10:57:01 host kernel: [<ffffffff810c5ae9>] ? user_path_at+0x48/0x79
>> Apr 9 10:57:01 host kernel: [<ffffffff8100e39f>] ? xen_restore_fl_direct_end+0x0/0x1
>> Apr 9 10:57:01 host kernel: [<ffffffff810cfd6b>] ? mntput_no_expire+0x23/0xed
>> Apr 9 10:57:01 host kernel: [<ffffffff8129b703>] ? _atomic_dec_and_lock+0x33/0x50
>> Apr 9 10:57:01 host kernel: [<ffffffff810cfd6b>] ? mntput_no_expire+0x23/0xed
>> Apr 9 10:57:01 host kernel: [<ffffffff810bdf4d>] ? sys_readlinkat+0x25/0x8d
>> Apr 9 10:57:01 host kernel: [<ffffffff81010ac2>] ? system_call_fastpath+0x16/0x1b
>> Apr 9 11:00:36 host kernel: BUG: soft lockup - CPU#1 stuck for 200s! [swapper:0]
>> Apr 9 11:00:36 host kernel: Modules linked in:
>> Apr 9 11:00:36 host kernel: CPU 1:
>> Apr 9 11:00:36 host kernel: Modules linked in:
>> Apr 9 11:00:36 host kernel: Pid: 0, comm: swapper Not tainted 2.6.32.9xls-domU #2
>> Apr 9 11:00:36 host kernel: RIP: e030:[<ffffffff810093aa>] [<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
>> Apr 9 11:00:36 host kernel: RSP: e02b:ffff88007f8a1f08 EFLAGS: 00000246
>> Apr 9 11:00:36 host kernel: RAX: 0000000000000000 RBX: ffff88007f8a0000 RCX: ffffffff810093aa
>> Apr 9 11:00:36 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000001
>> Apr 9 11:00:36 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000 R09: ffffffff8100e3b2
>> Apr 9 11:00:36 host kernel: R10: 0000000000000001 R11: 0000000000000246 R12: 0000000000000000
>> Apr 9 11:00:36 host kernel: R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
>> Apr 9 11:00:36 host kernel: FS: 00007f719f5fa6e0(0000) GS:ffff8800022c7000(0000) knlGS:0000000000000000
>> Apr 9 11:00:36 host kernel: CS: e033 DS: 002b ES: 002b CR0: 000000008005003b
>> Apr 9 11:00:36 host kernel: CR2: 00007ff39c43cfa8 CR3: 000000007d0db000 CR4: 0000000000002660
>> Apr 9 11:00:36 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
>> Apr 9 11:00:36 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
>> Apr 9 11:00:37 host kernel: Call Trace:
>> Apr 9 11:00:37 host kernel: [<ffffffff8100ddb7>] ? xen_safe_halt+0xc/0x15
>> Apr 9 11:00:37 host kernel: [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40
>> Apr 9 11:00:37 host kernel: [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82
>> Apr 9 11:00:37 host kernel: BUG: soft lockup - CPU#0 stuck for 200s! [swapper:0]
>> Apr 9 11:00:37 host kernel: Modules linked in:
>> Apr 9 11:00:37 host kernel: CPU 0:
>> Apr 9 11:00:37 host kernel: Modules linked in:
>> Apr 9 11:00:37 host kernel: Pid: 0, comm: swapper Not tainted 2.6.32.9xls-domU #2
>> Apr 9 11:00:37 host kernel: RIP: e030:[<ffffffff810093aa>] [<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
>> Apr 9 11:00:37 host kernel: RSP: e02b:ffffffff81691f70 EFLAGS: 00000246
>> Apr 9 11:00:37 host kernel: RAX: 0000000000000000 RBX: ffffffff81690000 RCX: ffffffff810093aa
>> Apr 9 11:00:37 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000001
>> Apr 9 11:00:37 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000 R09: ffffffff8100e3b2
>> Apr 9 11:00:37 host kernel: R10: 0000000000000001 R11: 0000000000000246 R12: ffffffffffffffff
>> Apr 9 11:00:37 host kernel: R13: ffffffff818ebf20 R14: ffffffff818eec70 R15: 0000000000000000
>> Apr 9 11:00:37 host kernel: FS: 00007f07704fe6e0(0000) GS:ffff8800022ac000(0000) knlGS:0000000000000000
>> Apr 9 11:00:37 host kernel: CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b
>> Apr 9 11:00:37 host kernel: CR2: 00000000035750c8 CR3: 000000007ea99000 CR4: 0000000000002660
>> Apr 9 11:00:37 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
>> Apr 9 11:00:37 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
>> Apr 9 11:00:37 host kernel: Call Trace:
>> Apr 9 11:00:37 host kernel: [<ffffffff8100ddb7>] ? xen_safe_halt+0xc/0x15
>> Apr 9 11:00:37 host kernel: [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40
>> Apr 9 11:00:37 host kernel: [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82
>> Apr 9 11:00:37 host kernel: [<ffffffff818b6c42>] ? start_kernel+0x353/0x35f
>> Apr 9 11:08:50 host kernel: BUG: soft lockup - CPU#0 stuck for 460s! [httpd:19360]
>> Apr 9 11:08:50 host kernel: Modules linked in:
>> Apr 9 11:08:50 host kernel: CPU 0:
>> Apr 9 11:08:50 host kernel: Modules linked in:
>> Apr 9 11:08:50 host kernel: Pid: 19360, comm: httpd Not tainted 2.6.32.9xls-domU #2
>> Apr 9 11:08:50 host kernel: RIP: e030:[<ffffffff813b995e>] [<ffffffff813b995e>] sock_alloc_send_skb+0x3/0xc
>> Apr 9 11:08:50 host kernel: RSP: e02b:ffff88005675dc80 EFLAGS: 00000246
>> Apr 9 11:08:50 host kernel: RAX: 0000000000003e00 RBX: ffff88005675dcb8 RCX: ffff88005675dce4
>> Apr 9 11:08:50 host kernel: RDX: 0000000000000000 RSI: 0000000000000057 RDI: ffff88002678c900
>> Apr 9 11:08:50 host kernel: RBP: ffff88007f740b80 R08: ffff88005675dce4 R09: 75646f72705f6d76
>> Apr 9 11:08:50 host kernel: R10: 5245485720607463 R11: ffffffff81261683 R12: 0000000000000057
>> Apr 9 11:08:50 host kernel: R13: 0000000000000000 R14: ffff88006add3c80 R15: ffff88005675dd60
>> Apr 9 11:08:50 host kernel: FS: 00007ff39e495710(0000) GS:ffff8800022ac000(0000) knlGS:0000000000000000
>> Apr 9 11:08:50 host kernel: CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b
>> Apr 9 11:08:50 host kernel: CR2: 00000000045cf000 CR3: 000000007f97d000 CR4: 0000000000002660
>> Apr 9 11:08:50 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
>> Apr 9 11:08:50 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
>> Apr 9 11:08:50 host kernel: Call Trace:
>> Apr 9 11:08:50 host kernel: [<ffffffff814573c8>] ? unix_stream_sendmsg+0x18a/0x31f
>> Apr 9 11:08:50 host kernel: [<ffffffff813b61ec>] ? sock_aio_write+0xf6/0x10a
>> Apr 9 11:08:50 host kernel: [<ffffffff8100dd39>] ? xen_force_evtchn_callback+0x9/0xa
>> Apr 9 11:08:50 host kernel: [<ffffffff8100e39f>] ? xen_restore_fl_direct_end+0x0/0x1
>> Apr 9 11:08:50 host kernel: [<ffffffff810bab7e>] ? do_sync_write+0xce/0x113
>> Apr 9 11:08:50 host kernel: [<ffffffff810c5af3>] ? user_path_at+0x52/0x79
>> Apr 9 11:08:50 host kernel: [<ffffffff810be0b0>] ? cp_new_stat+0xe9/0xfc
>> Apr 9 11:08:50 host kernel: [<ffffffff810561b8>] ? autoremove_wake_function+0x0/0x2e
>> Apr 9 11:08:50 host kernel: [<ffffffff8100e3b2>] ? check_events+0x12/0x20
>> Apr 9 11:08:50 host kernel: [<ffffffff8100dd39>] ? xen_force_evtchn_callback+0x9/0xa
>> Apr 9 11:08:50 host kernel: [<ffffffff8100e3b2>] ? check_events+0x12/0x20
>> Apr 9 11:08:50 host kernel: [<ffffffff810bb4c5>] ? vfs_write+0xc0/0x149
>> Apr 9 11:08:50 host kernel: [<ffffffff810bb60a>] ? sys_write+0x45/0x6e
>> Apr 9 11:08:50 host kernel: [<ffffffff81010ac2>] ? system_call_fastpath+0x16/0x1b
>> Apr 9 11:08:50 host kernel: BUG: soft lockup - CPU#1 stuck for 460s! [swapper:0]
>> Apr 9 11:08:50 host kernel: Modules linked in:
>> Apr 9 11:08:50 host kernel: CPU 1:
>> Apr 9 11:08:50 host kernel: Modules linked in:
>> Apr 9 11:08:50 host kernel: Pid: 0, comm: swapper Not tainted 2.6.32.9xls-domU #2
>> Apr 9 11:08:50 host kernel: RIP: e030:[<ffffffff810093aa>] [<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
>> Apr 9 11:08:50 host kernel: RSP: e02b:ffff88007f8a1f08 EFLAGS: 00000246
>> Apr 9 11:08:50 host kernel: RAX: 0000000000000000 RBX: ffff88007f8a0000 RCX: ffffffff810093aa
>> Apr 9 11:08:50 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000001
>> Apr 9 11:08:50 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000 R09: ffffffff8100e3b2
>> Apr 9 11:08:50 host kernel: R10: 0000000000000001 R11: 0000000000000246 R12: 0000000000000000
>> Apr 9 11:08:50 host kernel: R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
>> Apr 9 11:08:50 host kernel: FS: 00007f0e429cb6e0(0000) GS:ffff8800022c7000(0000) knlGS:0000000000000000
>> Apr 9 11:08:50 host kernel: CS: e033 DS: 002b ES: 002b CR0: 000000008005003b
>> Apr 9 11:08:50 host kernel: CR2: 0000000004598000 CR3: 000000007e2a6000 CR4: 0000000000002660
>> Apr 9 11:08:50 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
>> Apr 9 11:08:50 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
>> Apr 9 11:08:50 host kernel: Call Trace:
>> Apr 9 11:08:50 host kernel: [<ffffffff8100ddb7>] ? xen_safe_halt+0xc/0x15
>> Apr 9 11:08:50 host kernel: [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40
>> Apr 9 11:08:50 host kernel: [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82
>> Apr 9 12:00:40 host kernel: BUG: soft lockup - CPU#1 stuck for 2897s! [httpd:26903]
>> Apr 9 12:00:40 host kernel: Modules linked in:
>> Apr 9 12:00:40 host kernel: CPU 1:
>> Apr 9 12:00:40 host kernel: Modules linked in:
>> Apr 9 12:00:40 host kernel: Pid: 26903, comm: httpd Not tainted 2.6.32.9xls-domU #2
>> Apr 9 12:00:40 host kernel: RIP: e033:[<00007ff39a427cd2>] [<00007ff39a427cd2>] 0x7ff39a427cd2
>> Apr 9 12:00:40 host kernel: RSP: e02b:00007fff4b9afa20 EFLAGS: 00000202
>> Apr 9 12:00:40 host kernel: RAX: 0000000000000020 RBX: 00007fff4b9aff30 RCX: 0000000000000013
>> Apr 9 12:00:40 host kernel: RDX: 00007ff39a77be20 RSI: 00007fff4b9aff50 RDI: 0000000003bd6418
>> Apr 9 12:00:40 host kernel: RBP: 0000000000000000 R08: 00007ff39a782e60 R09: 00007ff39a783d80
>> Apr 9 12:00:40 host kernel: R10: 0000000000000000 R11: 00007ff39c4ac0a2 R12: 00007ff39aaaa3e0
>> Apr 9 12:00:40 host kernel: R13: 00007fff4b9aff50 R14: 00007fff4b9aff50 R15: 000000000000015e
>> Apr 9 12:00:40 host kernel: FS: 00007ff39e495710(0000) GS:ffff8800022c7000(0000) knlGS:0000000000000000
>> Apr 9 12:00:40 host kernel: CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b
>> Apr 9 12:00:40 host kernel: CR2: 0000000003bdc000 CR3: 0000000056566000 CR4: 0000000000002660
>> Apr 9 12:00:40 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
>> Apr 9 12:00:40 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
>> Apr 9 12:00:40 host kernel: Call Trace:
>> Apr 9 12:00:40 host kernel: BUG: soft lockup - CPU#0 stuck for 2897s! [swapper:0]
>> Apr 9 12:00:40 host kernel: Modules linked in:
>> Apr 9 12:00:40 host kernel: CPU 0:
>> Apr 9 12:00:40 host kernel: Modules linked in:
>> Apr 9 12:00:40 host kernel: Pid: 0, comm: swapper Not tainted 2.6.32.9xls-domU #2
>> Apr 9 12:00:40 host kernel: RIP: e030:[<ffffffff810093aa>] [<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
>> Apr 9 12:00:40 host kernel: RSP: e02b:ffffffff81691f70 EFLAGS: 00000246
>> Apr 9 12:00:40 host kernel: RAX: 0000000000000000 RBX: ffffffff81690000 RCX: ffffffff810093aa
>> Apr 9 12:00:40 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000001
>> Apr 9 12:00:40 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000 R09: ffffffff8100e3b2
>> Apr 9 12:00:40 host kernel: R10: 0000000000000001 R11: 0000000000000246 R12: ffffffffffffffff
>> Apr 9 12:00:40 host kernel: R13: ffffffff818ebf20 R14: ffffffff818eec70 R15: 0000000000000000
>> Apr 9 12:00:40 host kernel: FS: 00007f74fde746e0(0000) GS:ffff8800022ac000(0000) knlGS:0000000000000000
>> Apr 9 12:00:40 host kernel: CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b
>> Apr 9 12:00:40 host kernel: CR2: 00007ff39aec11a7 CR3: 000000007e2a6000 CR4: 0000000000002660
>> Apr 9 12:00:40 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
>> Apr 9 12:00:40 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
>> Apr 9 12:00:40 host kernel: Call Trace:
>> Apr 9 12:00:40 host kernel: [<ffffffff8100ddb7>] ? xen_safe_halt+0xc/0x15
>> Apr 9 12:00:40 host kernel: [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40
>> Apr 9 12:00:40 host kernel: [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82
>> Apr 9 12:00:40 host kernel: [<ffffffff818b6c42>] ? start_kernel+0x353/0x35f
>> Apr 9 12:19:45 host kernel: BUG: soft lockup - CPU#1 stuck for 1065s! [swapper:0]
>> Apr 9 12:19:45 host kernel: Modules linked in:
>> Apr 9 12:19:45 host kernel: CPU 1:
>> Apr 9 12:19:45 host kernel: Modules linked in:
>> Apr 9 12:19:45 host kernel: Pid: 0, comm: swapper Not tainted 2.6.32.9xls-domU #2
>> Apr 9 12:19:45 host kernel: RIP: e030:[<ffffffff810093aa>] [<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
>> Apr 9 12:19:45 host kernel: RSP: e02b:ffff88007f8a1f08 EFLAGS: 00000246
>> Apr 9 12:19:45 host kernel: RAX: 0000000000000000 RBX: ffff88007f8a0000 RCX: ffffffff810093aa
>> Apr 9 12:19:45 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000001
>> Apr 9 12:19:45 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000 R09: ffffffff8100e3b2
>> Apr 9 12:19:45 host kernel: R10: 0000000000000001 R11: 0000000000000246 R12: 0000000000000000
>> Apr 9 12:19:45 host kernel: R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
>> Apr 9 12:19:45 host kernel: FS: 00007ff39e495710(0000) GS:ffff8800022c7000(0000) knlGS:0000000000000000
>> Apr 9 12:19:45 host kernel: CS: e033 DS: 002b ES: 002b CR0: 000000008005003b
>> Apr 9 12:19:45 host kernel: CR2: 00007ff39c5002b0 CR3: 0000000079588000 CR4: 0000000000002660
>> Apr 9 12:19:45 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
>> Apr 9 12:19:45 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
>> Apr 9 12:19:45 host kernel: Call Trace:
>> Apr 9 12:19:45 host kernel: [<ffffffff8100ddb7>] ? xen_safe_halt+0xc/0x15
>> Apr 9 12:19:45 host kernel: [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40
>> Apr 9 12:19:45 host kernel: [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82
>> Apr 9 12:19:45 host kernel: BUG: soft lockup - CPU#0 stuck for 1065s! [swapper:0]
>> Apr 9 12:19:45 host kernel: Modules linked in:
>> Apr 9 12:19:45 host kernel: CPU 0:
>> Apr 9 12:19:45 host kernel: Modules linked in:
>> Apr 9 12:19:45 host kernel: Pid: 0, comm: swapper Not tainted 2.6.32.9xls-domU #2
>> Apr 9 12:19:45 host kernel: RIP: e030:[<ffffffff810093aa>] [<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
>> Apr 9 12:19:45 host kernel: RSP: e02b:ffffffff81691f70 EFLAGS: 00000246
>> Apr 9 12:19:45 host kernel: RAX: 0000000000000000 RBX: ffffffff81690000 RCX: ffffffff810093aa
>> Apr 9 12:19:45 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000001
>> Apr 9 12:19:45 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000 R09: ffffffff8100e3b2
>> Apr 9 12:19:45 host kernel: R10: 0000000000000001 R11: 0000000000000246 R12: ffffffffffffffff
>> Apr 9 12:19:45 host kernel: R13: ffffffff818ebf20 R14: ffffffff818eec70 R15: 0000000000000000
>> Apr 9 12:19:45 host kernel: FS: 00007ff39e495710(0000) GS:ffff8800022ac000(0000) knlGS:0000000000000000
>> Apr 9 12:19:45 host kernel: CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b
>> Apr 9 12:19:45 host kernel: CR2: 00007ff39e404000 CR3: 000000007e2a6000 CR4: 0000000000002660
>> Apr 9 12:19:45 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
>> Apr 9 12:19:45 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
>> Apr 9 12:19:45 host kernel: Call Trace:
>> Apr 9 12:19:45 host kernel: [<ffffffff8100ddb7>] ? xen_safe_halt+0xc/0x15
>> Apr 9 12:19:45 host kernel: [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40
>> Apr 9 12:19:45 host kernel: [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82
>> Apr 9 12:19:45 host kernel: [<ffffffff818b6c42>] ? start_kernel+0x353/0x35f
>> Apr 9 12:37:26 host kernel: BUG: soft lockup - CPU#1 stuck for 989s! [swapper:0]
>> Apr 9 12:37:26 host kernel: Modules linked in:
>> Apr 9 12:37:26 host kernel: CPU 1:
>> Apr 9 12:37:26 host kernel: Modules linked in:
>> Apr 9 12:37:26 host kernel: Pid: 0, comm: swapper Not tainted 2.6.32.9xls-domU #2
>> Apr 9 12:37:26 host kernel: RIP: e030:[<ffffffff810093aa>] [<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
>> Apr 9 12:37:26 host kernel: RSP: e02b:ffff88007f8a1f08 EFLAGS: 00000246
>> Apr 9 12:37:26 host kernel: RAX: 0000000000000000 RBX: ffff88007f8a0000 RCX: ffffffff810093aa
>> Apr 9 12:37:26 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000001
>> Apr 9 12:37:26 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000 R09: ffffffff8100e3b2
>> Apr 9 12:37:26 host kernel: R10: 0000000000000001 R11: 0000000000000246 R12: 0000000000000000
>> Apr 9 12:37:26 host kernel: R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
>> Apr 9 12:37:26 host kernel: FS: 00007ff39e495710(0000) GS:ffff8800022c7000(0000) knlGS:0000000000000000
>> Apr 9 12:37:26 host kernel: CS: e033 DS: 002b ES: 002b CR0: 000000008005003b
>> Apr 9 12:37:26 host kernel: CR2: 00000000039f7330 CR3: 000000007e2a6000 CR4: 0000000000002660
>> Apr 9 12:37:26 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
>> Apr 9 12:37:26 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
>> Apr 9 12:37:26 host kernel: Call Trace:
>> Apr 9 12:37:26 host kernel: [<ffffffff8100ddb7>] ? xen_safe_halt+0xc/0x15
>> Apr 9 12:37:26 host kernel: [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40
>> Apr 9 12:37:26 host kernel: [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82
>> Apr 9 12:37:26 host kernel: BUG: soft lockup - CPU#0 stuck for 989s! [httpd:31684]
>> Apr 9 12:37:26 host kernel: Modules linked in:
>> Apr 9 12:37:26 host kernel: CPU 0:
>> Apr 9 12:37:26 host kernel: Modules linked in:
>> Apr 9 12:37:26 host kernel: Pid: 31684, comm: httpd Not tainted 2.6.32.9xls-domU #2
>> Apr 9 12:37:26 host kernel: RIP: e033:[<00007ff39a41f7d3>] [<00007ff39a41f7d3>] 0x7ff39a41f7d3
>> Apr 9 12:37:26 host kernel: RSP: e02b:00007fff4b9ad330 EFLAGS: 00000246
>> Apr 9 12:37:26 host kernel: RAX: 00007ff39a41f7ac RBX: 00007ff39aaaa3e0 RCX: 0000000000000001
>> Apr 9 12:37:26 host kernel: RDX: 00007ff39a781bdc RSI: 000000000000003e RDI: 0000000000000000
>> Apr 9 12:37:26 host kernel: RBP: 00007ff39aaa9e60 R08: 00007ff39a782e60 R09: 0000000000000003
>> Apr 9 12:37:26 host kernel: R10: 0000000000000000 R11: 00007ff39c4ac0a2 R12: 0000000003a2b297
>> Apr 9 12:37:26 host kernel: R13: 00007ff39aaaa3e0 R14: 00007ff39a784da0 R15: 00007ff39a783e80
>> Apr 9 12:37:26 host kernel: FS: 00007ff39e495710(0000) GS:ffff8800022ac000(0000) knlGS:0000000000000000
>> Apr 9 12:37:26 host kernel: CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b
>> Apr 9 12:37:26 host kernel: CR2: 0000000003aa1000 CR3: 000000004debb000 CR4: 0000000000002660
>> Apr 9 12:37:26 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
>> Apr 9 12:37:26 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
>> Apr 9 12:37:26 host kernel: Call Trace:
>> Apr 9 13:44:52 host kernel: BUG: soft lockup - CPU#1 stuck for 3768s! [swapper:0]
>> Apr 9 13:44:52 host kernel: Modules linked in:
>> Apr 9 13:44:52 host kernel: CPU 1:
>> Apr 9 13:44:52 host kernel: Modules linked in:
>> Apr 9 13:44:52 host kernel: Pid: 0, comm: swapper Not tainted 2.6.32.9xls-domU #2
>> Apr 9 13:44:52 host kernel: RIP: e030:[<ffffffff810093aa>] [<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
>> Apr 9 13:44:52 host kernel: RSP: e02b:ffff88007f8a1f08 EFLAGS: 00000246
>> Apr 9 13:44:52 host kernel: RAX: 0000000000000000 RBX: ffff88007f8a0000 RCX: ffffffff810093aa
>> Apr 9 13:44:52 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000001
>> Apr 9 13:44:52 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000 R09: ffffffff8100e3b2
>> Apr 9 13:44:52 host kernel: R10: 0000000000000001 R11: 0000000000000246 R12: 0000000000000000
>> Apr 9 13:44:52 host kernel: R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
>> Apr 9 13:44:52 host kernel: FS: 00007fd75c048710(0000) GS:ffff8800022c7000(0000) knlGS:0000000000000000
>> Apr 9 13:44:52 host kernel: CS: e033 DS: 002b ES: 002b CR0: 000000008005003b
>> Apr 9 13:44:52 host kernel: CR2: 00007fff892a1e30 CR3: 0000000071c86000 CR4: 0000000000002660
>> Apr 9 13:44:52 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
>> Apr 9 13:44:52 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
>> Apr 9 13:44:52 host kernel: Call Trace:
>> Apr 9 13:44:52 host kernel: [<ffffffff8100ddb7>] ? xen_safe_halt+0xc/0x15
>> Apr 9 13:44:52 host kernel: [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40
>> Apr 9 13:44:52 host kernel: [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82
>> Apr 9 13:44:52 host kernel: BUG: soft lockup - CPU#0 stuck for 3768s! [swapper:0]
>> Apr 9 13:44:52 host kernel: Modules linked in:
>> Apr 9 13:44:52 host kernel: CPU 0:
>> Apr 9 13:44:52 host kernel: Modules linked in:
>> Apr 9 13:44:52 host kernel: Pid: 0, comm: swapper Not tainted 2.6.32.9xls-domU #2
>> Apr 9 13:44:52 host kernel: RIP: e030:[<ffffffff810093aa>] [<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
>> Apr 9 13:44:52 host kernel: RSP: e02b:ffffffff81691f70 EFLAGS: 00000246
>> Apr 9 13:44:52 host kernel: RAX: 0000000000000000 RBX: ffffffff81690000 RCX: ffffffff810093aa
>> Apr 9 13:44:52 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000001
>> Apr 9 13:44:52 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000 R09: ffffffff8100e3b2
>> Apr 9 13:44:52 host kernel: R10: 0000000000000001 R11: 0000000000000246 R12: ffffffffffffffff
>> Apr 9 13:44:52 host kernel: R13: ffffffff818ebf20 R14: ffffffff818eec70 R15: 0000000000000000
>> Apr 9 13:44:52 host kernel: FS: 00007f0e429cb6e0(0000) GS:ffff8800022ac000(0000) knlGS:0000000000000000
>> Apr 9 13:44:52 host kernel: CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b
>> Apr 9 13:44:52 host kernel: CR2: 00007f0e429d6000 CR3: 000000007d0de000 CR4: 0000000000002660
>> Apr 9 13:44:52 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
>> Apr 9 13:44:52 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
>> Apr 9 13:44:52 host kernel: Call Trace:
>> Apr 9 13:44:52 host kernel: [<ffffffff8100ddb7>] ? xen_safe_halt+0xc/0x15
>> Apr 9 13:44:52 host kernel: [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40
>> Apr 9 13:44:52 host kernel: [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82
>> Apr 9 13:44:52 host kernel: [<ffffffff818b6c42>] ? start_kernel+0x353/0x35f
>> Apr 9 13:55:43 host kernel: BUG: soft lockup - CPU#1 stuck for 607s! [swapper:0]
>> Apr 9 13:55:43 host kernel: Modules linked in:
>> Apr 9 13:55:43 host kernel: CPU 1:
>> Apr 9 13:55:43 host kernel: Modules linked in:
>> Apr 9 13:55:43 host kernel: Pid: 0, comm: swapper Not tainted 2.6.32.9xls-domU #2
>> Apr 9 13:55:43 host kernel: RIP: e030:[<ffffffff810093aa>] [<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
>> Apr 9 13:55:43 host kernel: RSP: e02b:ffff88007f8a1f08 EFLAGS: 00000246
>> Apr 9 13:55:43 host kernel: RAX: 0000000000000000 RBX: ffff88007f8a0000 RCX: ffffffff810093aa
>> Apr 9 13:55:43 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000001
>> Apr 9 13:55:43 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000 R09: ffffffff8100e3b2
>> Apr 9 13:55:43 host kernel: R10: 0000000000000001 R11: 0000000000000246 R12: 0000000000000000
>> Apr 9 13:55:43 host kernel: R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
>> Apr 9 13:55:43 host kernel: FS: 00007ff39e495710(0000) GS:ffff8800022c7000(0000) knlGS:0000000000000000
>> Apr 9 13:55:43 host kernel: CS: e033 DS: 002b ES: 002b CR0: 000000008005003b
>> Apr 9 13:55:43 host kernel: CR2: 0000000003c7b27f CR3: 00000000266e8000 CR4: 0000000000002660
>> Apr 9 13:55:43 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
>> Apr 9 13:55:43 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
>> Apr 9 13:55:43 host kernel: Call Trace:
>> Apr 9 13:55:43 host kernel: [<ffffffff8100ddb7>] ? xen_safe_halt+0xc/0x15
>> Apr 9 13:55:43 host kernel: [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40
>> Apr 9 13:55:43 host kernel: [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82
>> Apr 9 13:55:43 host kernel: BUG: soft lockup - CPU#0 stuck for 607s! [swapper:0]
>> Apr 9 13:55:43 host kernel: Modules linked in:
>> Apr 9 13:55:43 host kernel: CPU 0:
>> Apr 9 13:55:43 host kernel: Modules linked in:
>> Apr 9 13:55:43 host kernel: Pid: 0, comm: swapper Not tainted 2.6.32.9xls-domU #2
>> Apr 9 13:55:43 host kernel: RIP: e030:[<ffffffff810093aa>] [<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
>> Apr 9 13:55:43 host kernel: RSP: e02b:ffffffff81691f70 EFLAGS: 00000246
>> Apr 9 13:55:43 host kernel: RAX: 0000000000000000 RBX: ffffffff81690000 RCX: ffffffff810093aa
>> Apr 9 13:55:43 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000001
>> Apr 9 13:55:43 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000 R09: ffffffff8100e3b2
>> Apr 9 13:55:43 host kernel: R10: 0000000000000001 R11: 0000000000000246 R12: ffffffffffffffff
>> Apr 9 13:55:43 host kernel: R13: ffffffff818ebf20 R14: ffffffff818eec70 R15: 0000000000000000
>> Apr 9 13:55:43 host kernel: FS: 00007ff39e495710(0000) GS:ffff8800022ac000(0000) knlGS:0000000000000000
>> Apr 9 13:55:43 host kernel: CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b
>> Apr 9 13:55:43 host kernel: CR2: 00007f2f28bf62e8 CR3: 000000007e2a6000 CR4: 0000000000002660
>> Apr 9 13:55:43 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
>> Apr 9 13:55:43 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
>> Apr 9 13:55:43 host kernel: Call Trace:
>> Apr 9 13:55:43 host kernel: [<ffffffff8100ddb7>] ? xen_safe_halt+0xc/0x15
>> Apr 9 13:55:43 host kernel: [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40
>> Apr 9 13:55:43 host kernel: [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82
>> Apr 9 13:55:43 host kernel: [<ffffffff818b6c42>] ? start_kernel+0x353/0x35f
>> Apr 9 14:16:21 host kernel: BUG: soft lockup - CPU#1 stuck for 1153s! [swapper:0]
>> Apr 9 14:16:21 host kernel: Modules linked in:
>> Apr 9 14:16:21 host kernel: CPU 1:
>> Apr 9 14:16:21 host kernel: Modules linked in:
>> Apr 9 14:16:21 host kernel: Pid: 0, comm: swapper Not tainted 2.6.32.9xls-domU #2
>> Apr 9 14:16:21 host kernel: RIP: e030:[<ffffffff8100922a>] [<ffffffff8100922a>] hypercall_page+0x22a/0x1001
>> Apr 9 14:16:21 host kernel: RSP: e02b:ffff88007f8a1dd8 EFLAGS: 00000246
>> Apr 9 14:16:21 host kernel: RAX: 0000000000030004 RBX: ffff88007db55810 RCX: ffffffff8100922a
>> Apr 9 14:16:21 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
>> Apr 9 14:16:21 host kernel: RBP: 0000000000000001 R08: ffff88007f8a0000 R09: ffffffff8100e3b2
>> Apr 9 14:16:21 host kernel: R10: 0000000000000001 R11: 0000000000000246 R12: 0000000000000000
>> Apr 9 14:16:21 host kernel: R13: ffff88007e0040c0 R14: 0000000000000000 R15: 0000000000001000
>> Apr 9 14:16:21 host kernel: FS: 00007ff39e495710(0000) GS:ffff8800022c7000(0000) knlGS:0000000000000000
>> Apr 9 14:16:21 host kernel: CS: e033 DS: 002b ES: 002b CR0: 000000008005003b
>> Apr 9 14:16:21 host kernel: CR2: 00007ff39e4a7000 CR3: 000000007e2a6000 CR4: 0000000000002660
>> Apr 9 14:16:21 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
>> Apr 9 14:16:21 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
>> Apr 9 14:16:21 host kernel: Call Trace:
>> Apr 9 14:16:21 host kernel: [<ffffffff8100dd39>] ? xen_force_evtchn_callback+0x9/0xa
>> Apr 9 14:16:21 host kernel: [<ffffffff8100e3b2>] ? check_events+0x12/0x20
>> Apr 9 14:16:21 host kernel: [<ffffffff8100e3b2>] ? check_events+0x12/0x20
>> Apr 9 14:16:21 host kernel: [<ffffffff8100e359>] ? xen_irq_enable_direct_end+0x0/0x7
>> Apr 9 14:16:21 host kernel: [<ffffffff8103b070>] ? finish_task_switch+0x27/0x6e
>> Apr 9 14:16:21 host kernel: [<ffffffff814d6a77>] ? thread_return+0x3e/0x109
>> Apr 9 14:16:21 host kernel: [<ffffffff814d87ba>] ? error_exit+0x2a/0x60
>> Apr 9 14:16:21 host kernel: [<ffffffff8101149d>] ? retint_restore_args+0x5/0x6
>> Apr 9 14:16:21 host kernel: [<ffffffff8100fe5f>] ? cpu_idle+0x80/0x82
>> Apr 9 14:16:21 host kernel: BUG: soft lockup - CPU#0 stuck for 1153s! [swapper:0]
>> Apr 9 14:16:21 host kernel: Modules linked in:
>> Apr 9 14:16:21 host kernel: CPU 0:
>> Apr 9 14:16:21 host kernel: Modules linked in:
>> Apr 9 14:16:21 host kernel: Pid: 0, comm: swapper Not tainted 2.6.32.9xls-domU #2
>> Apr 9 14:16:21 host kernel: RIP: e030:[<ffffffff810093aa>] [<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
>> Apr 9 14:16:21 host kernel: RSP: e02b:ffffffff81691f70 EFLAGS: 00000246
>> Apr 9 14:16:21 host kernel: RAX: 0000000000000000 RBX: ffffffff81690000 RCX: ffffffff810093aa
>> Apr 9 14:16:21 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000001
>> Apr 9 14:16:21 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000 R09: ffffffff8100e3b2
>> Apr 9 14:16:21 host kernel: R10: 0000000000000001 R11: 0000000000000246 R12: ffffffffffffffff
>> Apr 9 14:16:21 host kernel: R13: ffffffff818ebf20 R14: ffffffff818eec70 R15: 0000000000000000
>> Apr 9 14:16:21 host kernel: FS: 00007ff39e495710(0000) GS:ffff8800022ac000(0000) knlGS:0000000000000000
>> Apr 9 14:16:21 host kernel: CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b
>> Apr 9 14:16:21 host kernel: CR2: 00007ff39e4a7000 CR3: 000000007e2a6000 CR4: 0000000000002660
>> Apr 9 14:16:21 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
>> Apr 9 14:16:21 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
>> Apr 9 14:16:21 host kernel: Call Trace:
>> Apr 9 14:16:21 host kernel: [<ffffffff8100ddb7>] ? xen_safe_halt+0xc/0x15
>> Apr 9 14:16:21 host kernel: [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40
>> Apr 9 14:16:21 host kernel: [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82
>> Apr 9 14:16:21 host kernel: [<ffffffff818b6c42>] ? start_kernel+0x353/0x35f
>> Apr 9 15:40:21 host kernel: BUG: soft lockup - CPU#1 stuck for 4104s! [swapper:0]
>> Apr 9 15:40:21 host kernel: Modules linked in:
>> Apr 9 15:40:21 host kernel: CPU 1:
>> Apr 9 15:40:21 host kernel: Modules linked in:
>> Apr 9 15:40:21 host kernel: Pid: 0, comm: swapper Not tainted 2.6.32.9xls-domU #2
>> Apr 9 15:40:21 host kernel: RIP: e030:[<ffffffff810093aa>] [<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
>> Apr 9 15:40:21 host kernel: RSP: e02b:ffff88007f8a1f08 EFLAGS: 00000246
>> Apr 9 15:40:21 host kernel: RAX: 0000000000000000 RBX: ffff88007f8a0000 RCX: ffffffff810093aa
>> Apr 9 15:40:21 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000001
>> Apr 9 15:40:21 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000 R09: ffffffff8100e3b2
>> Apr 9 15:40:21 host kernel: R10: 0000000000000001 R11: 0000000000000246 R12: 0000000000000000
>> Apr 9 15:40:21 host kernel: R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
>> Apr 9 15:40:21 host kernel: FS: 00007fd75c048710(0000) GS:ffff8800022c7000(0000) knlGS:0000000000000000
>> Apr 9 15:40:21 host kernel: CS: e033 DS: 002b ES: 002b CR0: 000000008005003b
>> Apr 9 15:40:21 host kernel: CR2: 000000000040d0e2 CR3: 0000000071c86000 CR4: 0000000000002660
>> Apr 9 15:40:21 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
>> Apr 9 15:40:21 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
>> Apr 9 15:40:21 host kernel: Call Trace:
>> Apr 9 15:40:21 host kernel: [<ffffffff8100ddb7>] ? xen_safe_halt+0xc/0x15
>> Apr 9 15:40:21 host kernel: [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40
>> Apr 9 15:40:21 host kernel: [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82
>> Apr 9 15:40:21 host kernel: BUG: soft lockup - CPU#0 stuck for 4104s! [swapper:0]
>> Apr 9 15:40:21 host kernel: Modules linked in:
>> Apr 9 15:40:21 host kernel: CPU 0:
>> Apr 9 15:40:21 host kernel: Modules linked in:
>> Apr 9 15:40:21 host kernel: Pid: 0, comm: swapper Not tainted 2.6.32.9xls-domU #2
>> Apr 9 15:40:21 host kernel: RIP: e030:[<ffffffff810093aa>] [<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
>> Apr 9 15:40:21 host kernel: RSP: e02b:ffffffff81691f70 EFLAGS: 00000246
>> Apr 9 15:40:21 host kernel: RAX: 0000000000000000 RBX: ffffffff81690000 RCX: ffffffff810093aa
>> Apr 9 15:40:21 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000001
>> Apr 9 15:40:21 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000 R09: ffffffff8100e3b2
>> Apr 9 15:40:21 host kernel: R10: 0000000000000001 R11: 0000000000000246 R12: ffffffffffffffff
>> Apr 9 15:40:21 host kernel: R13: ffffffff818ebf20 R14: ffffffff818eec70 R15: 0000000000000000
>> Apr 9 15:40:21 host kernel: FS: 00007f07704fe6e0(0000) GS:ffff8800022ac000(0000) knlGS:0000000000000000
>> Apr 9 15:40:21 host kernel: CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b
>> Apr 9 15:40:21 host kernel: CR2: 00000000004c04a8 CR3: 0000000071c86000 CR4: 0000000000002660
>> Apr 9 15:40:21 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
>> Apr 9 15:40:21 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
>> Apr 9 15:40:21 host kernel: Call Trace:
>> Apr 9 15:40:21 host kernel: [<ffffffff8100ddb7>] ? xen_safe_halt+0xc/0x15
>> Apr 9 15:40:21 host kernel: [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40
>> Apr 9 15:40:21 host kernel: [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82
>> Apr 9 15:40:21 host kernel: [<ffffffff818b6c42>] ? start_kernel+0x353/0x35f
>> Apr 9 16:06:21 host kernel: BUG: soft lockup - CPU#1 stuck for 1452s! [swapper:0]
>> Apr 9 16:06:21 host kernel: Modules linked in:
>> Apr 9 16:06:21 host kernel: CPU 1:
>> Apr 9 16:06:21 host kernel: Modules linked in:
>> Apr 9 16:06:21 host kernel: Pid: 0, comm: swapper Not tainted 2.6.32.9xls-domU #2
>> Apr 9 16:06:21 host kernel: RIP: e030:[<ffffffff810093aa>] [<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
>> Apr 9 16:06:21 host kernel: RSP: e02b:ffff88007f8a1f08 EFLAGS: 00000246
>> Apr 9 16:06:21 host kernel: RAX: 0000000000000000 RBX: ffff88007f8a0000 RCX: ffffffff810093aa
>> Apr 9 16:06:21 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000001
>> Apr 9 16:06:21 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000 R09: ffffffff8100e3b2
>> Apr 9 16:06:21 host kernel: R10: 0000000000000001 R11: 0000000000000246 R12: 0000000000000000
>> Apr 9 16:06:21 host kernel: R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
>> Apr 9 16:06:21 host kernel: FS: 00007ff39e495710(0000) GS:ffff8800022c7000(0000) knlGS:0000000000000000
>> Apr 9 16:06:25 host kernel: CS: e033 DS: 002b ES: 002b CR0: 000000008005003b
>> Apr 9 16:06:25 host kernel: CR2: 00007fbd3c012909 CR3: 0000000071c86000 CR4: 0000000000002660
>> Apr 9 16:06:26 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
>> Apr 9 16:06:26 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
>> Apr 9 16:06:26 host kernel: Call Trace:
>> Apr 9 16:06:26 host kernel: [<ffffffff8100ddb7>] ? xen_safe_halt+0xc/0x15
>> Apr 9 16:06:26 host kernel: [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40
>> Apr 9 16:06:26 host kernel: [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82
>> Apr 9 16:06:26 host kernel: BUG: soft lockup - CPU#0 stuck for 1452s! [swapper:0]
>> Apr 9 16:06:26 host kernel: Modules linked in:
>> Apr 9 16:06:26 host kernel: CPU 0:
>> Apr 9 16:06:26 host kernel: Modules linked in:
>> Apr 9 16:06:26 host kernel: Pid: 0, comm: swapper Not tainted 2.6.32.9xls-domU #2
>> Apr 9 16:06:26 host kernel: RIP: e030:[<ffffffff810093aa>] [<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
>> Apr 9 16:06:26 host kernel: RSP: e02b:ffffffff81691f70 EFLAGS: 00000246
>> Apr 9 16:06:26 host kernel: RAX: 0000000000000000 RBX: ffffffff81690000 RCX: ffffffff810093aa
>> Apr 9 16:06:26 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000001
>> Apr 9 16:06:26 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000 R09: ffffffff8100e3b2
>> Apr 9 16:06:26 host kernel: R10: 0000000000000001 R11: 0000000000000246 R12: ffffffffffffffff
>> Apr 9 16:06:26 host kernel: R13: ffffffff818ebf20 R14: ffffffff818eec70 R15: 0000000000000000
>> Apr 9 16:06:26 host kernel: FS: 00007f07704fe6e0(0000) GS:ffff8800022ac000(0000) knlGS:0000000000000000
>> Apr 9 16:06:26 host kernel: CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b
>> Apr 9 16:06:26 host kernel: CR2: 00007f09a4413000 CR3: 000000007ea99000 CR4: 0000000000002660
>> Apr 9 16:06:26 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
>> Apr 9 16:06:26 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
>> Apr 9 16:06:26 host kernel: Call Trace:
>> Apr 9 16:06:26 host kernel: [<ffffffff8100ddb7>] ? xen_safe_halt+0xc/0x15
>> Apr 9 16:06:26 host kernel: [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40
>> Apr 9 16:06:26 host kernel: [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82
>> Apr 9 16:06:26 host kernel: [<ffffffff818b6c42>] ? start_kernel+0x353/0x35f
>> Apr 9 16:17:43 host kernel: BUG: soft lockup - CPU#1 stuck for 637s! [httpd:32342]
>> Apr 9 16:17:43 host kernel: Modules linked in:
>> Apr 9 16:17:43 host kernel: CPU 1:
>> Apr 9 16:17:43 host kernel: Modules linked in:
>> Apr 9 16:17:43 host kernel: Pid: 32342, comm: httpd Not tainted 2.6.32.9xls-domU #2
>> Apr 9 16:17:43 host kernel: RIP: e030:[<ffffffff8100922a>] [<ffffffff8100922a>] hypercall_page+0x22a/0x1001
>> Apr 9 16:17:43 host kernel: RSP: e02b:ffff88007955fea0 EFLAGS: 00000246
>> Apr 9 16:17:43 host kernel: RAX: 0000000000030004 RBX: 0000000000000000 RCX: ffffffff8100922a
>> Apr 9 16:17:43 host kernel: RDX: 000000000002ba30 RSI: 0000000000000000 RDI: 0000000000000000
>> Apr 9 16:17:43 host kernel: RBP: 000000000394ca98 R08: 0000000000033240 R09: 000000310f6de059
>> Apr 9 16:17:43 host kernel: R10: 0000000000000001 R11: 0000000000000246 R12: 0000000000000006
>> Apr 9 16:17:43 host kernel: R13: ffff88007955ff58 R14: ffff88007f9a8ab0 R15: ffff88007d093b40
>> Apr 9 16:17:43 host kernel: FS: 00007ff39e495710(0000) GS:ffff8800022c7000(0000) knlGS:0000000000000000
>> Apr 9 16:17:43 host kernel: CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b
>> Apr 9 16:17:43 host kernel: CR2: 000000000394ca98 CR3: 000000006adde000 CR4: 0000000000002660
>> Apr 9 16:17:43 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
>> Apr 9 16:17:43 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
>> Apr 9 16:17:43 host kernel: Call Trace:
>> Apr 9 16:17:43 host kernel: [<ffffffff8100dd39>] ? xen_force_evtchn_callback+0x9/0xa
>> Apr 9 16:17:43 host kernel: [<ffffffff8100e3b2>] ? check_events+0x12/0x20
>> Apr 9 16:17:43 host kernel: [<ffffffff8100e359>] ? xen_irq_enable_direct_end+0x0/0x7
>> Apr 9 16:17:43 host kernel: [<ffffffff814da477>] ? do_page_fault+0xfb/0x28d
>> Apr 9 16:17:43 host kernel: [<ffffffff814d85b5>] ? page_fault+0x25/0x30
>> Apr 9 16:17:43 host kernel: BUG: soft lockup - CPU#0 stuck for 637s! [swapper:0]
>> Apr 9 16:17:43 host kernel: Modules linked in:
>> Apr 9 16:17:43 host kernel: CPU 0:
>> Apr 9 16:17:43 host kernel: Modules linked in:
>> Apr 9 16:17:43 host kernel: Pid: 0, comm: swapper Not tainted 2.6.32.9xls-domU #2
>> Apr 9 16:17:43 host kernel: RIP: e030:[<ffffffff810093aa>] [<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
>> Apr 9 16:17:44 host kernel: RSP: e02b:ffffffff81691f70 EFLAGS: 00000246
>> Apr 9 16:17:44 host kernel: RAX: 0000000000000000 RBX: ffffffff81690000 RCX: ffffffff810093aa
>> Apr 9 16:17:44 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000001
>> Apr 9 16:17:44 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000 R09: ffffffff8100e3b2
>> Apr 9 16:17:44 host kernel: R10: 0000000000000001 R11: 0000000000000246 R12: ffffffffffffffff
>> Apr 9 16:17:44 host kernel: R13: ffffffff818ebf20 R14: ffffffff818eec70 R15: 0000000000000000
>> Apr 9 16:17:44 host kernel: FS: 00007f07704fe6e0(0000) GS:ffff8800022ac000(0000) knlGS:0000000000000000
>> Apr 9 16:17:44 host kernel: CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b
>> Apr 9 16:17:44 host kernel: CR2: 00000000004505b0 CR3: 000000007ea99000 CR4: 0000000000002660
>> Apr 9 16:17:44 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
>> Apr 9 16:17:44 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
>> Apr 9 16:17:44 host kernel: Call Trace:
>> Apr 9 16:17:44 host kernel: [<ffffffff8100ddb7>] ? xen_safe_halt+0xc/0x15
>> Apr 9 16:17:44 host kernel: [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40
>> Apr 9 16:17:44 host kernel: [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82
>> Apr 9 16:17:44 host kernel: [<ffffffff818b6c42>] ? start_kernel+0x353/0x35f
>> Apr 9 16:20:22 host kernel: BUG: soft lockup - CPU#1 stuck for 147s! [swapper:0]
>> Apr 9 16:20:22 host kernel: Modules linked in:
>> Apr 9 16:20:22 host kernel: CPU 1:
>> Apr 9 16:20:22 host kernel: Modules linked in:
>> Apr 9 16:20:22 host kernel: Pid: 0, comm: swapper Not tainted 2.6.32.9xls-domU #2
>> Apr 9 16:20:22 host kernel: RIP: e030:[<ffffffff810093aa>] [<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
>> Apr 9 16:20:22 host kernel: RSP: e02b:ffff88007f8a1f08 EFLAGS: 00000246
>> Apr 9 16:20:22 host kernel: RAX: 0000000000000000 RBX: ffff88007f8a0000 RCX: ffffffff810093aa
>> Apr 9 16:20:22 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000001
>> Apr 9 16:20:22 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000 R09: ffffffff8100e3b2
>> Apr 9 16:20:22 host kernel: R10: 0000000000000001 R11: 0000000000000246 R12: 0000000000000000
>> Apr 9 16:20:22 host kernel: R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
>> Apr 9 16:20:22 host kernel: FS: 00007ff39e495710(0000) GS:ffff8800022c7000(0000) knlGS:0000000000000000
>> Apr 9 16:20:22 host kernel: CS: e033 DS: 002b ES: 002b CR0: 000000008005003b
>> Apr 9 16:20:22 host kernel: CR2: 00007ff39aec11a7 CR3: 0000000071c86000 CR4: 0000000000002660
>> Apr 9 16:20:22 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
>> Apr 9 16:20:22 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
>> Apr 9 16:20:22 host kernel: Call Trace:
>> Apr 9 16:20:22 host kernel: [<ffffffff8100ddb7>] ? xen_safe_halt+0xc/0x15
>> Apr 9 16:20:22 host kernel: [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40
>> Apr 9 16:20:22 host kernel: [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82
>> Apr 9 16:20:22 host kernel: BUG: soft lockup - CPU#0 stuck for 147s! [swapper:0]
>> Apr 9 16:20:22 host kernel: Modules linked in:
>> Apr 9 16:20:22 host kernel: CPU 0:
>> Apr 9 16:20:22 host kernel: Modules linked in:
>> Apr 9 16:20:22 host kernel: Pid: 0, comm: swapper Not tainted 2.6.32.9xls-domU #2
>> Apr 9 16:20:22 host kernel: RIP: e030:[<ffffffff810093aa>] [<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
>> Apr 9 16:20:22 host kernel: RSP: e02b:ffffffff81691f70 EFLAGS: 00000246
>> Apr 9 16:20:22 host kernel: RAX: 0000000000000000 RBX: ffffffff81690000 RCX: ffffffff810093aa
>> Apr 9 16:20:22 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000001
>> Apr 9 16:20:22 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000 R09: ffffffff8100e3b2
>> Apr 9 16:20:22 host kernel: R10: 0000000000000001 R11: 0000000000000246 R12: ffffffffffffffff
>> Apr 9 16:20:22 host kernel: R13: ffffffff818ebf20 R14: ffffffff818eec70 R15: 0000000000000000
>> Apr 9 16:20:22 host kernel: FS: 00007f07704fe6e0(0000) GS:ffff8800022ac000(0000) knlGS:0000000000000000
>> Apr 9 16:20:22 host kernel: CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b
>> Apr 9 16:20:22 host kernel: CR2: 00000000006ddc20 CR3: 0000000071c86000 CR4: 0000000000002660
>> Apr 9 16:20:22 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
>> Apr 9 16:20:22 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
>> Apr 9 16:20:22 host kernel: Call Trace:
>> Apr 9 16:20:22 host kernel: [<ffffffff8100ddb7>] ? xen_safe_halt+0xc/0x15
>> Apr 9 16:20:22 host kernel: [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40
>> Apr 9 16:20:22 host kernel: [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82
>> Apr 9 16:20:22 host kernel: [<ffffffff818b6c42>] ? start_kernel+0x353/0x35f
>> Apr 9 16:22:29 host kernel: BUG: soft lockup - CPU#1 stuck for 119s! [swapper:0]
>> Apr 9 16:22:29 host kernel: Modules linked in:
>> Apr 9 16:22:29 host kernel: CPU 1:
>> Apr 9 16:22:29 host kernel: Modules linked in:
>> Apr 9 16:22:29 host kernel: Pid: 0, comm: swapper Not tainted 2.6.32.9xls-domU #2
>> Apr 9 16:22:29 host kernel: RIP: e030:[<ffffffff810093aa>] [<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
>> Apr 9 16:22:29 host kernel: RSP: e02b:ffff88007f8a1f08 EFLAGS: 00000246
>> Apr 9 16:22:29 host kernel: RAX: 0000000000000000 RBX: ffff88007f8a0000 RCX: ffffffff810093aa
>> Apr 9 16:22:29 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000001
>> Apr 9 16:22:29 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000 R09: ffffffff8100e3b2
>> Apr 9 16:22:29 host kernel: R10: 0000000000000001 R11: 0000000000000246 R12: 0000000000000000
>> Apr 9 16:22:29 host kernel: R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
>> Apr 9 16:22:29 host kernel: FS: 00007ff39e495710(0000) GS:ffff8800022c7000(0000) knlGS:0000000000000000
>> Apr 9 16:22:29 host kernel: CS: e033 DS: 002b ES: 002b CR0: 000000008005003b
>> Apr 9 16:22:29 host kernel: CR2: 00007ff39c506460 CR3: 0000000026469000 CR4: 0000000000002660
>> Apr 9 16:22:29 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
>> Apr 9 16:22:29 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
>> Apr 9 16:22:29 host kernel: Call Trace:
>> Apr 9 16:22:29 host kernel: [<ffffffff8100ddb7>] ? xen_safe_halt+0xc/0x15
>> Apr 9 16:22:29 host kernel: [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40
>> Apr 9 16:22:29 host kernel: [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82
>> Apr 9 16:22:29 host kernel: BUG: soft lockup - CPU#0 stuck for 119s! [syslogd:1848]
>> Apr 9 16:22:29 host kernel: Modules linked in:
>> Apr 9 16:22:29 host kernel: CPU 0:
>> Apr 9 16:22:29 host kernel: Modules linked in:
>> Apr 9 16:22:29 host kernel: Pid: 1848, comm: syslogd Not tainted 2.6.32.9xls-domU #2
>> Apr 9 16:22:29 host kernel: RIP: 0033:[<00007f74fd9e4ad5>] [<00007f74fd9e4ad5>] 0x7f74fd9e4ad5
>> Apr 9 16:22:29 host kernel: RSP: 002b:00007fff22fa00d0 EFLAGS: 00000246
>> Apr 9 16:22:29 host kernel: RAX: 000000000000002e RBX: 0000000000000000 RCX: ffffffff810092eb
>> Apr 9 16:22:29 host kernel: RDX: 00000000000003fe RSI: 00007fff22fa01b0 RDI: 0000000000000000
>> Apr 9 16:22:30 host kernel: RBP: 00000000000003fe R08: 0000000000000000 R09: 0000000000000000
>> Apr 9 16:22:30 host kernel: R10: 0000000000000000 R11: 0000000000000246 R12: 00007fff22fa01b0
>> Apr 9 16:22:30 host kernel: R13: 0000000000000000 R14: 00007fff22fa0120 R15: 00007f74fde74690
>> Apr 9 16:22:30 host kernel: FS: 00007f74fde746e0(0000) GS:ffff8800022ac000(0000) knlGS:0000000000000000
>> Apr 9 16:22:30 host kernel: CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b
>> Apr 9 16:22:30 host kernel: CR2: 00007f0e429d6000 CR3: 000000007d03a000 CR4: 0000000000002660
>> Apr 9 16:22:30 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
>> Apr 9 16:22:30 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
>> Apr 9 16:22:30 host kernel: Call Trace:
>>
>
> On Apr 9, 2010, at 15:22 , Pim van Riezen wrote:
>
>
>> Good day,
>>
>> I'm getting lots of these from a couple of pvops DomU guests that are pretty heavy in their i/o and cpu-load. The issue reproduces on pvops 2.6.31 and 2.6.32 trees under Xen 3.3 as well as 3.4.1 (in the process of trying out 3.4.2 but I'm not holding my breath). Is there anything we can do be more helpful towards solving this issue?
>>
>> BUG: soft lockup - CPU#0 stuck for 561s! [swapper:0]
>> Modules linked in:
>> CPU 0:
>> Modules linked in:
>> Pid: 0, comm: swapper Not tainted 2.6.32.9xls-domU #2
>> RIP: e030:[<ffffffff810093aa>] [<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
>> RSP: e02b:ffffffff81691f70 EFLAGS: 00000246
>> RAX: 0000000000000000 RBX: ffffffff81690000 RCX: ffffffff810093aa
>> RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000001
>> RBP: ffffffff81896d30 R08: 0000000000000000 R09: ffffffff8100e3b2
>> R10: 0000000000000001 R11: 0000000000000246 R12: ffffffffffffffff
>> R13: ffffffff818ebf20 R14: ffffffff818eec70 R15: 0000000000000000
>> FS: 00007f8ac7a9c6e0(0000) GS:ffff8800022ac000(0000) knlGS:0000000000000000
>> CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b
>> CR2: 00007f291b4c9000 CR3: 000000007d8c1000 CR4: 0000000000002660
>> DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
>> DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
>> Call Trace:
>> [<ffffffff8100ddb7>] ? xen_safe_halt+0xc/0x15
>> [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40
>> [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82
>> [<ffffffff818b6c42>] ? start_kernel+0x353/0x35f
>>
>> Kind regards,
>> Pim van Riezen
>>
>>
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xensource.com
>> http://lists.xensource.com/xen-devel
>>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
>
>
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: pvops domu soft lockup under load (more logs)
2010-04-09 18:43 ` Jeremy Fitzhardinge
@ 2010-04-10 7:12 ` Pim van Riezen
2010-04-12 9:44 ` Pim van Riezen
0 siblings, 1 reply; 17+ messages in thread
From: Pim van Riezen @ 2010-04-10 7:12 UTC (permalink / raw)
To: Jeremy Fitzhardinge
Hi Jeremy,
On Apr 9, 2010, at 20:43 , Jeremy Fitzhardinge wrote:
> On 04/09/2010 09:50 AM, Pim van Riezen wrote:
>> Some more logs of this bug in action. Stack seems to be all over the place.
>>
>
> What kernel is this? Did you build it, or does it come from a distro?
This is a a kernel.org vanilla build of 2.6.32.9, I pasted the .config at the bottom of this message. It's built as x86_64 on a CentOS5 system with the CentOS dom0-kernel and Xen 3.4.1 from the gitco repo.
> What devices are you using within the domain? What storage backend are
> you using?
It's a reasonably bog standard internet server. All domains get their disk devices from a fibrechannel lun with lvm, through phy:/dev/mapper/volume in their config. Briding is used for the vifs.
> Could you include the full kernel output?
I assume you mean the boottime kmesg? Pasted underneath.
>
> When you're getting lockups, could you capture the output of sysrq-t (xm
> sysrq <domain> t, or "Ctrl-O t" on the domain's console)?
This one is going to be a bit tough. I only get the error after the fact. The domain has multiple vcpu's, so it's usually not apparent that it's hanging, all depending on which process gets stuck. I'll keep poking at it and if I manage to catch it in the act I'll follow up with a sysrq-t.
In terms of behavior, an interesting point to note is that one of the effects I see is that processes that rely in any way on timing (for example through select() or usleep()) get seriously messed up. I've been able to catch a process in the act a couple of days back with gdb and a select()-statement that had a {tv_sec=1, tv_usec=0} in its source code had its timeval struct severely mutilated. The only one I logged ended up like {tv_sec = 3833, tv_usec = 27346}.
*** Kernel output:
Apr 8 00:54:10 host kernel: klogd 1.4.1, log source = /proc/kmsg started.
Apr 8 00:54:10 host kernel: Linux version 2.6.32.9xls-domU (root@pablo) (gcc version 4.3.3 (Ubuntu 4.3.3-5ubuntu4) ) #2 SMP Mon Apr 5 21:43:30 CEST 2010
Apr 8 00:54:10 host kernel: Command line: root=/dev/xvda1 ro
Apr 8 00:54:10 host kernel: KERNEL supported cpus:
Apr 8 00:54:10 host kernel: Intel GenuineIntel
Apr 8 00:54:10 host kernel: AMD AuthenticAMD
Apr 8 00:54:10 host kernel: Centaur CentaurHauls
Apr 8 00:54:10 host kernel: ACPI in unprivileged domain disabled
Apr 8 00:54:10 host kernel: BIOS-provided physical RAM map:
Apr 8 00:54:10 host kernel: Xen: 0000000000000000 - 00000000000a0000 (usable)
Apr 8 00:54:10 host kernel: Xen: 00000000000a0000 - 0000000000100000 (reserved)
Apr 8 00:54:10 host kernel: Xen: 0000000000100000 - 0000000080000000 (usable)
Apr 8 00:54:10 host kernel: DMI not present or invalid.
Apr 8 00:54:10 host kernel: last_pfn = 0x80000 max_arch_pfn = 0x400000000
Apr 8 00:54:10 host kernel: init_memory_mapping: 0000000000000000-0000000080000000
Apr 8 00:54:10 host kernel: (6 early reservations) ==> bootmem [0000000000 - 0080000000]
Apr 8 00:54:10 host kernel: #0 [0000000000 - 0000001000] BIOS data page ==> [0000000000 - 0000001000]
Apr 8 00:54:10 host kernel: #1 [0002284000 - 0002299000] XEN PAGETABLES ==> [0002284000 - 0002299000]
Apr 8 00:54:10 host kernel: #2 [0000006000 - 0000008000] TRAMPOLINE ==> [0000006000 - 0000008000]
Apr 8 00:54:10 host kernel: #3 [0001000000 - 0001e60748] TEXT DATA BSS ==> [0001000000 - 0001e60748]
Apr 8 00:54:10 host kernel: #4 [0001e81000 - 0002284000] XEN START INFO ==> [0001e81000 - 0002284000]
Apr 8 00:54:10 host kernel: #5 [0000100000 - 00004eb000] PGTABLE ==> [0000100000 - 00004eb000]
Apr 8 00:54:10 host kernel: Zone PFN ranges:
Apr 8 00:54:10 host kernel: DMA 0x00000000 -> 0x00001000
Apr 8 00:54:10 host kernel: DMA32 0x00001000 -> 0x00100000
Apr 8 00:54:10 host kernel: Normal 0x00100000 -> 0x00100000
Apr 8 00:54:10 host kernel: Movable zone start PFN for each node
Apr 8 00:54:10 host kernel: early_node_map[2] active PFN ranges
Apr 8 00:54:10 host kernel: 0: 0x00000000 -> 0x000000a0
Apr 8 00:54:10 host kernel: 0: 0x00000100 -> 0x00080000
Apr 8 00:54:10 host kernel: SMP: Allowing 2 CPUs, 0 hotplug CPUs
Apr 8 00:54:10 host kernel: No local APIC present
Apr 8 00:54:10 host kernel: APIC: disable apic facility
Apr 8 00:54:10 host kernel: Allocating PCI resources starting at 80000000 (gap: 80000000:80000000)
Apr 8 00:54:10 host kernel: Booting paravirtualized kernel on Xen
Apr 8 00:54:10 host kernel: Xen version: 3.4.1 (preserve-AD)
Apr 8 00:54:10 host kernel: NR_CPUS:255 nr_cpumask_bits:255 nr_cpu_ids:2 nr_node_ids:1
Apr 8 00:54:10 host kernel: PERCPU: Embedded 27 pages/cpu @ffff8800022ac000 s79128 r8192 d23272 u110592
Apr 8 00:54:10 host kernel: pcpu-alloc: s79128 r8192 d23272 u110592 alloc=27*4096
Apr 8 00:54:10 host kernel: pcpu-alloc: [0] 0 [0] 1
Apr 8 00:54:10 host kernel: Xen: using vcpu_info placement
Apr 8 00:54:10 host kernel: Built 1 zonelists in Zone order, mobility grouping on. Total pages: 516018
Apr 8 00:54:10 host kernel: Kernel command line: root=/dev/xvda1 ro
Apr 8 00:54:10 host kernel: PID hash table entries: 4096 (order: 3, 32768 bytes)
Apr 8 00:54:10 host kernel: Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes)
Apr 8 00:54:10 host kernel: Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes)
Apr 8 00:54:10 host kernel: Initializing CPU#0
Apr 8 00:54:10 host kernel: Checking aperture...
Apr 8 00:54:10 host kernel: No AGP bridge found
Apr 8 00:54:10 host kernel: Memory: 2041016k/2097152k available (4984k kernel code, 384k absent, 55100k reserved, 3849k data, 508k init)
Apr 8 00:54:10 host kernel: Hierarchical RCU implementation.
Apr 8 00:54:10 host kernel: NR_IRQS:4352
Apr 8 00:54:10 host kernel: Console: colour dummy device 80x25
Apr 8 00:54:11 host kernel: console [tty0] enabled
Apr 8 00:54:11 host kernel: console [hvc0] enabled
Apr 8 00:54:11 host kernel: installing Xen timer for CPU 0
Apr 8 00:54:11 host kernel: Detected 2826.248 MHz processor.
Apr 8 00:54:11 host kernel: Calibrating delay loop (skipped), value calculated using timer frequency.. 5652.49 BogoMIPS (lpj=11304992)
Apr 8 00:54:11 host kernel: Security Framework initialized
Apr 8 00:54:11 host kernel: SELinux: Disabled at boot.
Apr 8 00:54:11 host kernel: Mount-cache hash table entries: 256
Apr 8 00:54:11 host kernel: CPU: L1 I cache: 32K, L1 D cache: 32K
Apr 8 00:54:11 host kernel: CPU: L2 cache: 6144K
Apr 8 00:54:11 host kernel: CPU: Unsupported number of siblings 4
Apr 8 00:54:11 host kernel: SMP alternatives: switching to UP code
Apr 8 00:54:11 host kernel: installing Xen timer for CPU 1
Apr 8 00:54:11 host kernel: SMP alternatives: switching to SMP code
Apr 8 00:54:11 host kernel: Initializing CPU#1
Apr 8 00:54:11 host kernel: CPU: L1 I cache: 32K, L1 D cache: 32K
Apr 8 00:54:11 host kernel: CPU: L2 cache: 6144K
Apr 8 00:54:11 host kernel: CPU: Unsupported number of siblings 4
Apr 8 00:54:11 host kernel: Brought up 2 CPUs
Apr 8 00:54:11 host kernel: xor: automatically using best checksumming function: generic_sse
Apr 8 00:54:11 host kernel: generic_sse: 2977.000 MB/sec
Apr 8 00:54:11 host kernel: xor: using function: generic_sse (2977.000 MB/sec)
Apr 8 00:54:11 host kernel: Grant table initialized
Apr 8 00:54:11 host kernel: NET: Registered protocol family 16
Apr 8 00:54:11 host kernel: PCI: Fatal: No config space access function found
Apr 8 00:54:11 host kernel: bio: create slab <bio-0> at 0
Apr 8 00:54:11 host kernel: ACPI: Interpreter disabled.
Apr 8 00:54:11 host kernel: xen_balloon: Initialising balloon driver.
Apr 8 00:54:11 host kernel: vgaarb: loaded
Apr 8 00:54:11 host kernel: SCSI subsystem initialized
Apr 8 00:54:11 host kernel: raid6: int64x1 2274 MB/s
Apr 8 00:54:11 host kernel: raid6: int64x2 3065 MB/s
Apr 8 00:54:11 host kernel: raid6: int64x4 2541 MB/s
Apr 8 00:54:11 host kernel: raid6: int64x8 2060 MB/s
Apr 8 00:54:11 host kernel: raid6: sse2x1 2855 MB/s
Apr 8 00:54:11 host kernel: raid6: sse2x2 6404 MB/s
Apr 8 00:54:11 host kernel: raid6: sse2x4 6320 MB/s
Apr 8 00:54:11 host kernel: raid6: using algorithm sse2x2 (6404 MB/s)
Apr 8 00:54:12 host kernel: PCI: System does not support PCI
Apr 8 00:54:12 host kernel: PCI: System does not support PCI
Apr 8 00:54:12 host kernel: NetLabel: Initializing
Apr 8 00:54:12 host kernel: NetLabel: domain hash size = 128
Apr 8 00:54:12 host kernel: NetLabel: protocols = UNLABELED CIPSOv4
Apr 8 00:54:12 host kernel: NetLabel: unlabeled traffic allowed by default
Apr 8 00:54:12 host kernel: Switching to clocksource xen
Apr 8 00:54:12 host kernel: Slow work thread pool: Starting up
Apr 8 00:54:12 host kernel: Slow work thread pool: Ready
Apr 8 00:54:12 host kernel: FS-Cache: Loaded
Apr 8 00:54:12 host kernel: pnp: PnP ACPI: disabled
Apr 8 00:54:12 host kernel: NET: Registered protocol family 2
Apr 8 00:54:12 host kernel: IP route cache hash table entries: 65536 (order: 7, 524288 bytes)
Apr 8 00:54:12 host kernel: TCP established hash table entries: 262144 (order: 10, 4194304 bytes)
Apr 8 00:54:12 host kernel: TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)
Apr 8 00:54:12 host kernel: TCP: Hash tables configured (established 262144 bind 65536)
Apr 8 00:54:12 host kernel: TCP reno registered
Apr 8 00:54:13 host kernel: NET: Registered protocol family 1
Apr 8 00:54:13 host kernel: RPC: Registered udp transport module.
Apr 8 00:54:13 host kernel: RPC: Registered tcp transport module.
Apr 8 00:54:13 host kernel: RPC: Registered tcp NFSv4.1 backchannel transport module.
Apr 8 00:54:13 host kernel: platform rtc_cmos: registered platform RTC device (no PNP device found)
Apr 8 00:54:13 host kernel: audit: initializing netlink socket (disabled)
Apr 8 00:54:13 host kernel: type=2000 audit(1270680838.227:1): initialized
Apr 8 00:54:13 host kernel: VFS: Disk quotas dquot_6.5.2
Apr 8 00:54:13 host kernel: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
Apr 8 00:54:13 host kernel: FS-Cache: Netfs 'nfs' registered for caching
Apr 8 00:54:13 host kernel: Installing knfsd (copyright (C) 1996 okir@monad.swb.de).
Apr 8 00:54:13 host kernel: fuse init (API version 7.13)
Apr 8 00:54:13 host kernel: SGI XFS with ACLs, security attributes, large block/inode numbers, no debug enabled
Apr 8 00:54:13 host kernel: SGI XFS Quota Management subsystem
Apr 8 00:54:13 host kernel: msgmni has been set to 3987
Apr 8 00:54:13 host kernel: alg: No test for stdrng (krng)
Apr 8 00:54:13 host kernel: io scheduler noop registered
Apr 8 00:54:13 host kernel: io scheduler anticipatory registered
Apr 8 00:54:13 host kernel: io scheduler deadline registered
Apr 8 00:54:13 host kernel: io scheduler cfq registered (default)
Apr 8 00:54:13 host kernel: pci_hotplug: PCI Hot Plug PCI Core version: 0.5
Apr 8 00:54:13 host kernel: Event-channel device installed.
Apr 8 00:54:13 host kernel: Non-volatile memory driver v1.3
Apr 8 00:54:13 host kernel: Linux agpgart interface v0.103
Apr 8 00:54:13 host kernel: brd: module loaded
Apr 8 00:54:13 host kernel: loop: module loaded
Apr 8 00:54:13 host kernel: nbd: registered device at major 43
Apr 8 00:54:13 host kernel: Loading iSCSI transport class v2.0-870.
Apr 8 00:54:13 host kernel: iscsi: registered transport (tcp)
Apr 8 00:54:13 host kernel: PPP generic driver version 2.4.2
Apr 8 00:54:13 host kernel: PPP Deflate Compression module registered
Apr 8 00:54:13 host kernel: PPP BSD Compression module registered
Apr 8 00:54:13 host kernel: PPP MPPE Compression module registered
Apr 8 00:54:13 host kernel: Initialising Xen virtual ethernet driver.
Apr 8 00:54:14 host kernel: tun: Universal TUN/TAP device driver, 1.6
Apr 8 00:54:14 host kernel: tun: (C) 1999-2004 Max Krasnyansky <maxk@qualcomm.com>
Apr 8 00:54:14 host kernel: aoe: AoE v47 initialised.
Apr 8 00:54:14 host kernel: PNP: No PS/2 controller found. Probing ports directly.
Apr 8 00:54:14 host kernel: i8042.c: No controller found.
Apr 8 00:54:14 host kernel: mice: PS/2 mouse device common for all mice
Apr 8 00:54:14 host kernel: rtc_cmos: probe of rtc_cmos failed with error -16
Apr 8 00:54:14 host kernel: md: linear personality registered for level -1
Apr 8 00:54:14 host kernel: md: raid0 personality registered for level 0
Apr 8 00:54:14 host kernel: md: raid1 personality registered for level 1
Apr 8 00:54:14 host kernel: md: raid6 personality registered for level 6
Apr 8 00:54:14 host kernel: md: raid5 personality registered for level 5
Apr 8 00:54:14 host kernel: md: raid4 personality registered for level 4
Apr 8 00:54:14 host kernel: md: multipath personality registered for level -4
Apr 8 00:54:14 host kernel: device-mapper: uevent: version 1.0.3
Apr 8 00:54:14 host kernel: device-mapper: ioctl: 4.15.0-ioctl (2009-04-01) initialised: dm-devel@redhat.com
Apr 8 00:54:14 host kernel: device-mapper: multipath: version 1.1.0 loaded
Apr 8 00:54:14 host kernel: device-mapper: multipath round-robin: version 1.0.0 loaded
Apr 8 00:54:14 host kernel: cpuidle: using governor ladder
Apr 8 00:54:14 host kernel: GACT probability on
Apr 8 00:54:14 host kernel: Mirror/redirect action on
Apr 8 00:54:14 host kernel: netem: version 1.2
Apr 8 00:54:14 host kernel: Netfilter messages via NETLINK v0.30.
Apr 8 00:54:14 host kernel: nf_conntrack version 0.5.0 (16384 buckets, 65536 max)
Apr 8 00:54:14 host kernel: CONFIG_NF_CT_ACCT is deprecated and will be removed soon. Please use
Apr 8 00:54:14 host kernel: nf_conntrack.acct=1 kernel parameter, acct=1 nf_conntrack module option or
Apr 8 00:54:14 host kernel: sysctl net.netfilter.nf_conntrack_acct=1 to enable it.
Apr 8 00:54:14 host kernel: ctnetlink v0.93: registering with nfnetlink.
Apr 8 00:54:14 host kernel: xt_time: kernel timezone is -0000
Apr 8 00:54:14 host kernel: IPv4 over IPv4 tunneling driver
Apr 8 00:54:14 host kernel: GRE over IPv4 tunneling driver
Apr 8 00:54:14 host kernel: ip_tables: (C) 2000-2006 Netfilter Core Team
Apr 8 00:54:14 host kernel: TCP bic registered
Apr 8 00:54:14 host kernel: Initializing XFRM netlink socket
Apr 8 00:54:14 host kernel: NET: Registered protocol family 10
Apr 8 00:54:14 host kernel: lo: Disabled Privacy Extensions
Apr 8 00:54:14 host kernel: tunl0: Disabled Privacy Extensions
Apr 8 00:54:14 host kernel: ip6_tables: (C) 2000-2006 Netfilter Core Team
Apr 8 00:54:14 host kernel: NET: Registered protocol family 17
Apr 8 00:54:14 host kernel: SCTP: Hash tables configured (established 65536 bind 65536)
Apr 8 00:54:14 host kernel: registered taskstats version 1
Apr 8 00:54:15 host kernel: XENBUS: Device with no driver: device/console/0
Apr 8 00:54:15 host kernel: drivers/rtc/hctosys.c: unable to open rtc device (rtc0)
Apr 8 00:54:15 host kernel: md: Waiting for all devices to be available before autodetect
Apr 8 00:54:15 host kernel: md: If you don't use raid, use raid=noautodetect
Apr 8 00:54:15 host kernel: md: Autodetecting RAID arrays.
Apr 8 00:54:15 host kernel: md: Scanned 0 and added 0 devices.
Apr 8 00:54:15 host kernel: md: autorun ...
Apr 8 00:54:15 host kernel: md: ... autorun DONE.
Apr 8 00:54:15 host kernel: kjournald starting. Commit interval 5 seconds
Apr 8 00:54:15 host kernel: EXT3-fs: mounted filesystem with writeback data mode.
Apr 8 00:54:15 host kernel: VFS: Mounted root (ext3 filesystem) readonly on device 202:1.
Apr 8 00:54:15 host kernel: Freeing unused kernel memory: 508k freed
Apr 8 00:54:15 host kernel: Write protecting the kernel read-only data: 6684k
Apr 8 00:54:15 host kernel: md: Autodetecting RAID arrays.
Apr 8 00:54:15 host kernel: md: Scanned 0 and added 0 devices.
Apr 8 00:54:15 host kernel: md: autorun ...
Apr 8 00:54:15 host kernel: md: ... autorun DONE.
Apr 8 00:54:15 host kernel: EXT3 FS on xvda1, internal journal
Apr 8 00:54:15 host kernel: Adding 4194296k swap on /dev/xvda2. Priority:-1 extents:1 across:4194296k SS
Apr 8 00:54:15 host kernel: warning: process `kudzu' used the deprecated sysctl system call with 1.23.
Apr 8 00:54:15 host kernel: process `sysctl' is using deprecated sysctl (syscall) net.ipv6.neigh.default.retrans_time; Use net.ipv6.neigh.default.retrans_time_ms instead.
Apr 8 00:54:15 host kernel: warning: `dbus-daemon' uses 32-bit capabilities (legacy support in use)
*** Kernel config:
#
# Automatically generated make config: don't edit
# Linux kernel version: 2.6.32.9
# Mon Apr 5 21:05:31 2010
#
CONFIG_64BIT=y
# CONFIG_X86_32 is not set
CONFIG_X86_64=y
CONFIG_X86=y
CONFIG_OUTPUT_FORMAT="elf64-x86-64"
CONFIG_ARCH_DEFCONFIG="arch/x86/configs/x86_64_defconfig"
CONFIG_GENERIC_TIME=y
CONFIG_GENERIC_CMOS_UPDATE=y
CONFIG_CLOCKSOURCE_WATCHDOG=y
CONFIG_GENERIC_CLOCKEVENTS=y
CONFIG_GENERIC_CLOCKEVENTS_BROADCAST=y
CONFIG_LOCKDEP_SUPPORT=y
CONFIG_STACKTRACE_SUPPORT=y
CONFIG_HAVE_LATENCYTOP_SUPPORT=y
CONFIG_MMU=y
CONFIG_ZONE_DMA=y
CONFIG_GENERIC_ISA_DMA=y
CONFIG_GENERIC_IOMAP=y
CONFIG_GENERIC_BUG=y
CONFIG_GENERIC_BUG_RELATIVE_POINTERS=y
CONFIG_GENERIC_HWEIGHT=y
CONFIG_ARCH_MAY_HAVE_PC_FDC=y
CONFIG_RWSEM_GENERIC_SPINLOCK=y
# CONFIG_RWSEM_XCHGADD_ALGORITHM is not set
CONFIG_ARCH_HAS_CPU_IDLE_WAIT=y
CONFIG_GENERIC_CALIBRATE_DELAY=y
CONFIG_GENERIC_TIME_VSYSCALL=y
CONFIG_ARCH_HAS_CPU_RELAX=y
CONFIG_ARCH_HAS_DEFAULT_IDLE=y
CONFIG_ARCH_HAS_CACHE_LINE_SIZE=y
CONFIG_HAVE_SETUP_PER_CPU_AREA=y
CONFIG_NEED_PER_CPU_EMBED_FIRST_CHUNK=y
CONFIG_NEED_PER_CPU_PAGE_FIRST_CHUNK=y
CONFIG_HAVE_CPUMASK_OF_CPU_MAP=y
CONFIG_ARCH_HIBERNATION_POSSIBLE=y
CONFIG_ARCH_SUSPEND_POSSIBLE=y
CONFIG_ZONE_DMA32=y
CONFIG_ARCH_POPULATES_NODE_MAP=y
CONFIG_AUDIT_ARCH=y
CONFIG_ARCH_SUPPORTS_OPTIMIZED_INLINING=y
CONFIG_ARCH_SUPPORTS_DEBUG_PAGEALLOC=y
CONFIG_GENERIC_HARDIRQS=y
CONFIG_GENERIC_HARDIRQS_NO__DO_IRQ=y
CONFIG_GENERIC_IRQ_PROBE=y
CONFIG_GENERIC_PENDING_IRQ=y
CONFIG_USE_GENERIC_SMP_HELPERS=y
CONFIG_X86_64_SMP=y
CONFIG_X86_HT=y
CONFIG_X86_TRAMPOLINE=y
# CONFIG_KTIME_SCALAR is not set
CONFIG_DEFCONFIG_LIST="/lib/modules/$UNAME_RELEASE/.config"
CONFIG_CONSTRUCTORS=y
#
# General setup
#
CONFIG_EXPERIMENTAL=y
CONFIG_LOCK_KERNEL=y
CONFIG_INIT_ENV_ARG_LIMIT=32
CONFIG_LOCALVERSION="xls-domU"
# CONFIG_LOCALVERSION_AUTO is not set
CONFIG_HAVE_KERNEL_GZIP=y
CONFIG_HAVE_KERNEL_BZIP2=y
CONFIG_HAVE_KERNEL_LZMA=y
CONFIG_KERNEL_GZIP=y
# CONFIG_KERNEL_BZIP2 is not set
# CONFIG_KERNEL_LZMA is not set
CONFIG_SWAP=y
CONFIG_SYSVIPC=y
CONFIG_SYSVIPC_SYSCTL=y
CONFIG_POSIX_MQUEUE=y
CONFIG_POSIX_MQUEUE_SYSCTL=y
CONFIG_BSD_PROCESS_ACCT=y
# CONFIG_BSD_PROCESS_ACCT_V3 is not set
CONFIG_TASKSTATS=y
CONFIG_TASK_DELAY_ACCT=y
# CONFIG_TASK_XACCT is not set
CONFIG_AUDIT=y
CONFIG_AUDITSYSCALL=y
CONFIG_AUDIT_TREE=y
#
# RCU Subsystem
#
CONFIG_TREE_RCU=y
# CONFIG_TREE_PREEMPT_RCU is not set
# CONFIG_RCU_TRACE is not set
CONFIG_RCU_FANOUT=64
# CONFIG_RCU_FANOUT_EXACT is not set
# CONFIG_TREE_RCU_TRACE is not set
CONFIG_IKCONFIG=y
CONFIG_IKCONFIG_PROC=y
CONFIG_LOG_BUF_SHIFT=17
CONFIG_HAVE_UNSTABLE_SCHED_CLOCK=y
# CONFIG_GROUP_SCHED is not set
# CONFIG_CGROUPS is not set
# CONFIG_SYSFS_DEPRECATED_V2 is not set
CONFIG_RELAY=y
CONFIG_NAMESPACES=y
# CONFIG_UTS_NS is not set
# CONFIG_IPC_NS is not set
# CONFIG_USER_NS is not set
# CONFIG_PID_NS is not set
# CONFIG_NET_NS is not set
CONFIG_BLK_DEV_INITRD=y
CONFIG_INITRAMFS_SOURCE=""
CONFIG_RD_GZIP=y
CONFIG_RD_BZIP2=y
CONFIG_RD_LZMA=y
CONFIG_CC_OPTIMIZE_FOR_SIZE=y
CONFIG_SYSCTL=y
CONFIG_ANON_INODES=y
# CONFIG_EMBEDDED is not set
CONFIG_UID16=y
CONFIG_SYSCTL_SYSCALL=y
CONFIG_KALLSYMS=y
# CONFIG_KALLSYMS_ALL is not set
CONFIG_KALLSYMS_EXTRA_PASS=y
CONFIG_HOTPLUG=y
CONFIG_PRINTK=y
CONFIG_BUG=y
CONFIG_ELF_CORE=y
CONFIG_PCSPKR_PLATFORM=y
CONFIG_BASE_FULL=y
CONFIG_FUTEX=y
CONFIG_EPOLL=y
CONFIG_SIGNALFD=y
CONFIG_TIMERFD=y
CONFIG_EVENTFD=y
CONFIG_SHMEM=y
CONFIG_AIO=y
CONFIG_HAVE_PERF_EVENTS=y
#
# Kernel Performance Events And Counters
#
# CONFIG_PERF_EVENTS is not set
# CONFIG_PERF_COUNTERS is not set
CONFIG_VM_EVENT_COUNTERS=y
CONFIG_PCI_QUIRKS=y
# CONFIG_COMPAT_BRK is not set
CONFIG_SLAB=y
# CONFIG_SLUB is not set
# CONFIG_SLOB is not set
# CONFIG_PROFILING is not set
CONFIG_TRACEPOINTS=y
CONFIG_HAVE_OPROFILE=y
CONFIG_KPROBES=y
CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS=y
CONFIG_KRETPROBES=y
CONFIG_HAVE_IOREMAP_PROT=y
CONFIG_HAVE_KPROBES=y
CONFIG_HAVE_KRETPROBES=y
CONFIG_HAVE_ARCH_TRACEHOOK=y
CONFIG_HAVE_DMA_ATTRS=y
CONFIG_HAVE_DMA_API_DEBUG=y
#
# GCOV-based kernel profiling
#
# CONFIG_GCOV_KERNEL is not set
CONFIG_SLOW_WORK=y
# CONFIG_SLOW_WORK_DEBUG is not set
# CONFIG_HAVE_GENERIC_DMA_COHERENT is not set
CONFIG_SLABINFO=y
CONFIG_RT_MUTEXES=y
CONFIG_BASE_SMALL=0
CONFIG_MODULES=y
# CONFIG_MODULE_FORCE_LOAD is not set
CONFIG_MODULE_UNLOAD=y
# CONFIG_MODULE_FORCE_UNLOAD is not set
CONFIG_MODVERSIONS=y
CONFIG_MODULE_SRCVERSION_ALL=y
CONFIG_STOP_MACHINE=y
CONFIG_BLOCK=y
# CONFIG_BLK_DEV_BSG is not set
# CONFIG_BLK_DEV_INTEGRITY is not set
CONFIG_BLOCK_COMPAT=y
#
# IO Schedulers
#
CONFIG_IOSCHED_NOOP=y
CONFIG_IOSCHED_AS=y
CONFIG_IOSCHED_DEADLINE=y
CONFIG_IOSCHED_CFQ=y
# CONFIG_DEFAULT_AS is not set
# CONFIG_DEFAULT_DEADLINE is not set
CONFIG_DEFAULT_CFQ=y
# CONFIG_DEFAULT_NOOP is not set
CONFIG_DEFAULT_IOSCHED="cfq"
CONFIG_FREEZER=y
#
# Processor type and features
#
# CONFIG_NO_HZ is not set
# CONFIG_HIGH_RES_TIMERS is not set
CONFIG_GENERIC_CLOCKEVENTS_BUILD=y
CONFIG_SMP=y
# CONFIG_SPARSE_IRQ is not set
CONFIG_X86_MPPARSE=y
CONFIG_X86_EXTENDED_PLATFORM=y
# CONFIG_X86_VSMP is not set
CONFIG_SCHED_OMIT_FRAME_POINTER=y
CONFIG_PARAVIRT_GUEST=y
CONFIG_XEN=y
CONFIG_XEN_MAX_DOMAIN_MEMORY=32
CONFIG_XEN_SAVE_RESTORE=y
# CONFIG_XEN_DEBUG_FS is not set
# CONFIG_KVM_CLOCK is not set
# CONFIG_KVM_GUEST is not set
CONFIG_PARAVIRT=y
# CONFIG_PARAVIRT_SPINLOCKS is not set
CONFIG_PARAVIRT_CLOCK=y
# CONFIG_PARAVIRT_DEBUG is not set
# CONFIG_MEMTEST is not set
# CONFIG_M386 is not set
# CONFIG_M486 is not set
# CONFIG_M586 is not set
# CONFIG_M586TSC is not set
# CONFIG_M586MMX is not set
# CONFIG_M686 is not set
# CONFIG_MPENTIUMII is not set
# CONFIG_MPENTIUMIII is not set
# CONFIG_MPENTIUMM is not set
# CONFIG_MPENTIUM4 is not set
# CONFIG_MK6 is not set
# CONFIG_MK7 is not set
# CONFIG_MK8 is not set
# CONFIG_MCRUSOE is not set
# CONFIG_MEFFICEON is not set
# CONFIG_MWINCHIPC6 is not set
# CONFIG_MWINCHIP3D is not set
# CONFIG_MGEODEGX1 is not set
# CONFIG_MGEODE_LX is not set
# CONFIG_MCYRIXIII is not set
# CONFIG_MVIAC3_2 is not set
# CONFIG_MVIAC7 is not set
# CONFIG_MPSC is not set
# CONFIG_MCORE2 is not set
# CONFIG_MATOM is not set
CONFIG_GENERIC_CPU=y
CONFIG_X86_CPU=y
CONFIG_X86_L1_CACHE_BYTES=64
CONFIG_X86_INTERNODE_CACHE_BYTES=64
CONFIG_X86_CMPXCHG=y
CONFIG_X86_L1_CACHE_SHIFT=6
CONFIG_X86_WP_WORKS_OK=y
CONFIG_X86_TSC=y
CONFIG_X86_CMPXCHG64=y
CONFIG_X86_CMOV=y
CONFIG_X86_MINIMUM_CPU_FAMILY=64
CONFIG_X86_DEBUGCTLMSR=y
CONFIG_CPU_SUP_INTEL=y
CONFIG_CPU_SUP_AMD=y
CONFIG_CPU_SUP_CENTAUR=y
# CONFIG_X86_DS is not set
CONFIG_HPET_TIMER=y
CONFIG_HPET_EMULATE_RTC=y
CONFIG_DMI=y
CONFIG_GART_IOMMU=y
# CONFIG_CALGARY_IOMMU is not set
# CONFIG_AMD_IOMMU is not set
CONFIG_SWIOTLB=y
CONFIG_IOMMU_HELPER=y
# CONFIG_IOMMU_API is not set
# CONFIG_MAXSMP is not set
CONFIG_NR_CPUS=255
# CONFIG_SCHED_SMT is not set
CONFIG_SCHED_MC=y
# CONFIG_PREEMPT_NONE is not set
CONFIG_PREEMPT_VOLUNTARY=y
# CONFIG_PREEMPT is not set
CONFIG_X86_LOCAL_APIC=y
CONFIG_X86_IO_APIC=y
# CONFIG_X86_REROUTE_FOR_BROKEN_BOOT_IRQS is not set
# CONFIG_X86_MCE is not set
# CONFIG_I8K is not set
CONFIG_MICROCODE=m
CONFIG_MICROCODE_INTEL=y
# CONFIG_MICROCODE_AMD is not set
CONFIG_MICROCODE_OLD_INTERFACE=y
CONFIG_X86_MSR=y
CONFIG_X86_CPUID=y
CONFIG_ARCH_PHYS_ADDR_T_64BIT=y
CONFIG_DIRECT_GBPAGES=y
# CONFIG_NUMA is not set
CONFIG_ARCH_PROC_KCORE_TEXT=y
CONFIG_ARCH_SPARSEMEM_DEFAULT=y
CONFIG_ARCH_SPARSEMEM_ENABLE=y
CONFIG_ARCH_SELECT_MEMORY_MODEL=y
CONFIG_SELECT_MEMORY_MODEL=y
# CONFIG_FLATMEM_MANUAL is not set
# CONFIG_DISCONTIGMEM_MANUAL is not set
CONFIG_SPARSEMEM_MANUAL=y
CONFIG_SPARSEMEM=y
CONFIG_HAVE_MEMORY_PRESENT=y
CONFIG_SPARSEMEM_EXTREME=y
CONFIG_SPARSEMEM_VMEMMAP_ENABLE=y
CONFIG_SPARSEMEM_VMEMMAP=y
# CONFIG_MEMORY_HOTPLUG is not set
CONFIG_PAGEFLAGS_EXTENDED=y
CONFIG_SPLIT_PTLOCK_CPUS=4
CONFIG_PHYS_ADDR_T_64BIT=y
CONFIG_ZONE_DMA_FLAG=1
CONFIG_BOUNCE=y
CONFIG_VIRT_TO_BUS=y
CONFIG_HAVE_MLOCK=y
CONFIG_HAVE_MLOCKED_PAGE_BIT=y
# CONFIG_KSM is not set
CONFIG_DEFAULT_MMAP_MIN_ADDR=4096
# CONFIG_X86_CHECK_BIOS_CORRUPTION is not set
CONFIG_X86_RESERVE_LOW_64K=y
CONFIG_MTRR=y
CONFIG_MTRR_SANITIZER=y
CONFIG_MTRR_SANITIZER_ENABLE_DEFAULT=0
CONFIG_MTRR_SANITIZER_SPARE_REG_NR_DEFAULT=1
# CONFIG_X86_PAT is not set
# CONFIG_EFI is not set
# CONFIG_SECCOMP is not set
# CONFIG_CC_STACKPROTECTOR is not set
# CONFIG_HZ_100 is not set
CONFIG_HZ_250=y
# CONFIG_HZ_300 is not set
# CONFIG_HZ_1000 is not set
CONFIG_HZ=250
# CONFIG_SCHED_HRTICK is not set
CONFIG_KEXEC=y
CONFIG_CRASH_DUMP=y
CONFIG_PHYSICAL_START=0x200000
# CONFIG_RELOCATABLE is not set
CONFIG_PHYSICAL_ALIGN=0x1000000
CONFIG_HOTPLUG_CPU=y
CONFIG_COMPAT_VDSO=y
# CONFIG_CMDLINE_BOOL is not set
CONFIG_ARCH_ENABLE_MEMORY_HOTPLUG=y
#
# Power management and ACPI options
#
CONFIG_PM=y
# CONFIG_PM_DEBUG is not set
CONFIG_PM_SLEEP_SMP=y
CONFIG_PM_SLEEP=y
CONFIG_SUSPEND=y
CONFIG_SUSPEND_FREEZER=y
# CONFIG_HIBERNATION is not set
# CONFIG_PM_RUNTIME is not set
CONFIG_ACPI=y
CONFIG_ACPI_SLEEP=y
# CONFIG_ACPI_PROCFS is not set
CONFIG_ACPI_PROCFS_POWER=y
CONFIG_ACPI_SYSFS_POWER=y
CONFIG_ACPI_PROC_EVENT=y
CONFIG_ACPI_AC=m
CONFIG_ACPI_BATTERY=m
CONFIG_ACPI_BUTTON=m
CONFIG_ACPI_VIDEO=m
CONFIG_ACPI_FAN=y
CONFIG_ACPI_DOCK=y
CONFIG_ACPI_PROCESSOR=y
CONFIG_ACPI_HOTPLUG_CPU=y
# CONFIG_ACPI_PROCESSOR_AGGREGATOR is not set
CONFIG_ACPI_THERMAL=y
# CONFIG_ACPI_CUSTOM_DSDT is not set
CONFIG_ACPI_BLACKLIST_YEAR=0
# CONFIG_ACPI_DEBUG is not set
# CONFIG_ACPI_PCI_SLOT is not set
CONFIG_X86_PM_TIMER=y
CONFIG_ACPI_CONTAINER=y
CONFIG_ACPI_SBS=m
# CONFIG_SFI is not set
#
# CPU Frequency scaling
#
CONFIG_CPU_FREQ=y
CONFIG_CPU_FREQ_TABLE=y
CONFIG_CPU_FREQ_DEBUG=y
CONFIG_CPU_FREQ_STAT=m
CONFIG_CPU_FREQ_STAT_DETAILS=y
# CONFIG_CPU_FREQ_DEFAULT_GOV_PERFORMANCE is not set
# CONFIG_CPU_FREQ_DEFAULT_GOV_POWERSAVE is not set
CONFIG_CPU_FREQ_DEFAULT_GOV_USERSPACE=y
# CONFIG_CPU_FREQ_DEFAULT_GOV_ONDEMAND is not set
# CONFIG_CPU_FREQ_DEFAULT_GOV_CONSERVATIVE is not set
CONFIG_CPU_FREQ_GOV_PERFORMANCE=y
CONFIG_CPU_FREQ_GOV_POWERSAVE=m
CONFIG_CPU_FREQ_GOV_USERSPACE=y
CONFIG_CPU_FREQ_GOV_ONDEMAND=m
CONFIG_CPU_FREQ_GOV_CONSERVATIVE=m
#
# CPUFreq processor drivers
#
CONFIG_X86_ACPI_CPUFREQ=m
CONFIG_X86_POWERNOW_K8=y
CONFIG_X86_SPEEDSTEP_CENTRINO=y
# CONFIG_X86_P4_CLOCKMOD is not set
#
# shared options
#
# CONFIG_X86_SPEEDSTEP_LIB is not set
CONFIG_CPU_IDLE=y
CONFIG_CPU_IDLE_GOV_LADDER=y
#
# Memory power savings
#
# CONFIG_I7300_IDLE is not set
#
# Bus options (PCI etc.)
#
CONFIG_PCI=y
CONFIG_PCI_DIRECT=y
CONFIG_PCI_MMCONFIG=y
CONFIG_PCI_DOMAINS=y
CONFIG_PCIEPORTBUS=y
CONFIG_HOTPLUG_PCI_PCIE=m
CONFIG_PCIEAER=y
# CONFIG_PCIE_ECRC is not set
# CONFIG_PCIEAER_INJECT is not set
# CONFIG_PCIEASPM is not set
CONFIG_ARCH_SUPPORTS_MSI=y
# CONFIG_PCI_MSI is not set
CONFIG_PCI_LEGACY=y
# CONFIG_PCI_DEBUG is not set
# CONFIG_PCI_STUB is not set
CONFIG_HT_IRQ=y
# CONFIG_PCI_IOV is not set
CONFIG_ISA_DMA_API=y
CONFIG_K8_NB=y
CONFIG_PCCARD=y
# CONFIG_PCMCIA_DEBUG is not set
CONFIG_PCMCIA=y
CONFIG_PCMCIA_LOAD_CIS=y
CONFIG_PCMCIA_IOCTL=y
CONFIG_CARDBUS=y
#
# PC-card bridges
#
CONFIG_YENTA=y
CONFIG_YENTA_O2=y
CONFIG_YENTA_RICOH=y
CONFIG_YENTA_TI=y
CONFIG_YENTA_ENE_TUNE=y
CONFIG_YENTA_TOSHIBA=y
CONFIG_PD6729=m
# CONFIG_I82092 is not set
CONFIG_PCCARD_NONSTATIC=y
CONFIG_HOTPLUG_PCI=y
CONFIG_HOTPLUG_PCI_FAKE=m
CONFIG_HOTPLUG_PCI_ACPI=m
CONFIG_HOTPLUG_PCI_ACPI_IBM=m
# CONFIG_HOTPLUG_PCI_CPCI is not set
CONFIG_HOTPLUG_PCI_SHPC=m
#
# Executable file formats / Emulations
#
CONFIG_BINFMT_ELF=y
CONFIG_COMPAT_BINFMT_ELF=y
# CONFIG_CORE_DUMP_DEFAULT_ELF_HEADERS is not set
# CONFIG_HAVE_AOUT is not set
CONFIG_BINFMT_MISC=y
CONFIG_IA32_EMULATION=y
# CONFIG_IA32_AOUT is not set
CONFIG_COMPAT=y
CONFIG_COMPAT_FOR_U64_ALIGNMENT=y
CONFIG_SYSVIPC_COMPAT=y
CONFIG_NET=y
CONFIG_COMPAT_NETLINK_MESSAGES=y
#
# Networking options
#
CONFIG_PACKET=y
CONFIG_PACKET_MMAP=y
CONFIG_UNIX=y
CONFIG_XFRM=y
CONFIG_XFRM_USER=y
# CONFIG_XFRM_SUB_POLICY is not set
# CONFIG_XFRM_MIGRATE is not set
# CONFIG_XFRM_STATISTICS is not set
CONFIG_XFRM_IPCOMP=y
# CONFIG_NET_KEY is not set
CONFIG_INET=y
CONFIG_IP_MULTICAST=y
CONFIG_IP_ADVANCED_ROUTER=y
CONFIG_ASK_IP_FIB_HASH=y
# CONFIG_IP_FIB_TRIE is not set
CONFIG_IP_FIB_HASH=y
CONFIG_IP_MULTIPLE_TABLES=y
CONFIG_IP_ROUTE_MULTIPATH=y
CONFIG_IP_ROUTE_VERBOSE=y
# CONFIG_IP_PNP is not set
CONFIG_NET_IPIP=y
CONFIG_NET_IPGRE=y
CONFIG_NET_IPGRE_BROADCAST=y
CONFIG_IP_MROUTE=y
CONFIG_IP_PIMSM_V1=y
CONFIG_IP_PIMSM_V2=y
# CONFIG_ARPD is not set
CONFIG_SYN_COOKIES=y
CONFIG_INET_AH=y
CONFIG_INET_ESP=y
CONFIG_INET_IPCOMP=y
CONFIG_INET_XFRM_TUNNEL=y
CONFIG_INET_TUNNEL=y
CONFIG_INET_XFRM_MODE_TRANSPORT=y
CONFIG_INET_XFRM_MODE_TUNNEL=y
CONFIG_INET_XFRM_MODE_BEET=y
CONFIG_INET_LRO=y
# CONFIG_INET_DIAG is not set
CONFIG_TCP_CONG_ADVANCED=y
CONFIG_TCP_CONG_BIC=y
CONFIG_TCP_CONG_CUBIC=m
CONFIG_TCP_CONG_WESTWOOD=m
CONFIG_TCP_CONG_HTCP=m
CONFIG_TCP_CONG_HSTCP=m
CONFIG_TCP_CONG_HYBLA=m
CONFIG_TCP_CONG_VEGAS=m
CONFIG_TCP_CONG_SCALABLE=m
CONFIG_TCP_CONG_LP=m
CONFIG_TCP_CONG_VENO=m
# CONFIG_TCP_CONG_YEAH is not set
# CONFIG_TCP_CONG_ILLINOIS is not set
CONFIG_DEFAULT_BIC=y
# CONFIG_DEFAULT_CUBIC is not set
# CONFIG_DEFAULT_HTCP is not set
# CONFIG_DEFAULT_VEGAS is not set
# CONFIG_DEFAULT_WESTWOOD is not set
# CONFIG_DEFAULT_RENO is not set
CONFIG_DEFAULT_TCP_CONG="bic"
# CONFIG_TCP_MD5SIG is not set
CONFIG_IPV6=y
CONFIG_IPV6_PRIVACY=y
CONFIG_IPV6_ROUTER_PREF=y
CONFIG_IPV6_ROUTE_INFO=y
CONFIG_IPV6_OPTIMISTIC_DAD=y
CONFIG_INET6_AH=m
CONFIG_INET6_ESP=m
CONFIG_INET6_IPCOMP=m
# CONFIG_IPV6_MIP6 is not set
CONFIG_INET6_XFRM_TUNNEL=m
CONFIG_INET6_TUNNEL=m
CONFIG_INET6_XFRM_MODE_TRANSPORT=m
CONFIG_INET6_XFRM_MODE_TUNNEL=m
CONFIG_INET6_XFRM_MODE_BEET=m
# CONFIG_INET6_XFRM_MODE_ROUTEOPTIMIZATION is not set
CONFIG_IPV6_SIT=m
CONFIG_IPV6_NDISC_NODETYPE=y
CONFIG_IPV6_TUNNEL=m
CONFIG_IPV6_MULTIPLE_TABLES=y
# CONFIG_IPV6_SUBTREES is not set
# CONFIG_IPV6_MROUTE is not set
CONFIG_NETLABEL=y
CONFIG_NETWORK_SECMARK=y
CONFIG_NETFILTER=y
# CONFIG_NETFILTER_DEBUG is not set
CONFIG_NETFILTER_ADVANCED=y
#
# Core Netfilter Configuration
#
CONFIG_NETFILTER_NETLINK=y
# CONFIG_NETFILTER_NETLINK_QUEUE is not set
CONFIG_NETFILTER_NETLINK_LOG=y
CONFIG_NF_CONNTRACK=y
CONFIG_NF_CT_ACCT=y
CONFIG_NF_CONNTRACK_MARK=y
CONFIG_NF_CONNTRACK_SECMARK=y
# CONFIG_NF_CONNTRACK_EVENTS is not set
# CONFIG_NF_CT_PROTO_DCCP is not set
CONFIG_NF_CT_PROTO_GRE=y
CONFIG_NF_CT_PROTO_SCTP=y
CONFIG_NF_CT_PROTO_UDPLITE=y
CONFIG_NF_CONNTRACK_AMANDA=y
CONFIG_NF_CONNTRACK_FTP=y
CONFIG_NF_CONNTRACK_H323=y
CONFIG_NF_CONNTRACK_IRC=y
# CONFIG_NF_CONNTRACK_NETBIOS_NS is not set
CONFIG_NF_CONNTRACK_PPTP=y
# CONFIG_NF_CONNTRACK_SANE is not set
CONFIG_NF_CONNTRACK_SIP=y
# CONFIG_NF_CONNTRACK_TFTP is not set
CONFIG_NF_CT_NETLINK=y
# CONFIG_NETFILTER_TPROXY is not set
CONFIG_NETFILTER_XTABLES=y
CONFIG_NETFILTER_XT_TARGET_CLASSIFY=y
CONFIG_NETFILTER_XT_TARGET_CONNMARK=y
CONFIG_NETFILTER_XT_TARGET_CONNSECMARK=y
CONFIG_NETFILTER_XT_TARGET_DSCP=y
CONFIG_NETFILTER_XT_TARGET_HL=y
CONFIG_NETFILTER_XT_TARGET_LED=y
CONFIG_NETFILTER_XT_TARGET_MARK=y
CONFIG_NETFILTER_XT_TARGET_NFLOG=y
CONFIG_NETFILTER_XT_TARGET_NFQUEUE=y
CONFIG_NETFILTER_XT_TARGET_RATEEST=y
CONFIG_NETFILTER_XT_TARGET_SECMARK=y
# CONFIG_NETFILTER_XT_TARGET_TCPMSS is not set
CONFIG_NETFILTER_XT_TARGET_TCPOPTSTRIP=y
# CONFIG_NETFILTER_XT_MATCH_CLUSTER is not set
CONFIG_NETFILTER_XT_MATCH_COMMENT=y
CONFIG_NETFILTER_XT_MATCH_CONNBYTES=y
CONFIG_NETFILTER_XT_MATCH_CONNLIMIT=y
CONFIG_NETFILTER_XT_MATCH_CONNMARK=y
CONFIG_NETFILTER_XT_MATCH_CONNTRACK=y
# CONFIG_NETFILTER_XT_MATCH_DCCP is not set
# CONFIG_NETFILTER_XT_MATCH_DSCP is not set
# CONFIG_NETFILTER_XT_MATCH_ESP is not set
# CONFIG_NETFILTER_XT_MATCH_HASHLIMIT is not set
# CONFIG_NETFILTER_XT_MATCH_HELPER is not set
CONFIG_NETFILTER_XT_MATCH_HL=y
CONFIG_NETFILTER_XT_MATCH_IPRANGE=y
CONFIG_NETFILTER_XT_MATCH_LENGTH=y
CONFIG_NETFILTER_XT_MATCH_LIMIT=y
CONFIG_NETFILTER_XT_MATCH_MAC=y
CONFIG_NETFILTER_XT_MATCH_MARK=y
CONFIG_NETFILTER_XT_MATCH_MULTIPORT=y
CONFIG_NETFILTER_XT_MATCH_OWNER=y
CONFIG_NETFILTER_XT_MATCH_POLICY=y
CONFIG_NETFILTER_XT_MATCH_PKTTYPE=y
# CONFIG_NETFILTER_XT_MATCH_QUOTA is not set
CONFIG_NETFILTER_XT_MATCH_RATEEST=y
# CONFIG_NETFILTER_XT_MATCH_REALM is not set
CONFIG_NETFILTER_XT_MATCH_RECENT=y
# CONFIG_NETFILTER_XT_MATCH_RECENT_PROC_COMPAT is not set
CONFIG_NETFILTER_XT_MATCH_SCTP=y
CONFIG_NETFILTER_XT_MATCH_STATE=y
CONFIG_NETFILTER_XT_MATCH_STATISTIC=y
CONFIG_NETFILTER_XT_MATCH_STRING=y
CONFIG_NETFILTER_XT_MATCH_TCPMSS=y
CONFIG_NETFILTER_XT_MATCH_TIME=y
CONFIG_NETFILTER_XT_MATCH_U32=y
# CONFIG_NETFILTER_XT_MATCH_OSF is not set
# CONFIG_IP_VS is not set
#
# IP: Netfilter Configuration
#
CONFIG_NF_DEFRAG_IPV4=y
CONFIG_NF_CONNTRACK_IPV4=y
CONFIG_NF_CONNTRACK_PROC_COMPAT=y
# CONFIG_IP_NF_QUEUE is not set
CONFIG_IP_NF_IPTABLES=y
# CONFIG_IP_NF_MATCH_ADDRTYPE is not set
# CONFIG_IP_NF_MATCH_AH is not set
# CONFIG_IP_NF_MATCH_ECN is not set
# CONFIG_IP_NF_MATCH_TTL is not set
CONFIG_IP_NF_FILTER=y
CONFIG_IP_NF_TARGET_REJECT=y
CONFIG_IP_NF_TARGET_LOG=y
CONFIG_IP_NF_TARGET_ULOG=y
CONFIG_NF_NAT=y
CONFIG_NF_NAT_NEEDED=y
# CONFIG_IP_NF_TARGET_MASQUERADE is not set
CONFIG_IP_NF_TARGET_NETMAP=y
CONFIG_IP_NF_TARGET_REDIRECT=y
# CONFIG_NF_NAT_SNMP_BASIC is not set
CONFIG_NF_NAT_PROTO_GRE=y
CONFIG_NF_NAT_PROTO_UDPLITE=y
CONFIG_NF_NAT_PROTO_SCTP=y
CONFIG_NF_NAT_FTP=y
CONFIG_NF_NAT_IRC=y
# CONFIG_NF_NAT_TFTP is not set
CONFIG_NF_NAT_AMANDA=y
CONFIG_NF_NAT_PPTP=y
CONFIG_NF_NAT_H323=y
CONFIG_NF_NAT_SIP=y
CONFIG_IP_NF_MANGLE=y
# CONFIG_IP_NF_TARGET_CLUSTERIP is not set
# CONFIG_IP_NF_TARGET_ECN is not set
# CONFIG_IP_NF_TARGET_TTL is not set
# CONFIG_IP_NF_RAW is not set
# CONFIG_IP_NF_SECURITY is not set
# CONFIG_IP_NF_ARPTABLES is not set
#
# IPv6: Netfilter Configuration
#
CONFIG_NF_CONNTRACK_IPV6=y
# CONFIG_IP6_NF_QUEUE is not set
CONFIG_IP6_NF_IPTABLES=y
# CONFIG_IP6_NF_MATCH_AH is not set
# CONFIG_IP6_NF_MATCH_EUI64 is not set
# CONFIG_IP6_NF_MATCH_FRAG is not set
# CONFIG_IP6_NF_MATCH_OPTS is not set
# CONFIG_IP6_NF_MATCH_HL is not set
CONFIG_IP6_NF_MATCH_IPV6HEADER=y
# CONFIG_IP6_NF_MATCH_MH is not set
# CONFIG_IP6_NF_MATCH_RT is not set
# CONFIG_IP6_NF_TARGET_HL is not set
CONFIG_IP6_NF_TARGET_LOG=y
CONFIG_IP6_NF_FILTER=y
CONFIG_IP6_NF_TARGET_REJECT=y
CONFIG_IP6_NF_MANGLE=y
# CONFIG_IP6_NF_RAW is not set
# CONFIG_IP6_NF_SECURITY is not set
# CONFIG_IP_DCCP is not set
CONFIG_IP_SCTP=y
# CONFIG_SCTP_DBG_MSG is not set
# CONFIG_SCTP_DBG_OBJCNT is not set
# CONFIG_SCTP_HMAC_NONE is not set
# CONFIG_SCTP_HMAC_SHA1 is not set
CONFIG_SCTP_HMAC_MD5=y
# CONFIG_RDS is not set
# CONFIG_TIPC is not set
# CONFIG_ATM is not set
# CONFIG_BRIDGE is not set
# CONFIG_NET_DSA is not set
# CONFIG_VLAN_8021Q is not set
# CONFIG_DECNET is not set
# CONFIG_LLC2 is not set
# CONFIG_IPX is not set
# CONFIG_ATALK is not set
# CONFIG_X25 is not set
# CONFIG_LAPB is not set
# CONFIG_ECONET is not set
# CONFIG_WAN_ROUTER is not set
# CONFIG_PHONET is not set
# CONFIG_IEEE802154 is not set
CONFIG_NET_SCHED=y
#
# Queueing/Scheduling
#
CONFIG_NET_SCH_CBQ=y
CONFIG_NET_SCH_HTB=y
CONFIG_NET_SCH_HFSC=m
CONFIG_NET_SCH_PRIO=m
# CONFIG_NET_SCH_MULTIQ is not set
CONFIG_NET_SCH_RED=y
CONFIG_NET_SCH_SFQ=y
# CONFIG_NET_SCH_TEQL is not set
CONFIG_NET_SCH_TBF=y
# CONFIG_NET_SCH_GRED is not set
CONFIG_NET_SCH_DSMARK=y
CONFIG_NET_SCH_NETEM=y
# CONFIG_NET_SCH_DRR is not set
CONFIG_NET_SCH_INGRESS=y
#
# Classification
#
CONFIG_NET_CLS=y
CONFIG_NET_CLS_BASIC=y
CONFIG_NET_CLS_TCINDEX=y
CONFIG_NET_CLS_ROUTE4=y
CONFIG_NET_CLS_ROUTE=y
CONFIG_NET_CLS_FW=y
# CONFIG_NET_CLS_U32 is not set
# CONFIG_NET_CLS_RSVP is not set
# CONFIG_NET_CLS_RSVP6 is not set
# CONFIG_NET_CLS_FLOW is not set
CONFIG_NET_EMATCH=y
CONFIG_NET_EMATCH_STACK=32
CONFIG_NET_EMATCH_CMP=y
CONFIG_NET_EMATCH_NBYTE=y
# CONFIG_NET_EMATCH_U32 is not set
# CONFIG_NET_EMATCH_META is not set
# CONFIG_NET_EMATCH_TEXT is not set
CONFIG_NET_CLS_ACT=y
CONFIG_NET_ACT_POLICE=y
CONFIG_NET_ACT_GACT=y
CONFIG_GACT_PROB=y
CONFIG_NET_ACT_MIRRED=y
CONFIG_NET_ACT_IPT=y
# CONFIG_NET_ACT_NAT is not set
# CONFIG_NET_ACT_PEDIT is not set
# CONFIG_NET_ACT_SIMP is not set
# CONFIG_NET_ACT_SKBEDIT is not set
CONFIG_NET_CLS_IND=y
CONFIG_NET_SCH_FIFO=y
# CONFIG_DCB is not set
#
# Network testing
#
# CONFIG_NET_PKTGEN is not set
# CONFIG_NET_TCPPROBE is not set
# CONFIG_NET_DROP_MONITOR is not set
# CONFIG_HAMRADIO is not set
# CONFIG_CAN is not set
# CONFIG_IRDA is not set
# CONFIG_BT is not set
# CONFIG_AF_RXRPC is not set
CONFIG_FIB_RULES=y
CONFIG_WIRELESS=y
CONFIG_CFG80211=m
# CONFIG_NL80211_TESTMODE is not set
# CONFIG_CFG80211_DEVELOPER_WARNINGS is not set
# CONFIG_CFG80211_REG_DEBUG is not set
CONFIG_CFG80211_DEFAULT_PS=y
CONFIG_CFG80211_DEFAULT_PS_VALUE=1
# CONFIG_CFG80211_DEBUGFS is not set
# CONFIG_WIRELESS_OLD_REGULATORY is not set
CONFIG_WIRELESS_EXT=y
CONFIG_WIRELESS_EXT_SYSFS=y
# CONFIG_LIB80211 is not set
CONFIG_MAC80211=m
CONFIG_MAC80211_RC_MINSTREL=y
# CONFIG_MAC80211_RC_DEFAULT_PID is not set
CONFIG_MAC80211_RC_DEFAULT_MINSTREL=y
CONFIG_MAC80211_RC_DEFAULT="minstrel"
# CONFIG_MAC80211_MESH is not set
CONFIG_MAC80211_LEDS=y
# CONFIG_MAC80211_DEBUGFS is not set
# CONFIG_MAC80211_DEBUG_MENU is not set
# CONFIG_WIMAX is not set
CONFIG_RFKILL=m
CONFIG_RFKILL_LEDS=y
CONFIG_RFKILL_INPUT=y
# CONFIG_NET_9P is not set
#
# Device Drivers
#
#
# Generic Driver Options
#
CONFIG_UEVENT_HELPER_PATH="/sbin/hotplug"
# CONFIG_DEVTMPFS is not set
CONFIG_STANDALONE=y
CONFIG_PREVENT_FIRMWARE_BUILD=y
CONFIG_FW_LOADER=y
CONFIG_FIRMWARE_IN_KERNEL=y
CONFIG_EXTRA_FIRMWARE=""
# CONFIG_DEBUG_DRIVER is not set
# CONFIG_DEBUG_DEVRES is not set
CONFIG_SYS_HYPERVISOR=y
CONFIG_CONNECTOR=y
CONFIG_PROC_EVENTS=y
# CONFIG_MTD is not set
# CONFIG_PARPORT is not set
CONFIG_PNP=y
CONFIG_PNP_DEBUG_MESSAGES=y
#
# Protocols
#
CONFIG_PNPACPI=y
CONFIG_BLK_DEV=y
# CONFIG_BLK_DEV_FD is not set
# CONFIG_BLK_CPQ_DA is not set
# CONFIG_BLK_CPQ_CISS_DA is not set
# CONFIG_BLK_DEV_DAC960 is not set
# CONFIG_BLK_DEV_UMEM is not set
# CONFIG_BLK_DEV_COW_COMMON is not set
CONFIG_BLK_DEV_LOOP=y
CONFIG_BLK_DEV_CRYPTOLOOP=y
CONFIG_BLK_DEV_NBD=y
# CONFIG_BLK_DEV_SX8 is not set
CONFIG_BLK_DEV_RAM=y
CONFIG_BLK_DEV_RAM_COUNT=16
CONFIG_BLK_DEV_RAM_SIZE=16384
# CONFIG_BLK_DEV_XIP is not set
# CONFIG_CDROM_PKTCDVD is not set
CONFIG_ATA_OVER_ETH=y
CONFIG_XEN_BLKDEV_FRONTEND=y
# CONFIG_BLK_DEV_HD is not set
CONFIG_MISC_DEVICES=y
# CONFIG_IBM_ASM is not set
# CONFIG_PHANTOM is not set
# CONFIG_SGI_IOC4 is not set
# CONFIG_TIFM_CORE is not set
# CONFIG_ICS932S401 is not set
# CONFIG_ENCLOSURE_SERVICES is not set
# CONFIG_HP_ILO is not set
# CONFIG_DELL_LAPTOP is not set
# CONFIG_ISL29003 is not set
# CONFIG_C2PORT is not set
#
# EEPROM support
#
# CONFIG_EEPROM_AT24 is not set
# CONFIG_EEPROM_LEGACY is not set
# CONFIG_EEPROM_MAX6875 is not set
# CONFIG_EEPROM_93CX6 is not set
# CONFIG_CB710_CORE is not set
CONFIG_HAVE_IDE=y
# CONFIG_IDE is not set
#
# SCSI device support
#
CONFIG_RAID_ATTRS=y
CONFIG_SCSI=y
CONFIG_SCSI_DMA=y
CONFIG_SCSI_TGT=y
CONFIG_SCSI_NETLINK=y
CONFIG_SCSI_PROC_FS=y
#
# SCSI support type (disk, tape, CD-ROM)
#
CONFIG_BLK_DEV_SD=y
# CONFIG_CHR_DEV_ST is not set
# CONFIG_CHR_DEV_OSST is not set
# CONFIG_BLK_DEV_SR is not set
CONFIG_CHR_DEV_SG=y
# CONFIG_CHR_DEV_SCH is not set
CONFIG_SCSI_MULTI_LUN=y
# CONFIG_SCSI_CONSTANTS is not set
# CONFIG_SCSI_LOGGING is not set
# CONFIG_SCSI_SCAN_ASYNC is not set
CONFIG_SCSI_WAIT_SCAN=m
#
# SCSI Transports
#
CONFIG_SCSI_SPI_ATTRS=m
CONFIG_SCSI_FC_ATTRS=m
# CONFIG_SCSI_FC_TGT_ATTRS is not set
CONFIG_SCSI_ISCSI_ATTRS=y
CONFIG_SCSI_SAS_ATTRS=m
CONFIG_SCSI_SAS_LIBSAS=m
# CONFIG_SCSI_SAS_HOST_SMP is not set
# CONFIG_SCSI_SAS_LIBSAS_DEBUG is not set
CONFIG_SCSI_SRP_ATTRS=m
# CONFIG_SCSI_SRP_TGT_ATTRS is not set
CONFIG_SCSI_LOWLEVEL=y
CONFIG_ISCSI_TCP=y
# CONFIG_SCSI_BNX2_ISCSI is not set
# CONFIG_BE2ISCSI is not set
# CONFIG_BLK_DEV_3W_XXXX_RAID is not set
# CONFIG_SCSI_3W_9XXX is not set
# CONFIG_SCSI_ACARD is not set
# CONFIG_SCSI_AACRAID is not set
# CONFIG_SCSI_AIC7XXX is not set
# CONFIG_SCSI_AIC7XXX_OLD is not set
# CONFIG_SCSI_AIC79XX is not set
# CONFIG_SCSI_AIC94XX is not set
# CONFIG_SCSI_MVSAS is not set
# CONFIG_SCSI_DPT_I2O is not set
# CONFIG_SCSI_ADVANSYS is not set
# CONFIG_SCSI_ARCMSR is not set
# CONFIG_MEGARAID_NEWGEN is not set
# CONFIG_MEGARAID_LEGACY is not set
# CONFIG_MEGARAID_SAS is not set
# CONFIG_SCSI_MPT2SAS is not set
# CONFIG_SCSI_HPTIOP is not set
# CONFIG_SCSI_BUSLOGIC is not set
# CONFIG_LIBFC is not set
# CONFIG_LIBFCOE is not set
# CONFIG_FCOE is not set
# CONFIG_FCOE_FNIC is not set
# CONFIG_SCSI_DMX3191D is not set
# CONFIG_SCSI_EATA is not set
# CONFIG_SCSI_FUTURE_DOMAIN is not set
# CONFIG_SCSI_GDTH is not set
# CONFIG_SCSI_IPS is not set
# CONFIG_SCSI_INITIO is not set
# CONFIG_SCSI_INIA100 is not set
# CONFIG_SCSI_STEX is not set
# CONFIG_SCSI_SYM53C8XX_2 is not set
# CONFIG_SCSI_QLOGIC_1280 is not set
# CONFIG_SCSI_QLA_FC is not set
# CONFIG_SCSI_QLA_ISCSI is not set
# CONFIG_SCSI_LPFC is not set
# CONFIG_SCSI_DC395x is not set
# CONFIG_SCSI_DC390T is not set
# CONFIG_SCSI_DEBUG is not set
# CONFIG_SCSI_PMCRAID is not set
# CONFIG_SCSI_SRP is not set
# CONFIG_SCSI_BFA_FC is not set
# CONFIG_SCSI_LOWLEVEL_PCMCIA is not set
# CONFIG_SCSI_DH is not set
# CONFIG_SCSI_OSD_INITIATOR is not set
# CONFIG_ATA is not set
CONFIG_MD=y
CONFIG_BLK_DEV_MD=y
CONFIG_MD_AUTODETECT=y
CONFIG_MD_LINEAR=y
CONFIG_MD_RAID0=y
CONFIG_MD_RAID1=y
CONFIG_MD_RAID10=m
CONFIG_MD_RAID456=y
# CONFIG_MULTICORE_RAID456 is not set
CONFIG_MD_RAID6_PQ=y
# CONFIG_ASYNC_RAID6_TEST is not set
CONFIG_MD_MULTIPATH=y
# CONFIG_MD_FAULTY is not set
CONFIG_BLK_DEV_DM=y
# CONFIG_DM_DEBUG is not set
CONFIG_DM_CRYPT=y
CONFIG_DM_SNAPSHOT=y
CONFIG_DM_MIRROR=y
# CONFIG_DM_LOG_USERSPACE is not set
CONFIG_DM_ZERO=y
CONFIG_DM_MULTIPATH=y
# CONFIG_DM_MULTIPATH_QL is not set
# CONFIG_DM_MULTIPATH_ST is not set
# CONFIG_DM_DELAY is not set
CONFIG_DM_UEVENT=y
# CONFIG_FUSION is not set
#
# IEEE 1394 (FireWire) support
#
#
# You can enable one or both FireWire driver stacks.
#
#
# See the help texts for more information.
#
CONFIG_FIREWIRE=m
CONFIG_FIREWIRE_OHCI=m
CONFIG_FIREWIRE_OHCI_DEBUG=y
CONFIG_FIREWIRE_SBP2=m
# CONFIG_FIREWIRE_NET is not set
# CONFIG_IEEE1394 is not set
# CONFIG_I2O is not set
# CONFIG_MACINTOSH_DRIVERS is not set
CONFIG_NETDEVICES=y
CONFIG_IFB=m
CONFIG_DUMMY=y
# CONFIG_BONDING is not set
# CONFIG_MACVLAN is not set
# CONFIG_EQUALIZER is not set
CONFIG_TUN=y
CONFIG_VETH=y
# CONFIG_NET_SB1000 is not set
# CONFIG_ARCNET is not set
# CONFIG_NET_ETHERNET is not set
# CONFIG_NETDEV_1000 is not set
# CONFIG_NETDEV_10000 is not set
# CONFIG_TR is not set
# CONFIG_WLAN is not set
#
# Enable WiMAX (Networking options) to see the WiMAX drivers
#
# CONFIG_NET_PCMCIA is not set
# CONFIG_WAN is not set
CONFIG_XEN_NETDEV_FRONTEND=y
# CONFIG_FDDI is not set
# CONFIG_HIPPI is not set
CONFIG_PPP=y
CONFIG_PPP_MULTILINK=y
CONFIG_PPP_FILTER=y
CONFIG_PPP_ASYNC=y
CONFIG_PPP_SYNC_TTY=y
CONFIG_PPP_DEFLATE=y
CONFIG_PPP_BSDCOMP=y
CONFIG_PPP_MPPE=y
# CONFIG_PPPOE is not set
# CONFIG_PPPOL2TP is not set
# CONFIG_SLIP is not set
CONFIG_SLHC=y
# CONFIG_NET_FC is not set
# CONFIG_NETCONSOLE is not set
# CONFIG_NETPOLL is not set
# CONFIG_NET_POLL_CONTROLLER is not set
# CONFIG_VMXNET3 is not set
# CONFIG_ISDN is not set
# CONFIG_PHONE is not set
#
# Input device support
#
CONFIG_INPUT=y
# CONFIG_INPUT_FF_MEMLESS is not set
CONFIG_INPUT_POLLDEV=m
#
# Userland interfaces
#
CONFIG_INPUT_MOUSEDEV=y
# CONFIG_INPUT_MOUSEDEV_PSAUX is not set
CONFIG_INPUT_MOUSEDEV_SCREEN_X=1024
CONFIG_INPUT_MOUSEDEV_SCREEN_Y=768
CONFIG_INPUT_JOYDEV=m
CONFIG_INPUT_EVDEV=y
# CONFIG_INPUT_EVBUG is not set
CONFIG_XEN_KBDDEV_FRONTEND=y
#
# Input Device Drivers
#
CONFIG_INPUT_KEYBOARD=y
# CONFIG_KEYBOARD_ADP5588 is not set
CONFIG_KEYBOARD_ATKBD=y
# CONFIG_QT2160 is not set
# CONFIG_KEYBOARD_LKKBD is not set
# CONFIG_KEYBOARD_LM8323 is not set
# CONFIG_KEYBOARD_MAX7359 is not set
# CONFIG_KEYBOARD_NEWTON is not set
# CONFIG_KEYBOARD_OPENCORES is not set
# CONFIG_KEYBOARD_STOWAWAY is not set
# CONFIG_KEYBOARD_SUNKBD is not set
# CONFIG_KEYBOARD_XTKBD is not set
CONFIG_INPUT_MOUSE=y
CONFIG_MOUSE_PS2=y
CONFIG_MOUSE_PS2_ALPS=y
CONFIG_MOUSE_PS2_LOGIPS2PP=y
CONFIG_MOUSE_PS2_SYNAPTICS=y
CONFIG_MOUSE_PS2_LIFEBOOK=y
CONFIG_MOUSE_PS2_TRACKPOINT=y
# CONFIG_MOUSE_PS2_ELANTECH is not set
# CONFIG_MOUSE_PS2_SENTELIC is not set
# CONFIG_MOUSE_PS2_TOUCHKIT is not set
CONFIG_MOUSE_SERIAL=m
CONFIG_MOUSE_VSXXXAA=m
# CONFIG_MOUSE_SYNAPTICS_I2C is not set
CONFIG_INPUT_JOYSTICK=y
# CONFIG_JOYSTICK_ANALOG is not set
# CONFIG_JOYSTICK_A3D is not set
# CONFIG_JOYSTICK_ADI is not set
# CONFIG_JOYSTICK_COBRA is not set
# CONFIG_JOYSTICK_GF2K is not set
# CONFIG_JOYSTICK_GRIP is not set
# CONFIG_JOYSTICK_GRIP_MP is not set
# CONFIG_JOYSTICK_GUILLEMOT is not set
# CONFIG_JOYSTICK_INTERACT is not set
# CONFIG_JOYSTICK_SIDEWINDER is not set
# CONFIG_JOYSTICK_TMDC is not set
# CONFIG_JOYSTICK_IFORCE is not set
# CONFIG_JOYSTICK_WARRIOR is not set
# CONFIG_JOYSTICK_MAGELLAN is not set
# CONFIG_JOYSTICK_SPACEORB is not set
# CONFIG_JOYSTICK_SPACEBALL is not set
# CONFIG_JOYSTICK_STINGER is not set
CONFIG_JOYSTICK_TWIDJOY=m
# CONFIG_JOYSTICK_ZHENHUA is not set
CONFIG_JOYSTICK_JOYDUMP=m
# CONFIG_INPUT_TABLET is not set
CONFIG_INPUT_TOUCHSCREEN=y
# CONFIG_TOUCHSCREEN_AD7879_I2C is not set
# CONFIG_TOUCHSCREEN_AD7879 is not set
# CONFIG_TOUCHSCREEN_EETI is not set
# CONFIG_TOUCHSCREEN_FUJITSU is not set
CONFIG_TOUCHSCREEN_GUNZE=m
CONFIG_TOUCHSCREEN_ELO=m
# CONFIG_TOUCHSCREEN_WACOM_W8001 is not set
# CONFIG_TOUCHSCREEN_MCS5000 is not set
CONFIG_TOUCHSCREEN_MTOUCH=m
# CONFIG_TOUCHSCREEN_INEXIO is not set
CONFIG_TOUCHSCREEN_MK712=m
# CONFIG_TOUCHSCREEN_PENMOUNT is not set
# CONFIG_TOUCHSCREEN_TOUCHRIGHT is not set
# CONFIG_TOUCHSCREEN_TOUCHWIN is not set
# CONFIG_TOUCHSCREEN_TOUCHIT213 is not set
# CONFIG_TOUCHSCREEN_TSC2007 is not set
CONFIG_INPUT_MISC=y
CONFIG_INPUT_PCSPKR=m
# CONFIG_INPUT_APANEL is not set
# CONFIG_INPUT_ATLAS_BTNS is not set
CONFIG_INPUT_UINPUT=m
# CONFIG_INPUT_WINBOND_CIR is not set
#
# Hardware I/O ports
#
CONFIG_SERIO=y
CONFIG_SERIO_I8042=y
CONFIG_SERIO_SERPORT=y
# CONFIG_SERIO_CT82C710 is not set
# CONFIG_SERIO_PCIPS2 is not set
CONFIG_SERIO_LIBPS2=y
CONFIG_SERIO_RAW=m
CONFIG_GAMEPORT=m
CONFIG_GAMEPORT_NS558=m
CONFIG_GAMEPORT_L4=m
CONFIG_GAMEPORT_EMU10K1=m
CONFIG_GAMEPORT_FM801=m
#
# Character devices
#
CONFIG_VT=y
CONFIG_CONSOLE_TRANSLATIONS=y
CONFIG_VT_CONSOLE=y
CONFIG_HW_CONSOLE=y
CONFIG_VT_HW_CONSOLE_BINDING=y
CONFIG_DEVKMEM=y
CONFIG_SERIAL_NONSTANDARD=y
# CONFIG_COMPUTONE is not set
# CONFIG_ROCKETPORT is not set
CONFIG_CYCLADES=m
# CONFIG_CYZ_INTR is not set
# CONFIG_DIGIEPCA is not set
# CONFIG_MOXA_INTELLIO is not set
# CONFIG_MOXA_SMARTIO is not set
# CONFIG_ISI is not set
CONFIG_SYNCLINK=m
CONFIG_SYNCLINKMP=m
CONFIG_SYNCLINK_GT=m
CONFIG_N_HDLC=m
# CONFIG_RISCOM8 is not set
# CONFIG_SPECIALIX is not set
# CONFIG_STALDRV is not set
# CONFIG_NOZOMI is not set
#
# Serial drivers
#
CONFIG_SERIAL_8250=m
CONFIG_FIX_EARLYCON_MEM=y
CONFIG_SERIAL_8250_PCI=m
CONFIG_SERIAL_8250_PNP=m
CONFIG_SERIAL_8250_CS=m
CONFIG_SERIAL_8250_NR_UARTS=32
CONFIG_SERIAL_8250_RUNTIME_UARTS=4
CONFIG_SERIAL_8250_EXTENDED=y
CONFIG_SERIAL_8250_MANY_PORTS=y
CONFIG_SERIAL_8250_SHARE_IRQ=y
CONFIG_SERIAL_8250_DETECT_IRQ=y
CONFIG_SERIAL_8250_RSA=y
#
# Non-8250 serial port support
#
CONFIG_SERIAL_CORE=m
CONFIG_SERIAL_JSM=m
CONFIG_UNIX98_PTYS=y
# CONFIG_DEVPTS_MULTIPLE_INSTANCES is not set
# CONFIG_LEGACY_PTYS is not set
CONFIG_HVC_DRIVER=y
CONFIG_HVC_IRQ=y
CONFIG_HVC_XEN=y
CONFIG_IPMI_HANDLER=m
CONFIG_IPMI_PANIC_EVENT=y
CONFIG_IPMI_PANIC_STRING=y
CONFIG_IPMI_DEVICE_INTERFACE=m
CONFIG_IPMI_SI=m
CONFIG_IPMI_WATCHDOG=m
CONFIG_IPMI_POWEROFF=m
CONFIG_HW_RANDOM=y
# CONFIG_HW_RANDOM_TIMERIOMEM is not set
CONFIG_HW_RANDOM_INTEL=m
CONFIG_HW_RANDOM_AMD=m
# CONFIG_HW_RANDOM_VIA is not set
CONFIG_NVRAM=y
# CONFIG_R3964 is not set
# CONFIG_APPLICOM is not set
#
# PCMCIA character devices
#
# CONFIG_SYNCLINK_CS is not set
CONFIG_CARDMAN_4000=m
CONFIG_CARDMAN_4040=m
# CONFIG_IPWIRELESS is not set
# CONFIG_MWAVE is not set
CONFIG_PC8736x_GPIO=m
CONFIG_NSC_GPIO=m
CONFIG_RAW_DRIVER=y
CONFIG_MAX_RAW_DEVS=8192
CONFIG_HPET=y
# CONFIG_HPET_MMAP is not set
CONFIG_HANGCHECK_TIMER=m
# CONFIG_TCG_TPM is not set
CONFIG_TELCLOCK=m
CONFIG_DEVPORT=y
CONFIG_I2C=m
CONFIG_I2C_BOARDINFO=y
CONFIG_I2C_COMPAT=y
CONFIG_I2C_CHARDEV=m
CONFIG_I2C_HELPER_AUTO=y
CONFIG_I2C_ALGOBIT=m
#
# I2C Hardware Bus support
#
#
# PC SMBus host controller drivers
#
# CONFIG_I2C_ALI1535 is not set
# CONFIG_I2C_ALI1563 is not set
# CONFIG_I2C_ALI15X3 is not set
CONFIG_I2C_AMD756=m
CONFIG_I2C_AMD756_S4882=m
CONFIG_I2C_AMD8111=m
CONFIG_I2C_I801=m
# CONFIG_I2C_ISCH is not set
CONFIG_I2C_PIIX4=m
CONFIG_I2C_NFORCE2=m
# CONFIG_I2C_NFORCE2_S4985 is not set
# CONFIG_I2C_SIS5595 is not set
# CONFIG_I2C_SIS630 is not set
CONFIG_I2C_SIS96X=m
CONFIG_I2C_VIA=m
CONFIG_I2C_VIAPRO=m
#
# ACPI drivers
#
# CONFIG_I2C_SCMI is not set
#
# I2C system bus drivers (mostly embedded / system-on-chip)
#
# CONFIG_I2C_OCORES is not set
# CONFIG_I2C_SIMTEC is not set
#
# External I2C/SMBus adapter drivers
#
CONFIG_I2C_PARPORT_LIGHT=m
# CONFIG_I2C_TAOS_EVM is not set
#
# Graphics adapter I2C/DDC channel drivers
#
CONFIG_I2C_VOODOO3=m
#
# Other I2C/SMBus bus drivers
#
# CONFIG_I2C_PCA_PLATFORM is not set
CONFIG_I2C_STUB=m
#
# Miscellaneous I2C Chip support
#
# CONFIG_DS1682 is not set
# CONFIG_SENSORS_TSL2550 is not set
# CONFIG_I2C_DEBUG_CORE is not set
# CONFIG_I2C_DEBUG_ALGO is not set
# CONFIG_I2C_DEBUG_BUS is not set
# CONFIG_I2C_DEBUG_CHIP is not set
# CONFIG_SPI is not set
#
# PPS support
#
# CONFIG_PPS is not set
CONFIG_ARCH_WANT_OPTIONAL_GPIOLIB=y
# CONFIG_GPIOLIB is not set
# CONFIG_W1 is not set
CONFIG_POWER_SUPPLY=y
# CONFIG_POWER_SUPPLY_DEBUG is not set
# CONFIG_PDA_POWER is not set
# CONFIG_BATTERY_DS2760 is not set
# CONFIG_BATTERY_DS2782 is not set
# CONFIG_BATTERY_BQ27x00 is not set
# CONFIG_BATTERY_MAX17040 is not set
# CONFIG_HWMON is not set
CONFIG_THERMAL=y
# CONFIG_WATCHDOG is not set
CONFIG_SSB_POSSIBLE=y
#
# Sonics Silicon Backplane
#
# CONFIG_SSB is not set
#
# Multifunction device drivers
#
# CONFIG_MFD_CORE is not set
# CONFIG_MFD_SM501 is not set
# CONFIG_HTC_PASIC3 is not set
# CONFIG_MFD_TMIO is not set
# CONFIG_MFD_WM8400 is not set
# CONFIG_MFD_WM831X is not set
# CONFIG_MFD_WM8350_I2C is not set
# CONFIG_MFD_PCF50633 is not set
# CONFIG_AB3100_CORE is not set
# CONFIG_REGULATOR is not set
# CONFIG_MEDIA_SUPPORT is not set
#
# Graphics support
#
CONFIG_AGP=y
CONFIG_AGP_AMD64=y
CONFIG_AGP_INTEL=y
CONFIG_AGP_SIS=y
CONFIG_AGP_VIA=y
CONFIG_VGA_ARB=y
CONFIG_DRM=m
CONFIG_DRM_KMS_HELPER=m
CONFIG_DRM_TTM=m
# CONFIG_DRM_TDFX is not set
CONFIG_DRM_R128=m
CONFIG_DRM_RADEON=m
CONFIG_DRM_I810=m
CONFIG_DRM_I830=m
CONFIG_DRM_I915=m
# CONFIG_DRM_I915_KMS is not set
CONFIG_DRM_MGA=m
# CONFIG_DRM_SIS is not set
CONFIG_DRM_VIA=m
CONFIG_DRM_SAVAGE=m
CONFIG_VGASTATE=m
CONFIG_VIDEO_OUTPUT_CONTROL=m
CONFIG_FB=y
# CONFIG_FIRMWARE_EDID is not set
CONFIG_FB_DDC=m
CONFIG_FB_BOOT_VESA_SUPPORT=y
CONFIG_FB_CFB_FILLRECT=y
CONFIG_FB_CFB_COPYAREA=y
CONFIG_FB_CFB_IMAGEBLIT=y
# CONFIG_FB_CFB_REV_PIXELS_IN_BYTE is not set
CONFIG_FB_SYS_FILLRECT=y
CONFIG_FB_SYS_COPYAREA=y
CONFIG_FB_SYS_IMAGEBLIT=y
# CONFIG_FB_FOREIGN_ENDIAN is not set
CONFIG_FB_SYS_FOPS=y
CONFIG_FB_DEFERRED_IO=y
# CONFIG_FB_SVGALIB is not set
# CONFIG_FB_MACMODES is not set
CONFIG_FB_BACKLIGHT=y
CONFIG_FB_MODE_HELPERS=y
CONFIG_FB_TILEBLITTING=y
#
# Frame buffer hardware drivers
#
CONFIG_FB_CIRRUS=m
# CONFIG_FB_PM2 is not set
# CONFIG_FB_CYBER2000 is not set
# CONFIG_FB_ARC is not set
# CONFIG_FB_ASILIANT is not set
# CONFIG_FB_IMSTT is not set
CONFIG_FB_VGA16=m
# CONFIG_FB_UVESA is not set
CONFIG_FB_VESA=y
# CONFIG_FB_N411 is not set
# CONFIG_FB_HGA is not set
# CONFIG_FB_S1D13XXX is not set
CONFIG_FB_NVIDIA=m
CONFIG_FB_NVIDIA_I2C=y
# CONFIG_FB_NVIDIA_DEBUG is not set
CONFIG_FB_NVIDIA_BACKLIGHT=y
CONFIG_FB_RIVA=m
# CONFIG_FB_RIVA_I2C is not set
# CONFIG_FB_RIVA_DEBUG is not set
CONFIG_FB_RIVA_BACKLIGHT=y
# CONFIG_FB_LE80578 is not set
# CONFIG_FB_MATROX is not set
# CONFIG_FB_RADEON is not set
# CONFIG_FB_ATY128 is not set
# CONFIG_FB_ATY is not set
# CONFIG_FB_S3 is not set
CONFIG_FB_SAVAGE=m
CONFIG_FB_SAVAGE_I2C=y
CONFIG_FB_SAVAGE_ACCEL=y
# CONFIG_FB_SIS is not set
# CONFIG_FB_VIA is not set
# CONFIG_FB_NEOMAGIC is not set
CONFIG_FB_KYRO=m
# CONFIG_FB_3DFX is not set
# CONFIG_FB_VOODOO1 is not set
# CONFIG_FB_VT8623 is not set
# CONFIG_FB_TRIDENT is not set
# CONFIG_FB_ARK is not set
# CONFIG_FB_PM3 is not set
# CONFIG_FB_CARMINE is not set
# CONFIG_FB_GEODE is not set
# CONFIG_FB_VIRTUAL is not set
CONFIG_XEN_FBDEV_FRONTEND=y
# CONFIG_FB_METRONOME is not set
# CONFIG_FB_MB862XX is not set
# CONFIG_FB_BROADSHEET is not set
CONFIG_BACKLIGHT_LCD_SUPPORT=y
CONFIG_LCD_CLASS_DEVICE=m
# CONFIG_LCD_ILI9320 is not set
# CONFIG_LCD_PLATFORM is not set
CONFIG_BACKLIGHT_CLASS_DEVICE=y
CONFIG_BACKLIGHT_GENERIC=y
# CONFIG_BACKLIGHT_PROGEAR is not set
# CONFIG_BACKLIGHT_MBP_NVIDIA is not set
# CONFIG_BACKLIGHT_SAHARA is not set
#
# Display device support
#
# CONFIG_DISPLAY_SUPPORT is not set
#
# Console display driver support
#
CONFIG_VGA_CONSOLE=y
CONFIG_VGACON_SOFT_SCROLLBACK=y
CONFIG_VGACON_SOFT_SCROLLBACK_SIZE=64
CONFIG_DUMMY_CONSOLE=y
CONFIG_FRAMEBUFFER_CONSOLE=y
# CONFIG_FRAMEBUFFER_CONSOLE_DETECT_PRIMARY is not set
CONFIG_FRAMEBUFFER_CONSOLE_ROTATION=y
# CONFIG_FONTS is not set
CONFIG_FONT_8x8=y
CONFIG_FONT_8x16=y
CONFIG_LOGO=y
# CONFIG_LOGO_LINUX_MONO is not set
# CONFIG_LOGO_LINUX_VGA16 is not set
CONFIG_LOGO_LINUX_CLUT224=y
# CONFIG_SOUND is not set
# CONFIG_HID_SUPPORT is not set
# CONFIG_USB_SUPPORT is not set
# CONFIG_UWB is not set
# CONFIG_MMC is not set
# CONFIG_MEMSTICK is not set
CONFIG_NEW_LEDS=y
CONFIG_LEDS_CLASS=y
#
# LED drivers
#
# CONFIG_LEDS_ALIX2 is not set
# CONFIG_LEDS_PCA9532 is not set
# CONFIG_LEDS_LP3944 is not set
# CONFIG_LEDS_CLEVO_MAIL is not set
# CONFIG_LEDS_PCA955X is not set
# CONFIG_LEDS_BD2802 is not set
#
# LED Triggers
#
CONFIG_LEDS_TRIGGERS=y
CONFIG_LEDS_TRIGGER_TIMER=m
CONFIG_LEDS_TRIGGER_HEARTBEAT=m
# CONFIG_LEDS_TRIGGER_BACKLIGHT is not set
# CONFIG_LEDS_TRIGGER_DEFAULT_ON is not set
#
# iptables trigger is under Netfilter config (LED target)
#
# CONFIG_ACCESSIBILITY is not set
# CONFIG_INFINIBAND is not set
# CONFIG_EDAC is not set
CONFIG_RTC_LIB=y
CONFIG_RTC_CLASS=y
CONFIG_RTC_HCTOSYS=y
CONFIG_RTC_HCTOSYS_DEVICE="rtc0"
# CONFIG_RTC_DEBUG is not set
#
# RTC interfaces
#
CONFIG_RTC_INTF_SYSFS=y
CONFIG_RTC_INTF_PROC=y
CONFIG_RTC_INTF_DEV=y
# CONFIG_RTC_INTF_DEV_UIE_EMUL is not set
# CONFIG_RTC_DRV_TEST is not set
#
# I2C RTC drivers
#
CONFIG_RTC_DRV_DS1307=m
# CONFIG_RTC_DRV_DS1374 is not set
CONFIG_RTC_DRV_DS1672=m
# CONFIG_RTC_DRV_MAX6900 is not set
CONFIG_RTC_DRV_RS5C372=m
CONFIG_RTC_DRV_ISL1208=m
CONFIG_RTC_DRV_X1205=m
CONFIG_RTC_DRV_PCF8563=m
CONFIG_RTC_DRV_PCF8583=m
# CONFIG_RTC_DRV_M41T80 is not set
# CONFIG_RTC_DRV_S35390A is not set
# CONFIG_RTC_DRV_FM3130 is not set
# CONFIG_RTC_DRV_RX8581 is not set
# CONFIG_RTC_DRV_RX8025 is not set
#
# SPI RTC drivers
#
#
# Platform RTC drivers
#
CONFIG_RTC_DRV_CMOS=y
# CONFIG_RTC_DRV_DS1286 is not set
# CONFIG_RTC_DRV_DS1511 is not set
CONFIG_RTC_DRV_DS1553=m
CONFIG_RTC_DRV_DS1742=m
# CONFIG_RTC_DRV_STK17TA8 is not set
# CONFIG_RTC_DRV_M48T86 is not set
# CONFIG_RTC_DRV_M48T35 is not set
# CONFIG_RTC_DRV_M48T59 is not set
# CONFIG_RTC_DRV_BQ4802 is not set
CONFIG_RTC_DRV_V3020=m
#
# on-CPU RTC drivers
#
# CONFIG_DMADEVICES is not set
# CONFIG_AUXDISPLAY is not set
# CONFIG_UIO is not set
#
# TI VLYNQ
#
CONFIG_XEN_BALLOON=y
CONFIG_XEN_SCRUB_PAGES=y
CONFIG_XEN_DEV_EVTCHN=y
CONFIG_XENFS=y
CONFIG_XEN_COMPAT_XENFS=y
CONFIG_XEN_SYS_HYPERVISOR=y
# CONFIG_STAGING is not set
CONFIG_X86_PLATFORM_DEVICES=y
# CONFIG_ACER_WMI is not set
# CONFIG_ASUS_LAPTOP is not set
# CONFIG_FUJITSU_LAPTOP is not set
# CONFIG_MSI_LAPTOP is not set
# CONFIG_PANASONIC_LAPTOP is not set
# CONFIG_COMPAL_LAPTOP is not set
# CONFIG_SONY_LAPTOP is not set
# CONFIG_THINKPAD_ACPI is not set
# CONFIG_INTEL_MENLOW is not set
# CONFIG_EEEPC_LAPTOP is not set
# CONFIG_ACPI_WMI is not set
CONFIG_ACPI_ASUS=m
# CONFIG_TOPSTAR_LAPTOP is not set
CONFIG_ACPI_TOSHIBA=m
#
# Firmware Drivers
#
# CONFIG_EDD is not set
CONFIG_FIRMWARE_MEMMAP=y
CONFIG_DELL_RBU=m
CONFIG_DCDBAS=m
CONFIG_DMIID=y
CONFIG_ISCSI_IBFT_FIND=y
CONFIG_ISCSI_IBFT=m
#
# File systems
#
CONFIG_EXT2_FS=y
CONFIG_EXT2_FS_XATTR=y
CONFIG_EXT2_FS_POSIX_ACL=y
CONFIG_EXT2_FS_SECURITY=y
CONFIG_EXT2_FS_XIP=y
CONFIG_EXT3_FS=y
# CONFIG_EXT3_DEFAULTS_TO_ORDERED is not set
CONFIG_EXT3_FS_XATTR=y
CONFIG_EXT3_FS_POSIX_ACL=y
CONFIG_EXT3_FS_SECURITY=y
CONFIG_EXT4_FS=y
CONFIG_EXT4_FS_XATTR=y
CONFIG_EXT4_FS_POSIX_ACL=y
CONFIG_EXT4_FS_SECURITY=y
# CONFIG_EXT4_DEBUG is not set
CONFIG_FS_XIP=y
CONFIG_JBD=y
# CONFIG_JBD_DEBUG is not set
CONFIG_JBD2=y
# CONFIG_JBD2_DEBUG is not set
CONFIG_FS_MBCACHE=y
# CONFIG_REISERFS_FS is not set
# CONFIG_JFS_FS is not set
CONFIG_FS_POSIX_ACL=y
CONFIG_XFS_FS=y
CONFIG_XFS_QUOTA=y
CONFIG_XFS_POSIX_ACL=y
# CONFIG_XFS_RT is not set
# CONFIG_XFS_DEBUG is not set
# CONFIG_GFS2_FS is not set
# CONFIG_OCFS2_FS is not set
# CONFIG_BTRFS_FS is not set
# CONFIG_NILFS2_FS is not set
CONFIG_FILE_LOCKING=y
CONFIG_FSNOTIFY=y
CONFIG_DNOTIFY=y
CONFIG_INOTIFY=y
CONFIG_INOTIFY_USER=y
CONFIG_QUOTA=y
# CONFIG_QUOTA_NETLINK_INTERFACE is not set
CONFIG_PRINT_QUOTA_WARNING=y
CONFIG_QUOTA_TREE=y
# CONFIG_QFMT_V1 is not set
CONFIG_QFMT_V2=y
CONFIG_QUOTACTL=y
# CONFIG_AUTOFS_FS is not set
CONFIG_AUTOFS4_FS=y
CONFIG_FUSE_FS=y
# CONFIG_CUSE is not set
#
# Caches
#
CONFIG_FSCACHE=y
# CONFIG_FSCACHE_STATS is not set
# CONFIG_FSCACHE_HISTOGRAM is not set
# CONFIG_FSCACHE_DEBUG is not set
# CONFIG_FSCACHE_OBJECT_LIST is not set
CONFIG_CACHEFILES=m
CONFIG_CACHEFILES_DEBUG=y
# CONFIG_CACHEFILES_HISTOGRAM is not set
#
# CD-ROM/DVD Filesystems
#
CONFIG_ISO9660_FS=y
CONFIG_JOLIET=y
CONFIG_ZISOFS=y
CONFIG_UDF_FS=y
CONFIG_UDF_NLS=y
#
# DOS/FAT/NT Filesystems
#
CONFIG_FAT_FS=y
CONFIG_MSDOS_FS=y
CONFIG_VFAT_FS=y
CONFIG_FAT_DEFAULT_CODEPAGE=437
CONFIG_FAT_DEFAULT_IOCHARSET="ascii"
# CONFIG_NTFS_FS is not set
#
# Pseudo filesystems
#
CONFIG_PROC_FS=y
CONFIG_PROC_KCORE=y
CONFIG_PROC_VMCORE=y
CONFIG_PROC_SYSCTL=y
CONFIG_PROC_PAGE_MONITOR=y
CONFIG_SYSFS=y
CONFIG_TMPFS=y
# CONFIG_TMPFS_POSIX_ACL is not set
# CONFIG_HUGETLBFS is not set
# CONFIG_HUGETLB_PAGE is not set
CONFIG_CONFIGFS_FS=m
CONFIG_MISC_FILESYSTEMS=y
# CONFIG_ADFS_FS is not set
# CONFIG_AFFS_FS is not set
# CONFIG_ECRYPT_FS is not set
# CONFIG_HFS_FS is not set
# CONFIG_HFSPLUS_FS is not set
# CONFIG_BEFS_FS is not set
# CONFIG_BFS_FS is not set
# CONFIG_EFS_FS is not set
# CONFIG_CRAMFS is not set
# CONFIG_SQUASHFS is not set
# CONFIG_VXFS_FS is not set
# CONFIG_MINIX_FS is not set
# CONFIG_OMFS_FS is not set
# CONFIG_HPFS_FS is not set
# CONFIG_QNX4FS_FS is not set
# CONFIG_ROMFS_FS is not set
# CONFIG_SYSV_FS is not set
# CONFIG_UFS_FS is not set
CONFIG_NETWORK_FILESYSTEMS=y
CONFIG_NFS_FS=y
CONFIG_NFS_V3=y
CONFIG_NFS_V3_ACL=y
CONFIG_NFS_V4=y
CONFIG_NFS_V4_1=y
CONFIG_NFS_FSCACHE=y
CONFIG_NFSD=y
CONFIG_NFSD_V2_ACL=y
CONFIG_NFSD_V3=y
CONFIG_NFSD_V3_ACL=y
CONFIG_NFSD_V4=y
CONFIG_LOCKD=y
CONFIG_LOCKD_V4=y
CONFIG_EXPORTFS=y
CONFIG_NFS_ACL_SUPPORT=y
CONFIG_NFS_COMMON=y
CONFIG_SUNRPC=y
CONFIG_SUNRPC_GSS=y
CONFIG_RPCSEC_GSS_KRB5=y
CONFIG_RPCSEC_GSS_SPKM3=m
# CONFIG_SMB_FS is not set
CONFIG_CIFS=y
# CONFIG_CIFS_STATS is not set
CONFIG_CIFS_WEAK_PW_HASH=y
# CONFIG_CIFS_UPCALL is not set
CONFIG_CIFS_XATTR=y
CONFIG_CIFS_POSIX=y
# CONFIG_CIFS_DEBUG2 is not set
# CONFIG_CIFS_DFS_UPCALL is not set
# CONFIG_CIFS_EXPERIMENTAL is not set
# CONFIG_NCP_FS is not set
# CONFIG_CODA_FS is not set
# CONFIG_AFS_FS is not set
#
# Partition Types
#
CONFIG_PARTITION_ADVANCED=y
# CONFIG_ACORN_PARTITION is not set
CONFIG_OSF_PARTITION=y
CONFIG_AMIGA_PARTITION=y
# CONFIG_ATARI_PARTITION is not set
CONFIG_MAC_PARTITION=y
CONFIG_MSDOS_PARTITION=y
CONFIG_BSD_DISKLABEL=y
CONFIG_MINIX_SUBPARTITION=y
CONFIG_SOLARIS_X86_PARTITION=y
CONFIG_UNIXWARE_DISKLABEL=y
# CONFIG_LDM_PARTITION is not set
CONFIG_SGI_PARTITION=y
# CONFIG_ULTRIX_PARTITION is not set
CONFIG_SUN_PARTITION=y
CONFIG_KARMA_PARTITION=y
CONFIG_EFI_PARTITION=y
# CONFIG_SYSV68_PARTITION is not set
CONFIG_NLS=y
CONFIG_NLS_DEFAULT="utf8"
CONFIG_NLS_CODEPAGE_437=y
CONFIG_NLS_CODEPAGE_737=m
CONFIG_NLS_CODEPAGE_775=m
CONFIG_NLS_CODEPAGE_850=m
CONFIG_NLS_CODEPAGE_852=m
CONFIG_NLS_CODEPAGE_855=m
CONFIG_NLS_CODEPAGE_857=m
CONFIG_NLS_CODEPAGE_860=m
CONFIG_NLS_CODEPAGE_861=m
CONFIG_NLS_CODEPAGE_862=m
CONFIG_NLS_CODEPAGE_863=m
CONFIG_NLS_CODEPAGE_864=m
CONFIG_NLS_CODEPAGE_865=m
CONFIG_NLS_CODEPAGE_866=m
CONFIG_NLS_CODEPAGE_869=m
CONFIG_NLS_CODEPAGE_936=m
CONFIG_NLS_CODEPAGE_950=m
CONFIG_NLS_CODEPAGE_932=m
CONFIG_NLS_CODEPAGE_949=m
CONFIG_NLS_CODEPAGE_874=m
CONFIG_NLS_ISO8859_8=m
CONFIG_NLS_CODEPAGE_1250=m
CONFIG_NLS_CODEPAGE_1251=m
CONFIG_NLS_ASCII=y
CONFIG_NLS_ISO8859_1=m
CONFIG_NLS_ISO8859_2=m
CONFIG_NLS_ISO8859_3=m
CONFIG_NLS_ISO8859_4=m
CONFIG_NLS_ISO8859_5=m
CONFIG_NLS_ISO8859_6=m
CONFIG_NLS_ISO8859_7=m
CONFIG_NLS_ISO8859_9=m
CONFIG_NLS_ISO8859_13=m
CONFIG_NLS_ISO8859_14=m
CONFIG_NLS_ISO8859_15=m
CONFIG_NLS_KOI8_R=m
CONFIG_NLS_KOI8_U=m
CONFIG_NLS_UTF8=m
# CONFIG_DLM is not set
#
# Kernel hacking
#
CONFIG_TRACE_IRQFLAGS_SUPPORT=y
# CONFIG_PRINTK_TIME is not set
CONFIG_ENABLE_WARN_DEPRECATED=y
CONFIG_ENABLE_MUST_CHECK=y
CONFIG_FRAME_WARN=2048
CONFIG_MAGIC_SYSRQ=y
# CONFIG_STRIP_ASM_SYMS is not set
# CONFIG_UNUSED_SYMBOLS is not set
CONFIG_DEBUG_FS=y
# CONFIG_HEADERS_CHECK is not set
CONFIG_DEBUG_KERNEL=y
# CONFIG_DEBUG_SHIRQ is not set
CONFIG_DETECT_SOFTLOCKUP=y
# CONFIG_BOOTPARAM_SOFTLOCKUP_PANIC is not set
CONFIG_BOOTPARAM_SOFTLOCKUP_PANIC_VALUE=0
CONFIG_DETECT_HUNG_TASK=y
# CONFIG_BOOTPARAM_HUNG_TASK_PANIC is not set
CONFIG_BOOTPARAM_HUNG_TASK_PANIC_VALUE=0
CONFIG_SCHED_DEBUG=y
CONFIG_SCHEDSTATS=y
# CONFIG_TIMER_STATS is not set
# CONFIG_DEBUG_OBJECTS is not set
# CONFIG_DEBUG_SLAB is not set
# CONFIG_DEBUG_KMEMLEAK is not set
# CONFIG_DEBUG_RT_MUTEXES is not set
# CONFIG_RT_MUTEX_TESTER is not set
# CONFIG_DEBUG_SPINLOCK is not set
# CONFIG_DEBUG_MUTEXES is not set
# CONFIG_DEBUG_LOCK_ALLOC is not set
# CONFIG_PROVE_LOCKING is not set
# CONFIG_LOCK_STAT is not set
# CONFIG_DEBUG_SPINLOCK_SLEEP is not set
# CONFIG_DEBUG_LOCKING_API_SELFTESTS is not set
CONFIG_STACKTRACE=y
# CONFIG_DEBUG_KOBJECT is not set
CONFIG_DEBUG_BUGVERBOSE=y
CONFIG_DEBUG_INFO=y
# CONFIG_DEBUG_VM is not set
# CONFIG_DEBUG_VIRTUAL is not set
# CONFIG_DEBUG_WRITECOUNT is not set
CONFIG_DEBUG_MEMORY_INIT=y
CONFIG_DEBUG_LIST=y
# CONFIG_DEBUG_SG is not set
# CONFIG_DEBUG_NOTIFIERS is not set
# CONFIG_DEBUG_CREDENTIALS is not set
CONFIG_ARCH_WANT_FRAME_POINTERS=y
# CONFIG_FRAME_POINTER is not set
# CONFIG_BOOT_PRINTK_DELAY is not set
# CONFIG_RCU_TORTURE_TEST is not set
# CONFIG_RCU_CPU_STALL_DETECTOR is not set
# CONFIG_KPROBES_SANITY_TEST is not set
# CONFIG_BACKTRACE_SELF_TEST is not set
# CONFIG_DEBUG_BLOCK_EXT_DEVT is not set
# CONFIG_DEBUG_FORCE_WEAK_PER_CPU is not set
# CONFIG_LKDTM is not set
# CONFIG_FAULT_INJECTION is not set
# CONFIG_LATENCYTOP is not set
# CONFIG_SYSCTL_SYSCALL_CHECK is not set
# CONFIG_DEBUG_PAGEALLOC is not set
CONFIG_USER_STACKTRACE_SUPPORT=y
CONFIG_NOP_TRACER=y
CONFIG_HAVE_FUNCTION_TRACER=y
CONFIG_HAVE_FUNCTION_GRAPH_TRACER=y
CONFIG_HAVE_FUNCTION_GRAPH_FP_TEST=y
CONFIG_HAVE_FUNCTION_TRACE_MCOUNT_TEST=y
CONFIG_HAVE_DYNAMIC_FTRACE=y
CONFIG_HAVE_FTRACE_MCOUNT_RECORD=y
CONFIG_HAVE_SYSCALL_TRACEPOINTS=y
CONFIG_RING_BUFFER=y
CONFIG_EVENT_TRACING=y
CONFIG_CONTEXT_SWITCH_TRACER=y
CONFIG_TRACING=y
CONFIG_GENERIC_TRACER=y
CONFIG_TRACING_SUPPORT=y
CONFIG_FTRACE=y
# CONFIG_FUNCTION_TRACER is not set
# CONFIG_IRQSOFF_TRACER is not set
# CONFIG_SYSPROF_TRACER is not set
# CONFIG_SCHED_TRACER is not set
# CONFIG_FTRACE_SYSCALLS is not set
# CONFIG_BOOT_TRACER is not set
CONFIG_BRANCH_PROFILE_NONE=y
# CONFIG_PROFILE_ANNOTATED_BRANCHES is not set
# CONFIG_PROFILE_ALL_BRANCHES is not set
# CONFIG_POWER_TRACER is not set
# CONFIG_STACK_TRACER is not set
# CONFIG_KMEMTRACE is not set
# CONFIG_WORKQUEUE_TRACER is not set
CONFIG_BLK_DEV_IO_TRACE=y
# CONFIG_FTRACE_STARTUP_TEST is not set
# CONFIG_MMIOTRACE is not set
# CONFIG_RING_BUFFER_BENCHMARK is not set
# CONFIG_PROVIDE_OHCI1394_DMA_INIT is not set
# CONFIG_FIREWIRE_OHCI_REMOTE_DMA is not set
# CONFIG_DYNAMIC_DEBUG is not set
# CONFIG_DMA_API_DEBUG is not set
# CONFIG_SAMPLES is not set
CONFIG_HAVE_ARCH_KGDB=y
# CONFIG_KGDB is not set
CONFIG_HAVE_ARCH_KMEMCHECK=y
# CONFIG_STRICT_DEVMEM is not set
CONFIG_X86_VERBOSE_BOOTUP=y
CONFIG_EARLY_PRINTK=y
# CONFIG_EARLY_PRINTK_DBGP is not set
CONFIG_DEBUG_STACKOVERFLOW=y
# CONFIG_DEBUG_STACK_USAGE is not set
# CONFIG_DEBUG_PER_CPU_MAPS is not set
# CONFIG_X86_PTDUMP is not set
CONFIG_DEBUG_RODATA=y
CONFIG_DEBUG_RODATA_TEST=y
# CONFIG_DEBUG_NX_TEST is not set
# CONFIG_IOMMU_DEBUG is not set
# CONFIG_IOMMU_STRESS is not set
CONFIG_HAVE_MMIOTRACE_SUPPORT=y
CONFIG_IO_DELAY_TYPE_0X80=0
CONFIG_IO_DELAY_TYPE_0XED=1
CONFIG_IO_DELAY_TYPE_UDELAY=2
CONFIG_IO_DELAY_TYPE_NONE=3
CONFIG_IO_DELAY_0X80=y
# CONFIG_IO_DELAY_0XED is not set
# CONFIG_IO_DELAY_UDELAY is not set
# CONFIG_IO_DELAY_NONE is not set
CONFIG_DEFAULT_IO_DELAY_TYPE=0
# CONFIG_DEBUG_BOOT_PARAMS is not set
# CONFIG_CPA_DEBUG is not set
# CONFIG_OPTIMIZE_INLINING is not set
#
# Security options
#
CONFIG_KEYS=y
CONFIG_KEYS_DEBUG_PROC_KEYS=y
CONFIG_SECURITY=y
# CONFIG_SECURITYFS is not set
CONFIG_SECURITY_NETWORK=y
CONFIG_SECURITY_NETWORK_XFRM=y
# CONFIG_SECURITY_PATH is not set
# CONFIG_SECURITY_FILE_CAPABILITIES is not set
CONFIG_LSM_MMAP_MIN_ADDR=65536
CONFIG_SECURITY_SELINUX=y
CONFIG_SECURITY_SELINUX_BOOTPARAM=y
CONFIG_SECURITY_SELINUX_BOOTPARAM_VALUE=0
CONFIG_SECURITY_SELINUX_DISABLE=y
CONFIG_SECURITY_SELINUX_DEVELOP=y
CONFIG_SECURITY_SELINUX_AVC_STATS=y
CONFIG_SECURITY_SELINUX_CHECKREQPROT_VALUE=1
# CONFIG_SECURITY_SELINUX_POLICYDB_VERSION_MAX is not set
# CONFIG_SECURITY_SMACK is not set
# CONFIG_SECURITY_TOMOYO is not set
# CONFIG_IMA is not set
CONFIG_XOR_BLOCKS=y
CONFIG_ASYNC_CORE=y
CONFIG_ASYNC_MEMCPY=y
CONFIG_ASYNC_XOR=y
CONFIG_ASYNC_PQ=y
CONFIG_ASYNC_RAID6_RECOV=y
CONFIG_CRYPTO=y
#
# Crypto core or helper
#
CONFIG_CRYPTO_ALGAPI=y
CONFIG_CRYPTO_ALGAPI2=y
CONFIG_CRYPTO_AEAD=y
CONFIG_CRYPTO_AEAD2=y
CONFIG_CRYPTO_BLKCIPHER=y
CONFIG_CRYPTO_BLKCIPHER2=y
CONFIG_CRYPTO_HASH=y
CONFIG_CRYPTO_HASH2=y
CONFIG_CRYPTO_RNG=m
CONFIG_CRYPTO_RNG2=y
CONFIG_CRYPTO_PCOMP=y
CONFIG_CRYPTO_MANAGER=y
CONFIG_CRYPTO_MANAGER2=y
# CONFIG_CRYPTO_GF128MUL is not set
CONFIG_CRYPTO_NULL=m
CONFIG_CRYPTO_WORKQUEUE=y
# CONFIG_CRYPTO_CRYPTD is not set
CONFIG_CRYPTO_AUTHENC=y
# CONFIG_CRYPTO_TEST is not set
#
# Authenticated Encryption with Associated Data
#
CONFIG_CRYPTO_CCM=m
# CONFIG_CRYPTO_GCM is not set
CONFIG_CRYPTO_SEQIV=m
#
# Block modes
#
CONFIG_CRYPTO_CBC=y
CONFIG_CRYPTO_CTR=m
# CONFIG_CRYPTO_CTS is not set
CONFIG_CRYPTO_ECB=y
# CONFIG_CRYPTO_LRW is not set
# CONFIG_CRYPTO_PCBC is not set
# CONFIG_CRYPTO_XTS is not set
#
# Hash modes
#
CONFIG_CRYPTO_HMAC=y
CONFIG_CRYPTO_XCBC=m
# CONFIG_CRYPTO_VMAC is not set
#
# Digest
#
CONFIG_CRYPTO_CRC32C=y
# CONFIG_CRYPTO_CRC32C_INTEL is not set
# CONFIG_CRYPTO_GHASH is not set
CONFIG_CRYPTO_MD4=m
CONFIG_CRYPTO_MD5=y
CONFIG_CRYPTO_MICHAEL_MIC=m
# CONFIG_CRYPTO_RMD128 is not set
# CONFIG_CRYPTO_RMD160 is not set
# CONFIG_CRYPTO_RMD256 is not set
# CONFIG_CRYPTO_RMD320 is not set
CONFIG_CRYPTO_SHA1=y
CONFIG_CRYPTO_SHA256=m
CONFIG_CRYPTO_SHA512=m
CONFIG_CRYPTO_TGR192=m
CONFIG_CRYPTO_WP512=m
#
# Ciphers
#
CONFIG_CRYPTO_AES=m
CONFIG_CRYPTO_AES_X86_64=m
# CONFIG_CRYPTO_AES_NI_INTEL is not set
CONFIG_CRYPTO_ANUBIS=m
CONFIG_CRYPTO_ARC4=y
CONFIG_CRYPTO_BLOWFISH=m
# CONFIG_CRYPTO_CAMELLIA is not set
CONFIG_CRYPTO_CAST5=m
CONFIG_CRYPTO_CAST6=m
CONFIG_CRYPTO_DES=y
# CONFIG_CRYPTO_FCRYPT is not set
CONFIG_CRYPTO_KHAZAD=m
# CONFIG_CRYPTO_SALSA20 is not set
# CONFIG_CRYPTO_SALSA20_X86_64 is not set
# CONFIG_CRYPTO_SEED is not set
CONFIG_CRYPTO_SERPENT=m
CONFIG_CRYPTO_TEA=m
CONFIG_CRYPTO_TWOFISH=m
CONFIG_CRYPTO_TWOFISH_COMMON=m
# CONFIG_CRYPTO_TWOFISH_X86_64 is not set
#
# Compression
#
CONFIG_CRYPTO_DEFLATE=y
# CONFIG_CRYPTO_ZLIB is not set
# CONFIG_CRYPTO_LZO is not set
#
# Random Number Generation
#
# CONFIG_CRYPTO_ANSI_CPRNG is not set
CONFIG_CRYPTO_HW=y
# CONFIG_CRYPTO_DEV_PADLOCK is not set
# CONFIG_CRYPTO_DEV_HIFN_795X is not set
CONFIG_HAVE_KVM=y
CONFIG_VIRTUALIZATION=y
# CONFIG_KVM is not set
# CONFIG_VIRTIO_PCI is not set
# CONFIG_VIRTIO_BALLOON is not set
CONFIG_BINARY_PRINTF=y
#
# Library routines
#
CONFIG_BITREVERSE=y
CONFIG_GENERIC_FIND_FIRST_BIT=y
CONFIG_GENERIC_FIND_NEXT_BIT=y
CONFIG_GENERIC_FIND_LAST_BIT=y
CONFIG_CRC_CCITT=y
CONFIG_CRC16=y
# CONFIG_CRC_T10DIF is not set
CONFIG_CRC_ITU_T=y
CONFIG_CRC32=y
# CONFIG_CRC7 is not set
CONFIG_LIBCRC32C=y
CONFIG_ZLIB_INFLATE=y
CONFIG_ZLIB_DEFLATE=y
CONFIG_DECOMPRESS_GZIP=y
CONFIG_DECOMPRESS_BZIP2=y
CONFIG_DECOMPRESS_LZMA=y
CONFIG_TEXTSEARCH=y
CONFIG_TEXTSEARCH_KMP=y
CONFIG_TEXTSEARCH_BM=y
CONFIG_TEXTSEARCH_FSM=y
CONFIG_HAS_IOMEM=y
CONFIG_HAS_IOPORT=y
CONFIG_HAS_DMA=y
CONFIG_NLATTR=y
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: pvops domu soft lockup under load (more logs)
2010-04-10 7:12 ` Pim van Riezen
@ 2010-04-12 9:44 ` Pim van Riezen
2010-04-14 17:48 ` Jeremy Fitzhardinge
0 siblings, 1 reply; 17+ messages in thread
From: Pim van Riezen @ 2010-04-12 9:44 UTC (permalink / raw)
To: xen-devel
On Apr 10, 2010, at 9:12 , Pim van Riezen wrote:
>> When you're getting lockups, could you capture the output of sysrq-t (xm
>> sysrq <domain> t, or "Ctrl-O t" on the domain's console)?
>
> This one is going to be a bit tough. I only get the error after the fact. The domain has multiple vcpu's, so it's usually not apparent that it's hanging, all depending on which process gets stuck. I'll keep poking at it and if I manage to catch it in the act I'll follow up with a sysrq-t.
xm sysrq $vps t doesn't seem to add anything to the console. Am I doing it wrong?
Pi
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: pvops domu soft lockup under load (more logs)
2010-04-12 9:44 ` Pim van Riezen
@ 2010-04-14 17:48 ` Jeremy Fitzhardinge
2010-04-15 10:56 ` Pim van Riezen
2010-04-16 14:19 ` Pim van Riezen
0 siblings, 2 replies; 17+ messages in thread
From: Jeremy Fitzhardinge @ 2010-04-14 17:48 UTC (permalink / raw)
To: Pim van Riezen; +Cc: xen-devel
On 04/12/2010 02:44 AM, Pim van Riezen wrote:
> On Apr 10, 2010, at 9:12 , Pim van Riezen wrote:
>
>
>>> When you're getting lockups, could you capture the output of sysrq-t (xm
>>> sysrq <domain> t, or "Ctrl-O t" on the domain's console)?
>>>
>> This one is going to be a bit tough. I only get the error after the fact. The domain has multiple vcpu's, so it's usually not apparent that it's hanging, all depending on which process gets stuck. I'll keep poking at it and if I manage to catch it in the act I'll follow up with a sysrq-t.
>>
> xm sysrq $vps t doesn't seem to add anything to the console. Am I doing it wrong?
>
Does it appear on the Xen console (visible with "xm dmesg")? You may
need to do a sysrq '9' first to get it to output all messages.
J
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: pvops domu soft lockup under load (more logs)
2010-04-14 17:48 ` Jeremy Fitzhardinge
@ 2010-04-15 10:56 ` Pim van Riezen
2010-04-15 17:21 ` Jeremy Fitzhardinge
2010-04-16 14:19 ` Pim van Riezen
1 sibling, 1 reply; 17+ messages in thread
From: Pim van Riezen @ 2010-04-15 10:56 UTC (permalink / raw)
To: Jeremy Fitzhardinge; +Cc: xen-devel
On Apr 14, 2010, at 19:48 , Jeremy Fitzhardinge wrote:
> Does it appear on the Xen console (visible with "xm dmesg")? You may
> need to do a sysrq '9' first to get it to output all messages.
Does that sysrq have to go to dom0 or to the domU?
I also got the tip from someone on xen-users to take a look at the clock source, and setting /sys/devices/system/clocksource/clocksource0/current_clocksource to 'jiffies' does seem to stop the lockups from happening. The fact that I've witnessed other kernel timekeeping going wonky on these guests, the fact that this helps does seem intuitively right, but I don't know if it gives you enough to work with.
Cheers,
Pim
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: pvops domu soft lockup under load (more logs)
2010-04-15 10:56 ` Pim van Riezen
@ 2010-04-15 17:21 ` Jeremy Fitzhardinge
2010-04-16 7:37 ` Pim van Riezen
2010-04-16 14:20 ` Pim van Riezen
0 siblings, 2 replies; 17+ messages in thread
From: Jeremy Fitzhardinge @ 2010-04-15 17:21 UTC (permalink / raw)
To: Pim van Riezen; +Cc: xen-devel
On 04/15/2010 03:56 AM, Pim van Riezen wrote:
> On Apr 14, 2010, at 19:48 , Jeremy Fitzhardinge wrote:
>
>
>> Does it appear on the Xen console (visible with "xm dmesg")? You may
>> need to do a sysrq '9' first to get it to output all messages.
>>
> Does that sysrq have to go to dom0 or to the domU?
>
To the locked-up domU.
> I also got the tip from someone on xen-users to take a look at the clock source, and setting /sys/devices/system/clocksource/clocksource0/current_clocksource to 'jiffies' does seem to stop the lockups from happening. The fact that I've witnessed other kernel timekeeping going wonky on these guests, the fact that this helps does seem intuitively right, but I don't know if it gives you enough to work with.
>
I've seen similar lockups at a very low rate. The clocksource
workaround just confuses me; the whole thing stumps me. The main piece
of evidence I haven't managed to get yet is a complete process dump
(sysrq-t) to see who's waiting on what.
Also, could you try 2.6.32.11, which has some timer-related fixes in it,
that may or may not help.
J
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: pvops domu soft lockup under load (more logs)
2010-04-15 17:21 ` Jeremy Fitzhardinge
@ 2010-04-16 7:37 ` Pim van Riezen
2010-04-16 7:55 ` Pim van Riezen
` (2 more replies)
2010-04-16 14:20 ` Pim van Riezen
1 sibling, 3 replies; 17+ messages in thread
From: Pim van Riezen @ 2010-04-16 7:37 UTC (permalink / raw)
To: Jeremy Fitzhardinge; +Cc: xen-devel
On Apr 15, 2010, at 19:21 , Jeremy Fitzhardinge wrote:
> On 04/15/2010 03:56 AM, Pim van Riezen wrote:
>> On Apr 14, 2010, at 19:48 , Jeremy Fitzhardinge wrote:
>>
>>
>>> Does it appear on the Xen console (visible with "xm dmesg")? You may
>>> need to do a sysrq '9' first to get it to output all messages.
>>>
>> Does that sysrq have to go to dom0 or to the domU?
>>
>
> To the locked-up domU.
Ok I'm going to try and work on a reproduction scenario at our test-cluster. These issues were with a customer vps and his patience sort of ran out on experimental reboots and playing a guinea-pig.
> I've seen similar lockups at a very low rate. The clocksource
> workaround just confuses me; the whole thing stumps me. The main piece
> of evidence I haven't managed to get yet is a complete process dump
> (sysrq-t) to see who's waiting on what.
Another datapoint. This customer has similarly loaded VPS machines on a number of different hardware nodes. Not all of them had the lockup problem. I applied the jiffies clocksource to all his machines, regardless of their current problem status. After a day without lockups, the customer complained about time drift (ntp was not activated). The guest that had experienced the soft lockups earlier had major clock drift and were way ahead:
16 Apr 09:29:26 ntpdate[11236]: step time server 194.109.22.18 offset -7337.731686 sec
That's over 2 hours accumulated in less than 24 hours of uptime. The guests that hadn't been excperiencing the lockup issues berfore switching to the jiffies clocksource hadn't drifted that much after the switch and were, at most, 120s behind after the same amount of runtime.
> Also, could you try 2.6.32.11, which has some timer-related fixes in it,
> that may or may not help.
If I can get a reproduction scenario going I'll look into that as well.
Cheers,
Pim
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: pvops domu soft lockup under load (more logs)
2010-04-16 7:37 ` Pim van Riezen
@ 2010-04-16 7:55 ` Pim van Riezen
2010-04-16 15:17 ` Dan Magenheimer
2010-04-16 15:24 ` Pim van Riezen
[not found] ` <9D75997F-4855-41A0-B159-18B6A3BFC776@panelsix.com>
2010-04-16 20:22 ` Jeremy Fitzhardinge
2 siblings, 2 replies; 17+ messages in thread
From: Pim van Riezen @ 2010-04-16 7:55 UTC (permalink / raw)
To: Pim van Riezen; +Cc: Jeremy Fitzhardinge, xen-devel
[-- Attachment #1: Type: text/plain, Size: 1344 bytes --]
Oh,
On Apr 16, 2010, at 9:37 , Pim van Riezen wrote:
> Another datapoint. This customer has similarly loaded VPS machines on a number of different hardware nodes. Not all of them had the lockup problem. I applied the jiffies clocksource to all his machines, regardless of their current problem status. After a day without lockups, the customer complained about time drift (ntp was not activated). The guest that had experienced the soft lockups earlier had major clock drift and were way ahead:
>
> 16 Apr 09:29:26 ntpdate[11236]: step time server 194.109.22.18 offset -7337.731686 sec
>
> That's over 2 hours accumulated in less than 24 hours of uptime. The guests that hadn't been excperiencing the lockup issues berfore switching to the jiffies clocksource hadn't drifted that much after the switch and were, at most, 120s behind after the same amount of runtime.
There's more correlation between the guests that had the lockups and those that didn't: the guests that locked up (and now have a way speedy jiffies clock) were all on the same hardware platform, with an older Xeon CPU than on the guests that had no issues. I attached cpuinfo for both the broken and the non-broken dom0s. All are on Xen-3.4.1 (hypervisor-version doesn't seem to affect this issue) and the latest CentOS 2.6.18 dom0-kernel.
Cheers,
Pim
[-- Attachment #2: cpuinfo_lockups.txt --]
[-- Type: text/plain, Size: 4750 bytes --]
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 23
model name : Intel(R) Xeon(R) CPU E5440 @ 2.83GHz
stepping : 10
cpu MHz : 2826.254
cache size : 6144 KB
physical id : 0
siblings : 1
core id : 0
cpu cores : 1
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu de tsc msr pae mce cx8 apic sep mtrr mca cmov pat clflush acpi mmx fxsr sse sse2 ss ht syscall nx lm constant_tsc pni est cx16 lahf_lm
bogomips : 7067.84
clflush size : 64
cache_alignment : 64
address sizes : 38 bits physical, 48 bits virtual
power management:
processor : 1
vendor_id : GenuineIntel
cpu family : 6
model : 23
model name : Intel(R) Xeon(R) CPU E5440 @ 2.83GHz
stepping : 10
cpu MHz : 2826.254
cache size : 6144 KB
physical id : 1
siblings : 1
core id : 0
cpu cores : 1
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu de tsc msr pae mce cx8 apic sep mtrr mca cmov pat clflush acpi mmx fxsr sse sse2 ss ht syscall nx lm constant_tsc pni est cx16 lahf_lm
bogomips : 7067.84
clflush size : 64
cache_alignment : 64
address sizes : 38 bits physical, 48 bits virtual
power management:
processor : 2
vendor_id : GenuineIntel
cpu family : 6
model : 23
model name : Intel(R) Xeon(R) CPU E5440 @ 2.83GHz
stepping : 10
cpu MHz : 2826.254
cache size : 6144 KB
physical id : 2
siblings : 1
core id : 0
cpu cores : 1
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu de tsc msr pae mce cx8 apic sep mtrr mca cmov pat clflush acpi mmx fxsr sse sse2 ss ht syscall nx lm constant_tsc pni est cx16 lahf_lm
bogomips : 7067.84
clflush size : 64
cache_alignment : 64
address sizes : 38 bits physical, 48 bits virtual
power management:
processor : 3
vendor_id : GenuineIntel
cpu family : 6
model : 23
model name : Intel(R) Xeon(R) CPU E5440 @ 2.83GHz
stepping : 10
cpu MHz : 2826.254
cache size : 6144 KB
physical id : 3
siblings : 1
core id : 0
cpu cores : 1
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu de tsc msr pae mce cx8 apic sep mtrr mca cmov pat clflush acpi mmx fxsr sse sse2 ss ht syscall nx lm constant_tsc pni est cx16 lahf_lm
bogomips : 7067.84
clflush size : 64
cache_alignment : 64
address sizes : 38 bits physical, 48 bits virtual
power management:
processor : 4
vendor_id : GenuineIntel
cpu family : 6
model : 23
model name : Intel(R) Xeon(R) CPU E5440 @ 2.83GHz
stepping : 10
cpu MHz : 2826.254
cache size : 6144 KB
physical id : 4
siblings : 1
core id : 0
cpu cores : 1
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu de tsc msr pae mce cx8 apic sep mtrr mca cmov pat clflush acpi mmx fxsr sse sse2 ss ht syscall nx lm constant_tsc pni est cx16 lahf_lm
bogomips : 7067.84
clflush size : 64
cache_alignment : 64
address sizes : 38 bits physical, 48 bits virtual
power management:
processor : 5
vendor_id : GenuineIntel
cpu family : 6
model : 23
model name : Intel(R) Xeon(R) CPU E5440 @ 2.83GHz
stepping : 10
cpu MHz : 2826.254
cache size : 6144 KB
physical id : 5
siblings : 1
core id : 0
cpu cores : 1
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu de tsc msr pae mce cx8 apic sep mtrr mca cmov pat clflush acpi mmx fxsr sse sse2 ss ht syscall nx lm constant_tsc pni est cx16 lahf_lm
bogomips : 7067.84
clflush size : 64
cache_alignment : 64
address sizes : 38 bits physical, 48 bits virtual
power management:
processor : 6
vendor_id : GenuineIntel
cpu family : 6
model : 23
model name : Intel(R) Xeon(R) CPU E5440 @ 2.83GHz
stepping : 10
cpu MHz : 2826.254
cache size : 6144 KB
physical id : 6
siblings : 1
core id : 0
cpu cores : 1
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu de tsc msr pae mce cx8 apic sep mtrr mca cmov pat clflush acpi mmx fxsr sse sse2 ss ht syscall nx lm constant_tsc pni est cx16 lahf_lm
bogomips : 7067.84
clflush size : 64
cache_alignment : 64
address sizes : 38 bits physical, 48 bits virtual
power management:
processor : 7
vendor_id : GenuineIntel
cpu family : 6
model : 23
model name : Intel(R) Xeon(R) CPU E5440 @ 2.83GHz
stepping : 10
cpu MHz : 2826.254
cache size : 6144 KB
physical id : 7
siblings : 1
core id : 0
cpu cores : 1
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu de tsc msr pae mce cx8 apic sep mtrr mca cmov pat clflush acpi mmx fxsr sse sse2 ss ht syscall nx lm constant_tsc pni est cx16 lahf_lm
bogomips : 7067.84
clflush size : 64
cache_alignment : 64
address sizes : 38 bits physical, 48 bits virtual
power management:
[-- Attachment #3: cpuinfo_nolockups.txt --]
[-- Type: text/plain, Size: 9500 bytes --]
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 26
model name : Intel(R) Xeon(R) CPU E5530 @ 2.40GHz
stepping : 5
cpu MHz : 2394.000
cache size : 8192 KB
physical id : 0
siblings : 1
core id : 0
cpu cores : 1
fpu : yes
fpu_exception : yes
cpuid level : 11
wp : yes
flags : fpu de tsc msr pae mce cx8 apic sep mtrr mca cmov pat clflush acpi mmx fxsr sse sse2 ss ht syscall nx lm constant_tsc pni est cx16 lahf_lm
bogomips : 5986.87
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management:
processor : 1
vendor_id : GenuineIntel
cpu family : 6
model : 26
model name : Intel(R) Xeon(R) CPU E5530 @ 2.40GHz
stepping : 5
cpu MHz : 2394.000
cache size : 8192 KB
physical id : 1
siblings : 1
core id : 0
cpu cores : 1
fpu : yes
fpu_exception : yes
cpuid level : 11
wp : yes
flags : fpu de tsc msr pae mce cx8 apic sep mtrr mca cmov pat clflush acpi mmx fxsr sse sse2 ss ht syscall nx lm constant_tsc pni est cx16 lahf_lm
bogomips : 5986.87
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management:
processor : 2
vendor_id : GenuineIntel
cpu family : 6
model : 26
model name : Intel(R) Xeon(R) CPU E5530 @ 2.40GHz
stepping : 5
cpu MHz : 2394.000
cache size : 8192 KB
physical id : 2
siblings : 1
core id : 0
cpu cores : 1
fpu : yes
fpu_exception : yes
cpuid level : 11
wp : yes
flags : fpu de tsc msr pae mce cx8 apic sep mtrr mca cmov pat clflush acpi mmx fxsr sse sse2 ss ht syscall nx lm constant_tsc pni est cx16 lahf_lm
bogomips : 5986.87
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management:
processor : 3
vendor_id : GenuineIntel
cpu family : 6
model : 26
model name : Intel(R) Xeon(R) CPU E5530 @ 2.40GHz
stepping : 5
cpu MHz : 2394.000
cache size : 8192 KB
physical id : 3
siblings : 1
core id : 0
cpu cores : 1
fpu : yes
fpu_exception : yes
cpuid level : 11
wp : yes
flags : fpu de tsc msr pae mce cx8 apic sep mtrr mca cmov pat clflush acpi mmx fxsr sse sse2 ss ht syscall nx lm constant_tsc pni est cx16 lahf_lm
bogomips : 5986.87
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management:
processor : 4
vendor_id : GenuineIntel
cpu family : 6
model : 26
model name : Intel(R) Xeon(R) CPU E5530 @ 2.40GHz
stepping : 5
cpu MHz : 2394.000
cache size : 8192 KB
physical id : 4
siblings : 1
core id : 0
cpu cores : 1
fpu : yes
fpu_exception : yes
cpuid level : 11
wp : yes
flags : fpu de tsc msr pae mce cx8 apic sep mtrr mca cmov pat clflush acpi mmx fxsr sse sse2 ss ht syscall nx lm constant_tsc pni est cx16 lahf_lm
bogomips : 5986.87
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management:
processor : 5
vendor_id : GenuineIntel
cpu family : 6
model : 26
model name : Intel(R) Xeon(R) CPU E5530 @ 2.40GHz
stepping : 5
cpu MHz : 2394.000
cache size : 8192 KB
physical id : 5
siblings : 1
core id : 0
cpu cores : 1
fpu : yes
fpu_exception : yes
cpuid level : 11
wp : yes
flags : fpu de tsc msr pae mce cx8 apic sep mtrr mca cmov pat clflush acpi mmx fxsr sse sse2 ss ht syscall nx lm constant_tsc pni est cx16 lahf_lm
bogomips : 5986.87
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management:
processor : 6
vendor_id : GenuineIntel
cpu family : 6
model : 26
model name : Intel(R) Xeon(R) CPU E5530 @ 2.40GHz
stepping : 5
cpu MHz : 2394.000
cache size : 8192 KB
physical id : 6
siblings : 1
core id : 0
cpu cores : 1
fpu : yes
fpu_exception : yes
cpuid level : 11
wp : yes
flags : fpu de tsc msr pae mce cx8 apic sep mtrr mca cmov pat clflush acpi mmx fxsr sse sse2 ss ht syscall nx lm constant_tsc pni est cx16 lahf_lm
bogomips : 5986.87
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management:
processor : 7
vendor_id : GenuineIntel
cpu family : 6
model : 26
model name : Intel(R) Xeon(R) CPU E5530 @ 2.40GHz
stepping : 5
cpu MHz : 2394.000
cache size : 8192 KB
physical id : 7
siblings : 1
core id : 0
cpu cores : 1
fpu : yes
fpu_exception : yes
cpuid level : 11
wp : yes
flags : fpu de tsc msr pae mce cx8 apic sep mtrr mca cmov pat clflush acpi mmx fxsr sse sse2 ss ht syscall nx lm constant_tsc pni est cx16 lahf_lm
bogomips : 5986.87
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management:
processor : 8
vendor_id : GenuineIntel
cpu family : 6
model : 26
model name : Intel(R) Xeon(R) CPU E5530 @ 2.40GHz
stepping : 5
cpu MHz : 2394.000
cache size : 8192 KB
physical id : 8
siblings : 1
core id : 0
cpu cores : 1
fpu : yes
fpu_exception : yes
cpuid level : 11
wp : yes
flags : fpu de tsc msr pae mce cx8 apic sep mtrr mca cmov pat clflush acpi mmx fxsr sse sse2 ss ht syscall nx lm constant_tsc pni est cx16 lahf_lm
bogomips : 5986.87
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management:
processor : 9
vendor_id : GenuineIntel
cpu family : 6
model : 26
model name : Intel(R) Xeon(R) CPU E5530 @ 2.40GHz
stepping : 5
cpu MHz : 2394.000
cache size : 8192 KB
physical id : 9
siblings : 1
core id : 0
cpu cores : 1
fpu : yes
fpu_exception : yes
cpuid level : 11
wp : yes
flags : fpu de tsc msr pae mce cx8 apic sep mtrr mca cmov pat clflush acpi mmx fxsr sse sse2 ss ht syscall nx lm constant_tsc pni est cx16 lahf_lm
bogomips : 5986.87
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management:
processor : 10
vendor_id : GenuineIntel
cpu family : 6
model : 26
model name : Intel(R) Xeon(R) CPU E5530 @ 2.40GHz
stepping : 5
cpu MHz : 2394.000
cache size : 8192 KB
physical id : 10
siblings : 1
core id : 0
cpu cores : 1
fpu : yes
fpu_exception : yes
cpuid level : 11
wp : yes
flags : fpu de tsc msr pae mce cx8 apic sep mtrr mca cmov pat clflush acpi mmx fxsr sse sse2 ss ht syscall nx lm constant_tsc pni est cx16 lahf_lm
bogomips : 5986.87
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management:
processor : 11
vendor_id : GenuineIntel
cpu family : 6
model : 26
model name : Intel(R) Xeon(R) CPU E5530 @ 2.40GHz
stepping : 5
cpu MHz : 2394.000
cache size : 8192 KB
physical id : 11
siblings : 1
core id : 0
cpu cores : 1
fpu : yes
fpu_exception : yes
cpuid level : 11
wp : yes
flags : fpu de tsc msr pae mce cx8 apic sep mtrr mca cmov pat clflush acpi mmx fxsr sse sse2 ss ht syscall nx lm constant_tsc pni est cx16 lahf_lm
bogomips : 5986.87
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management:
processor : 12
vendor_id : GenuineIntel
cpu family : 6
model : 26
model name : Intel(R) Xeon(R) CPU E5530 @ 2.40GHz
stepping : 5
cpu MHz : 2394.000
cache size : 8192 KB
physical id : 12
siblings : 1
core id : 0
cpu cores : 1
fpu : yes
fpu_exception : yes
cpuid level : 11
wp : yes
flags : fpu de tsc msr pae mce cx8 apic sep mtrr mca cmov pat clflush acpi mmx fxsr sse sse2 ss ht syscall nx lm constant_tsc pni est cx16 lahf_lm
bogomips : 5986.87
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management:
processor : 13
vendor_id : GenuineIntel
cpu family : 6
model : 26
model name : Intel(R) Xeon(R) CPU E5530 @ 2.40GHz
stepping : 5
cpu MHz : 2394.000
cache size : 8192 KB
physical id : 13
siblings : 1
core id : 0
cpu cores : 1
fpu : yes
fpu_exception : yes
cpuid level : 11
wp : yes
flags : fpu de tsc msr pae mce cx8 apic sep mtrr mca cmov pat clflush acpi mmx fxsr sse sse2 ss ht syscall nx lm constant_tsc pni est cx16 lahf_lm
bogomips : 5986.87
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management:
processor : 14
vendor_id : GenuineIntel
cpu family : 6
model : 26
model name : Intel(R) Xeon(R) CPU E5530 @ 2.40GHz
stepping : 5
cpu MHz : 2394.000
cache size : 8192 KB
physical id : 14
siblings : 1
core id : 0
cpu cores : 1
fpu : yes
fpu_exception : yes
cpuid level : 11
wp : yes
flags : fpu de tsc msr pae mce cx8 apic sep mtrr mca cmov pat clflush acpi mmx fxsr sse sse2 ss ht syscall nx lm constant_tsc pni est cx16 lahf_lm
bogomips : 5986.87
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management:
processor : 15
vendor_id : GenuineIntel
cpu family : 6
model : 26
model name : Intel(R) Xeon(R) CPU E5530 @ 2.40GHz
stepping : 5
cpu MHz : 2394.000
cache size : 8192 KB
physical id : 15
siblings : 1
core id : 0
cpu cores : 1
fpu : yes
fpu_exception : yes
cpuid level : 11
wp : yes
flags : fpu de tsc msr pae mce cx8 apic sep mtrr mca cmov pat clflush acpi mmx fxsr sse sse2 ss ht syscall nx lm constant_tsc pni est cx16 lahf_lm
bogomips : 5986.87
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management:
[-- Attachment #4: Type: text/plain, Size: 2 bytes --]
[-- Attachment #5: Type: text/plain, Size: 138 bytes --]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: pvops domu soft lockup under load (more logs)
[not found] ` <9D75997F-4855-41A0-B159-18B6A3BFC776@panelsix.com>
@ 2010-04-16 13:25 ` Pim van Riezen
0 siblings, 0 replies; 17+ messages in thread
From: Pim van Riezen @ 2010-04-16 13:25 UTC (permalink / raw)
To: Pim van Riezen; +Cc: Jeremy Fitzhardinge, xen-devel
On Apr 16, 2010, at 15:19 , Pim van Riezen wrote:
>
> On Apr 16, 2010, at 9:37 , Pim van Riezen wrote:
>
>> Another datapoint. This customer has similarly loaded VPS machines on a number of different hardware nodes. Not all of them had the lockup problem. I applied the jiffies clocksource to all his machines, regardless of their current problem status. After a day without lockups, the customer complained about time drift (ntp was not activated). The guest that had experienced the soft lockups earlier had major clock drift and were way ahead:
>>
>> 16 Apr 09:29:26 ntpdate[11236]: step time server 194.109.22.18 offset -7337.731686 sec
>
> 2.6.32.11 fixes the excessive drifting.
Sorry I spoke to soon. Investigating.
Pi
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: pvops domu soft lockup under load (more logs)
2010-04-14 17:48 ` Jeremy Fitzhardinge
2010-04-15 10:56 ` Pim van Riezen
@ 2010-04-16 14:19 ` Pim van Riezen
1 sibling, 0 replies; 17+ messages in thread
From: Pim van Riezen @ 2010-04-16 14:19 UTC (permalink / raw)
To: Jeremy Fitzhardinge; +Cc: xen-devel
On Apr 14, 2010, at 19:48 , Jeremy Fitzhardinge wrote:
> On 04/12/2010 02:44 AM, Pim van Riezen wrote:
>> On Apr 10, 2010, at 9:12 , Pim van Riezen wrote:
>>
>>
>>>> When you're getting lockups, could you capture the output of sysrq-t (xm
>>>> sysrq <domain> t, or "Ctrl-O t" on the domain's console)?
>>>>
>>> This one is going to be a bit tough. I only get the error after the fact. The domain has multiple vcpu's, so it's usually not apparent that it's hanging, all depending on which process gets stuck. I'll keep poking at it and if I manage to catch it in the act I'll follow up with a sysrq-t.
>>>
>> xm sysrq $vps t doesn't seem to add anything to the console. Am I doing it wrong?
>>
>
> Does it appear on the Xen console (visible with "xm dmesg")? You may
> need to do a sysrq '9' first to get it to output all messages.
I tried:
xm sysrq $vps 9
xm sysrq $vps t
nothing shows up on the guest's console, nor in xm dmesg.
Same for doing ^O-9 ^O-t / ^o-9 / ^o-t on the guest's console. CONFIG_MAGIC_SYSRQ is enabled in the domU kernel.
Pi
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: pvops domu soft lockup under load (more logs)
2010-04-15 17:21 ` Jeremy Fitzhardinge
2010-04-16 7:37 ` Pim van Riezen
@ 2010-04-16 14:20 ` Pim van Riezen
1 sibling, 0 replies; 17+ messages in thread
From: Pim van Riezen @ 2010-04-16 14:20 UTC (permalink / raw)
To: Jeremy Fitzhardinge; +Cc: xen-devel
On Apr 15, 2010, at 19:21 , Jeremy Fitzhardinge wrote:
> On 04/15/2010 03:56 AM, Pim van Riezen wrote:
>> On Apr 14, 2010, at 19:48 , Jeremy Fitzhardinge wrote:
>>
>>
>>> Does it appear on the Xen console (visible with "xm dmesg")? You may
>>> need to do a sysrq '9' first to get it to output all messages.
>>>
>> Does that sysrq have to go to dom0 or to the domU?
>>
>
> To the locked-up domU.
I tried:
xm sysrq $vps 9
xm sysrq $vps t
nothing shows up on the guest's console, nor in xm dmesg.
Same for doing ^O-9 ^O-t / ^o-9 / ^o-t on the guest's console. CONFIG_MAGIC_SYSRQ is enabled in the domU kernel.
Pi
^ permalink raw reply [flat|nested] 17+ messages in thread
* RE: pvops domu soft lockup under load (more logs)
2010-04-16 7:55 ` Pim van Riezen
@ 2010-04-16 15:17 ` Dan Magenheimer
2010-04-16 16:00 ` Dan Magenheimer
2010-04-16 15:24 ` Pim van Riezen
1 sibling, 1 reply; 17+ messages in thread
From: Dan Magenheimer @ 2010-04-16 15:17 UTC (permalink / raw)
To: Pim van Riezen; +Cc: Jeremy Fitzhardinge, xen-devel
Hi Pim --
I haven't read all of your previous postings, but with what you are
seeing, I'd be suspicious that your TSC's may be getting badly out
of sync, possibly due to power management.
Could you try booting xen on the "bad" machines with the
Xen boot parameter: max_cstate=0
There may also be power management settings in the BIOS that
can be changed.
Hope that helps!
Dan
> -----Original Message-----
> From: Pim van Riezen [mailto:pi+lists@panelsix.com]
> Sent: Friday, April 16, 2010 1:56 AM
> To: Pim van Riezen
> Cc: Jeremy Fitzhardinge; xen-devel@lists.xensource.com
> Subject: Re: [Xen-devel] pvops domu soft lockup under load (more logs)
>
> Oh,
>
> On Apr 16, 2010, at 9:37 , Pim van Riezen wrote:
>
> > Another datapoint. This customer has similarly loaded VPS machines on
> a number of different hardware nodes. Not all of them had the lockup
> problem. I applied the jiffies clocksource to all his machines,
> regardless of their current problem status. After a day without
> lockups, the customer complained about time drift (ntp was not
> activated). The guest that had experienced the soft lockups earlier had
> major clock drift and were way ahead:
> >
> > 16 Apr 09:29:26 ntpdate[11236]: step time server 194.109.22.18
> offset -7337.731686 sec
> >
> > That's over 2 hours accumulated in less than 24 hours of uptime. The
> guests that hadn't been excperiencing the lockup issues berfore
> switching to the jiffies clocksource hadn't drifted that much after the
> switch and were, at most, 120s behind after the same amount of runtime.
>
> There's more correlation between the guests that had the lockups and
> those that didn't: the guests that locked up (and now have a way speedy
> jiffies clock) were all on the same hardware platform, with an older
> Xeon CPU than on the guests that had no issues. I attached cpuinfo for
> both the broken and the non-broken dom0s. All are on Xen-3.4.1
> (hypervisor-version doesn't seem to affect this issue) and the latest
> CentOS 2.6.18 dom0-kernel.
>
> Cheers,
> Pim
>
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: pvops domu soft lockup under load (more logs)
2010-04-16 7:55 ` Pim van Riezen
2010-04-16 15:17 ` Dan Magenheimer
@ 2010-04-16 15:24 ` Pim van Riezen
1 sibling, 0 replies; 17+ messages in thread
From: Pim van Riezen @ 2010-04-16 15:24 UTC (permalink / raw)
To: Pim van Riezen; +Cc: Jeremy Fitzhardinge, xen-devel
On Apr 16, 2010, at 9:55 , Pim van Riezen wrote:
> There's more correlation between the guests that had the lockups and those that didn't: the guests that locked up (and now have a way speedy jiffies clock) were all on the same hardware platform, with an older Xeon CPU than on the guests that had no issues. I attached cpuinfo for both the broken and the non-broken dom0s. All are on Xen-3.4.1 (hypervisor-version doesn't seem to affect this issue) and the latest CentOS 2.6.18 dom0-kernel.
This seems to be the golden ticket. I managed to migrate two of the problem guests to nodes in the same cluster with different Xeons:
Xeon 5150: OK
Xeon E5440: Screwed
Xeon E5530: OK
Cheers,
Pim
^ permalink raw reply [flat|nested] 17+ messages in thread
* RE: pvops domu soft lockup under load (more logs)
2010-04-16 15:17 ` Dan Magenheimer
@ 2010-04-16 16:00 ` Dan Magenheimer
0 siblings, 0 replies; 17+ messages in thread
From: Dan Magenheimer @ 2010-04-16 16:00 UTC (permalink / raw)
To: dan.magenheimer; +Cc: Jeremy Fitzhardinge, xen-devel
(Resending to list... Our corporate email gateway sadly has problems with
a "+" in an email address so I can't reply to Pim directly... if someone
could forward, I would appreciate it.)
Hi Pim --
I haven't read all of your previous postings, but with what you are
seeing, I'd be suspicious that your TSC's may be getting badly out
of sync, possibly due to power management.
Could you try booting xen on the "bad" machines with the
Xen boot parameter: max_cstate=0
There may also be power management settings in the BIOS that
can be changed.
Hope that helps!
Dan
> > -----Original Message-----
> > From: Pim van Riezen [mailto:pi+lists@panelsix.com]
> > Sent: Friday, April 16, 2010 1:56 AM
> > To: Pim van Riezen
> > Cc: Jeremy Fitzhardinge; xen-devel@lists.xensource.com
> > Subject: Re: [Xen-devel] pvops domu soft lockup under load (more
> logs)
> >
> > Oh,
> >
> > On Apr 16, 2010, at 9:37 , Pim van Riezen wrote:
> >
> > > Another datapoint. This customer has similarly loaded VPS machines
> on
> > a number of different hardware nodes. Not all of them had the lockup
> > problem. I applied the jiffies clocksource to all his machines,
> > regardless of their current problem status. After a day without
> > lockups, the customer complained about time drift (ntp was not
> > activated). The guest that had experienced the soft lockups earlier
> had
> > major clock drift and were way ahead:
> > >
> > > 16 Apr 09:29:26 ntpdate[11236]: step time server 194.109.22.18
> > offset -7337.731686 sec
> > >
> > > That's over 2 hours accumulated in less than 24 hours of uptime.
> The
> > guests that hadn't been excperiencing the lockup issues berfore
> > switching to the jiffies clocksource hadn't drifted that much after
> the
> > switch and were, at most, 120s behind after the same amount of
> runtime.
> >
> > There's more correlation between the guests that had the lockups and
> > those that didn't: the guests that locked up (and now have a way
> speedy
> > jiffies clock) were all on the same hardware platform, with an older
> > Xeon CPU than on the guests that had no issues. I attached cpuinfo
> for
> > both the broken and the non-broken dom0s. All are on Xen-3.4.1
> > (hypervisor-version doesn't seem to affect this issue) and the latest
> > CentOS 2.6.18 dom0-kernel.
> >
> > Cheers,
> > Pim
> >
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: pvops domu soft lockup under load (more logs)
2010-04-16 7:37 ` Pim van Riezen
2010-04-16 7:55 ` Pim van Riezen
[not found] ` <9D75997F-4855-41A0-B159-18B6A3BFC776@panelsix.com>
@ 2010-04-16 20:22 ` Jeremy Fitzhardinge
2 siblings, 0 replies; 17+ messages in thread
From: Jeremy Fitzhardinge @ 2010-04-16 20:22 UTC (permalink / raw)
To: Pim van Riezen; +Cc: xen-devel
On 04/16/2010 12:37 AM, Pim van Riezen wrote:
> Another datapoint. This customer has similarly loaded VPS machines on a number of different hardware nodes. Not all of them had the lockup problem. I applied the jiffies clocksource to all his machines, regardless of their current problem status.
Does the host hardware differ? Are they multisocket? Intel? AMD?
> After a day without lockups, the customer complained about time drift (ntp was not activated). The guest that had experienced the soft lockups earlier had major clock drift and were way ahead:
>
> 16 Apr 09:29:26 ntpdate[11236]: step time server 194.109.22.18 offset -7337.731686 sec
>
> That's over 2 hours accumulated in less than 24 hours of uptime. The guests that hadn't been excperiencing the lockup issues berfore switching to the jiffies clocksource hadn't drifted that much after the switch and were, at most, 120s behind after the same amount of runtime.
>
Jiffies is a horrible clocksource. Its accuracy is highly dependent on
the how busy the host is (not just a given guest).
J
^ permalink raw reply [flat|nested] 17+ messages in thread
end of thread, other threads:[~2010-04-16 20:22 UTC | newest]
Thread overview: 17+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2010-04-09 13:22 pvops domu soft lockup under load Pim van Riezen
2010-04-09 16:50 ` pvops domu soft lockup under load (more logs) Pim van Riezen
2010-04-09 18:43 ` Jeremy Fitzhardinge
2010-04-10 7:12 ` Pim van Riezen
2010-04-12 9:44 ` Pim van Riezen
2010-04-14 17:48 ` Jeremy Fitzhardinge
2010-04-15 10:56 ` Pim van Riezen
2010-04-15 17:21 ` Jeremy Fitzhardinge
2010-04-16 7:37 ` Pim van Riezen
2010-04-16 7:55 ` Pim van Riezen
2010-04-16 15:17 ` Dan Magenheimer
2010-04-16 16:00 ` Dan Magenheimer
2010-04-16 15:24 ` Pim van Riezen
[not found] ` <9D75997F-4855-41A0-B159-18B6A3BFC776@panelsix.com>
2010-04-16 13:25 ` Pim van Riezen
2010-04-16 20:22 ` Jeremy Fitzhardinge
2010-04-16 14:20 ` Pim van Riezen
2010-04-16 14:19 ` Pim van Riezen
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.