From mboxrd@z Thu Jan  1 00:00:00 1970
From: Pim van Riezen <pi+lists@panelsix.com>
Subject: Re: pvops domu soft lockup under load (more logs)
Date: Fri, 9 Apr 2010 18:50:30 +0200
Message-ID: <2F17645D-999B-435C-97EE-508D39B71035@panelsix.com>
References: <C6F9EF58-539C-4B05-ACE1-8C96056944E8@panelsix.com>
Mime-Version: 1.0 (Apple Message framework v1078)
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: quoted-printable
Return-path: <xen-devel-bounces@lists.xensource.com>
In-Reply-To: <C6F9EF58-539C-4B05-ACE1-8C96056944E8@panelsix.com>
List-Unsubscribe: <http://lists.xensource.com/mailman/listinfo/xen-devel>,
	<mailto:xen-devel-request@lists.xensource.com?subject=unsubscribe>
List-Post: <mailto:xen-devel@lists.xensource.com>
List-Help: <mailto:xen-devel-request@lists.xensource.com?subject=help>
List-Subscribe: <http://lists.xensource.com/mailman/listinfo/xen-devel>,
	<mailto:xen-devel-request@lists.xensource.com?subject=subscribe>
Sender: xen-devel-bounces@lists.xensource.com
Errors-To: xen-devel-bounces@lists.xensource.com
To: xen-devel@lists.xensource.com
List-Id: xen-devel@lists.xenproject.org

Some more logs of this bug in action. Stack seems to be all over the =
place.

> Apr  9 08:21:01 host kernel: BUG: soft lockup - CPU#1 stuck for 628s! =
[gzip:27658]
> Apr  9 08:21:01 host kernel: Modules linked in:
> Apr  9 08:21:01 host kernel: CPU 1:
> Apr  9 08:21:01 host kernel: Modules linked in:
> Apr  9 08:21:01 host kernel: Pid: 27658, comm: gzip Not tainted =
2.6.32.9xls-domU #2=20
> Apr  9 08:21:01 host kernel: RIP: e033:[<0000000000401958>]  =
[<0000000000401958>] 0x401958
> Apr  9 08:21:01 host kernel: RSP: e02b:00007fff026d7e38  EFLAGS: =
00000297
> Apr  9 08:21:01 host kernel: RAX: 0000000000000020 RBX: =
0000000000000002 RCX: 0000000000657ce8
> Apr  9 08:21:01 host kernel: RDX: 0000000000000048 RSI: =
0000000000000002 RDI: 000000000000da48
> Apr  9 08:21:01 host kernel: RBP: 0000000000006a6c R08: =
0000000000000034 R09: 000000000000da48
> Apr  9 08:21:01 host kernel: R10: 0000000000000048 R11: =
0000000000000080 R12: 0000000000658c06
> Apr  9 08:21:01 host kernel: R13: 0000000000658d08 R14: =
0000000000658c06 R15: 0000000000000080
> Apr  9 08:21:01 host kernel: FS:  00007fa6088946e0(0000) =
GS:ffff8800022c7000(0000) knlGS:0000000000000000
> Apr  9 08:21:01 host kernel: CS:  e033 DS: 0000 ES: 0000 CR0: =
000000008005003b
> Apr  9 08:21:01 host kernel: CR2: 00007f7addede020 CR3: =
00000000267db000 CR4: 0000000000002660
> Apr  9 08:21:01 host kernel: DR0: 0000000000000000 DR1: =
0000000000000000 DR2: 0000000000000000
> Apr  9 08:21:02 host kernel: DR3: 0000000000000000 DR6: =
00000000ffff0ff0 DR7: 0000000000000400
> Apr  9 08:21:02 host kernel: Call Trace:
> Apr  9 08:29:39 host kernel: BUG: soft lockup - CPU#1 stuck for 483s! =
[swapper:0]
> Apr  9 08:29:39 host kernel: Modules linked in:
> Apr  9 08:29:39 host kernel: CPU 1:
> Apr  9 08:29:39 host kernel: Modules linked in:
> Apr  9 08:29:39 host kernel: Pid: 0, comm: swapper Not tainted =
2.6.32.9xls-domU #2=20
> Apr  9 08:29:39 host kernel: RIP: e030:[<ffffffff810093aa>]  =
[<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
> Apr  9 08:29:39 host kernel: RSP: e02b:ffff88007f8a1f08  EFLAGS: =
00000246
> Apr  9 08:29:39 host kernel: RAX: 0000000000000000 RBX: =
ffff88007f8a0000 RCX: ffffffff810093aa
> Apr  9 08:29:39 host kernel: RDX: 0000000000000000 RSI: =
0000000000000000 RDI: 0000000000000001
> Apr  9 08:29:39 host kernel: RBP: ffffffff81896d30 R08: =
0000000000000000 R09: ffffffff8100e3b2
> Apr  9 08:29:39 host kernel: R10: 0000000000000001 R11: =
0000000000000246 R12: 0000000000000000
> Apr  9 08:29:39 host kernel: R13: 0000000000000000 R14: =
0000000000000000 R15: 0000000000000000
> Apr  9 08:29:39 host kernel: FS:  00007fa913f42720(0000) =
GS:ffff8800022c7000(0000) knlGS:0000000000000000
> Apr  9 08:29:39 host kernel: CS:  e033 DS: 002b ES: 002b CR0: =
000000008005003b
> Apr  9 08:29:39 host kernel: CR2: 00007ff39862b1f0 CR3: =
0000000052c5e000 CR4: 0000000000002660
> Apr  9 08:29:39 host kernel: DR0: 0000000000000000 DR1: =
0000000000000000 DR2: 0000000000000000
> Apr  9 08:29:39 host kernel: DR3: 0000000000000000 DR6: =
00000000ffff0ff0 DR7: 0000000000000400
> Apr  9 08:29:39 host kernel: Call Trace:
> Apr  9 08:29:39 host kernel:  [<ffffffff8100ddb7>] ? =
xen_safe_halt+0xc/0x15
> Apr  9 08:29:39 host kernel:  [<ffffffff8100bdcf>] ? =
xen_idle+0x37/0x40
> Apr  9 08:29:39 host kernel:  [<ffffffff8100fe2e>] ? =
cpu_idle+0x4f/0x82
> Apr  9 08:29:39 host kernel: BUG: soft lockup - CPU#0 stuck for 483s! =
[swapper:0]
> Apr  9 08:29:39 host kernel: Modules linked in:
> Apr  9 08:29:39 host kernel: CPU 0:
> Apr  9 08:29:40 host kernel: Modules linked in:
> Apr  9 08:29:40 host kernel: Pid: 0, comm: swapper Not tainted =
2.6.32.9xls-domU #2=20
> Apr  9 08:29:40 host kernel: RIP: e030:[<ffffffff810093aa>]  =
[<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
> Apr  9 08:29:40 host kernel: RSP: e02b:ffffffff81691f70  EFLAGS: =
00000246
> Apr  9 08:29:40 host kernel: RAX: 0000000000000000 RBX: =
ffffffff81690000 RCX: ffffffff810093aa
> Apr  9 08:29:40 host kernel: RDX: 0000000000000000 RSI: =
0000000000000000 RDI: 0000000000000001
> Apr  9 08:29:40 host kernel: RBP: ffffffff81896d30 R08: =
0000000000000000 R09: ffffffff8100e3b2
> Apr  9 08:29:40 host kernel: R10: 0000000000000001 R11: =
0000000000000246 R12: ffffffffffffffff
> Apr  9 08:29:40 host kernel: R13: ffffffff818ebf20 R14: =
ffffffff818eec70 R15: 0000000000000000
> Apr  9 08:29:40 host kernel: FS:  00007f07704fe6e0(0000) =
GS:ffff8800022ac000(0000) knlGS:0000000000000000
> Apr  9 08:29:40 host kernel: CS:  e033 DS: 0000 ES: 0000 CR0: =
000000008005003b
> Apr  9 08:29:40 host kernel: CR2: 00007ff39e4a8000 CR3: =
000000007ea99000 CR4: 0000000000002660
> Apr  9 08:29:40 host kernel: DR0: 0000000000000000 DR1: =
0000000000000000 DR2: 0000000000000000
> Apr  9 08:29:40 host kernel: DR3: 0000000000000000 DR6: =
00000000ffff0ff0 DR7: 0000000000000400
> Apr  9 08:29:40 host kernel: Call Trace:
> Apr  9 08:29:40 host kernel:  [<ffffffff8100ddb7>] ? =
xen_safe_halt+0xc/0x15
> Apr  9 08:29:40 host kernel:  [<ffffffff8100bdcf>] ? =
xen_idle+0x37/0x40
> Apr  9 08:29:40 host kernel:  [<ffffffff8100fe2e>] ? =
cpu_idle+0x4f/0x82
> Apr  9 08:29:40 host kernel:  [<ffffffff818b6c42>] ? =
start_kernel+0x353/0x35f
> Apr  9 08:34:34 host kernel: BUG: soft lockup - CPU#1 stuck for 275s! =
[ssh:2324]
> Apr  9 08:34:34 host kernel: Modules linked in:
> Apr  9 08:34:34 host kernel: CPU 1:
> Apr  9 08:34:34 host kernel: Modules linked in:
> Apr  9 08:34:34 host kernel: Pid: 2324, comm: ssh Not tainted =
2.6.32.9xls-domU #2=20
> Apr  9 08:34:34 host kernel: RIP: e033:[<00007f1b0fcac76a>]  =
[<00007f1b0fcac76a>] 0x7f1b0fcac76a
> Apr  9 08:34:34 host kernel: RSP: e02b:00007fffa45a7480  EFLAGS: =
00010202
> Apr  9 08:34:34 host kernel: RAX: 0000000000000000 RBX: =
0000000000000000 RCX: 0000000000000181
> Apr  9 08:34:34 host kernel: RDX: 00007f1b0fcdca28 RSI: =
0000000000000013 RDI: 000000000000000a
> Apr  9 08:34:34 host kernel: RBP: 0000000000000010 R08: =
0000000000000004 R09: 0101010101010101
> Apr  9 08:34:34 host kernel: R10: 737361702e707466 R11: =
00007f1b0fcac690 R12: 0000000000000010
> Apr  9 08:34:34 host kernel: R13: 00007f1b11f825e0 R14: =
0000000000000001 R15: 0000000000000004
> Apr  9 08:34:34 host kernel: FS:  00007f1b10528710(0000) =
GS:ffff8800022c7000(0000) knlGS:0000000000000000
> Apr  9 08:34:34 host kernel: CS:  e033 DS: 0000 ES: 0000 CR0: =
000000008005003b
> Apr  9 08:34:34 host kernel: CR2: 00007fa913f54000 CR3: =
000000006ad44000 CR4: 0000000000002660
> Apr  9 08:34:34 host kernel: DR0: 0000000000000000 DR1: =
0000000000000000 DR2: 0000000000000000
> Apr  9 08:34:34 host kernel: DR3: 0000000000000000 DR6: =
00000000ffff0ff0 DR7: 0000000000000400
> Apr  9 08:34:34 host kernel: Call Trace:
> Apr  9 08:34:34 host kernel: BUG: soft lockup - CPU#0 stuck for 275s! =
[swapper:0]
> Apr  9 08:34:34 host kernel: Modules linked in:
> Apr  9 08:34:34 host kernel: CPU 0:
> Apr  9 08:34:34 host kernel: Modules linked in:
> Apr  9 08:34:34 host kernel: Pid: 0, comm: swapper Not tainted =
2.6.32.9xls-domU #2=20
> Apr  9 08:34:34 host kernel: RIP: e030:[<ffffffff810093aa>]  =
[<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
> Apr  9 08:34:34 host kernel: RSP: e02b:ffffffff81691f70  EFLAGS: =
00000246
> Apr  9 08:34:34 host kernel: RAX: 0000000000000000 RBX: =
ffffffff81690000 RCX: ffffffff810093aa
> Apr  9 08:34:34 host kernel: RDX: 0000000000000000 RSI: =
0000000000000000 RDI: 0000000000000001
> Apr  9 08:34:34 host kernel: RBP: ffffffff81896d30 R08: =
0000000000000000 R09: ffffffff8100e3b2
> Apr  9 08:34:34 host kernel: R10: 0000000000000001 R11: =
0000000000000246 R12: ffffffffffffffff
> Apr  9 08:34:34 host kernel: R13: ffffffff818ebf20 R14: =
ffffffff818eec70 R15: 0000000000000000
> Apr  9 08:34:34 host kernel: FS:  00007fa913f42720(0000) =
GS:ffff8800022ac000(0000) knlGS:0000000000000000
> Apr  9 08:34:34 host kernel: CS:  e033 DS: 0000 ES: 0000 CR0: =
000000008005003b
> Apr  9 08:34:34 host kernel: CR2: 00007fa913f54000 CR3: =
000000006acb1000 CR4: 0000000000002660
> Apr  9 08:34:34 host kernel: DR0: 0000000000000000 DR1: =
0000000000000000 DR2: 0000000000000000
> Apr  9 08:34:34 host kernel: DR3: 0000000000000000 DR6: =
00000000ffff0ff0 DR7: 0000000000000400
> Apr  9 08:34:34 host kernel: Call Trace:
> Apr  9 08:34:34 host kernel:  [<ffffffff8100ddb7>] ? =
xen_safe_halt+0xc/0x15
> Apr  9 08:34:34 host kernel:  [<ffffffff8100bdcf>] ? =
xen_idle+0x37/0x40
> Apr  9 08:34:34 host kernel:  [<ffffffff8100fe2e>] ? =
cpu_idle+0x4f/0x82
> Apr  9 08:34:34 host kernel:  [<ffffffff818b6c42>] ? =
start_kernel+0x353/0x35f
> Apr  9 08:39:49 host kernel: BUG: soft lockup - CPU#1 stuck for 293s! =
[gzip:30581]
> Apr  9 08:39:49 host kernel: Modules linked in:
> Apr  9 08:39:49 host kernel: CPU 1:
> Apr  9 08:39:49 host kernel: Modules linked in:
> Apr  9 08:39:49 host kernel: Pid: 30581, comm: gzip Not tainted =
2.6.32.9xls-domU #2=20
> Apr  9 08:39:49 host kernel: RIP: e030:[<ffffffff811ede38>]  =
[<ffffffff811ede38>] fuse_request_send+0x60/0x22b
> Apr  9 08:39:49 host kernel: RSP: e02b:ffff88007952bc28  EFLAGS: =
00000202
> Apr  9 08:39:49 host kernel: RAX: ffff880079652df0 RBX: =
ffff88007e07d2c0 RCX: fffffffffffdfb5e
> Apr  9 08:39:49 host kernel: RDX: 0000000000020001 RSI: =
000000000000001d RDI: ffff88007eb1f710
> Apr  9 08:39:49 host kernel: RBP: ffff88007e07d2c0 R08: =
000000037ede1c40 R09: ffff8800593170d8
> Apr  9 08:39:49 host kernel: R10: 0000000000000000 R11: =
000000000000000f R12: ffff88007eb1f400
> Apr  9 08:39:49 host kernel: R13: ffff88007e07d2e0 R14: =
0000000000001000 R15: 0000000000001000
> Apr  9 08:39:49 host kernel: FS:  00007f7b12f606e0(0000) =
GS:ffff8800022c7000(0000) knlGS:0000000000000000
> Apr  9 08:39:49 host kernel: CS:  e033 DS: 0000 ES: 0000 CR0: =
000000008005003b
> Apr  9 08:39:49 host kernel: CR2: 00000000006d9138 CR3: =
00000000264f7000 CR4: 0000000000002660
> Apr  9 08:39:49 host kernel: DR0: 0000000000000000 DR1: =
0000000000000000 DR2: 0000000000000000
> Apr  9 08:39:49 host kernel: DR3: 0000000000000000 DR6: =
00000000ffff0ff0 DR7: 0000000000000400
> Apr  9 08:39:49 host kernel: Call Trace:
> Apr  9 08:39:49 host kernel:  [<ffffffff811ede34>] ? =
fuse_request_send+0x5c/0x22b
> Apr  9 08:39:49 host kernel:  [<ffffffff811f1655>] ? =
fuse_wait_on_page_writeback+0x31/0x168
> Apr  9 08:39:49 host kernel:  [<ffffffff810904b4>] ? =
add_to_page_cache_lru+0x41/0x58
> Apr  9 08:39:49 host kernel:  [<ffffffff811f186a>] ? =
fuse_send_write+0xb8/0xc2
> Apr  9 08:39:49 host kernel:  [<ffffffff811f29d7>] ? =
fuse_file_aio_write+0x497/0x62f
> Apr  9 08:39:49 host kernel:  [<ffffffff810bab7e>] ? =
do_sync_write+0xce/0x113
> Apr  9 08:39:49 host kernel:  [<ffffffff8100f6af>] ? =
__switch_to+0x1ad/0x297
> Apr  9 08:39:49 host kernel:  [<ffffffff8100e3b2>] ? =
check_events+0x12/0x20
> Apr  9 08:39:49 host kernel:  [<ffffffff810561b8>] ? =
autoremove_wake_function+0x0/0x2e
> Apr  9 08:39:49 host kernel:  [<ffffffff8103b070>] ? =
finish_task_switch+0x27/0x6e
> Apr  9 08:39:49 host kernel:  [<ffffffff814d6a77>] ? =
thread_return+0x3e/0x109
> Apr  9 08:39:49 host kernel:  [<ffffffff810bb4b2>] ? =
vfs_write+0xad/0x149
> Apr  9 08:39:49 host kernel:  [<ffffffff810bb60a>] ? =
sys_write+0x45/0x6e
> Apr  9 08:39:49 host kernel:  [<ffffffff81010ac2>] ? =
system_call_fastpath+0x16/0x1b
> Apr  9 08:39:49 host kernel: BUG: soft lockup - CPU#0 stuck for 293s! =
[sshfs:2332]
> Apr  9 08:39:49 host kernel: Modules linked in:
> Apr  9 08:39:49 host kernel: CPU 0:
> Apr  9 08:39:49 host kernel: Modules linked in:
> Apr  9 08:39:49 host kernel: Pid: 2332, comm: sshfs Not tainted =
2.6.32.9xls-domU #2=20
> Apr  9 08:39:49 host kernel: RIP: e030:[<ffffffff813b93bd>]  =
[<ffffffff813b93bd>] sk_reset_timer+0x1/0x1d
> Apr  9 08:39:49 host kernel: RSP: e02b:ffff8800022afbc0  EFLAGS: =
00000206
> Apr  9 08:39:49 host kernel: RAX: 0000000101b312c6 RBX: =
0000000000000080 RCX: 0000000101b312ce
> Apr  9 08:39:49 host kernel: RDX: 0000000101b312ce RSI: =
ffff88007d1dc4d0 RDI: ffff88007d1dc1c0
> Apr  9 08:39:49 host kernel: RBP: ffff88007d1dc1c0 R08: =
000000007d1dc10f R09: 0000000101b312ce
> Apr  9 08:39:49 host kernel: R10: ffff880068b8f1c0 R11: =
ffffffff8126167a R12: ffff880068b8f1c0
> Apr  9 08:39:49 host kernel: R13: 0000000000000000 R14: =
0000000000000020 R15: ffff880068b8f1f8
> Apr  9 08:39:49 host kernel: FS:  00007f1b10528710(0063) =
GS:ffff8800022ac000(0000) knlGS:0000000000000000
> Apr  9 08:39:49 host kernel: CS:  e033 DS: 0000 ES: 0000 CR0: =
000000008005003b
> Apr  9 08:39:49 host kernel: CR2: 00007f66a4ede000 CR3: =
000000006acb1000 CR4: 0000000000002660
> Apr  9 08:39:49 host kernel: DR0: 0000000000000000 DR1: =
0000000000000000 DR2: 0000000000000000
> Apr  9 08:39:49 host kernel: DR3: 0000000000000000 DR6: =
00000000ffff0ff0 DR7: 0000000000000400
> Apr  9 08:39:49 host kernel: Call Trace:
> Apr  9 08:39:49 host kernel:  <IRQ>  [<ffffffff81419140>] ? =
tcp_rcv_established+0x587/0x87b
> Apr  9 08:39:49 host kernel:  [<ffffffff8141f7f1>] ? =
tcp_v4_do_rcv+0x2c/0x1c9
> Apr  9 08:39:49 host kernel:  [<ffffffff81046156>] ? =
local_bh_enable+0x22/0x8f
> Apr  9 08:39:49 host kernel:  [<ffffffff8141fe05>] ? =
tcp_v4_rcv+0x477/0x6f8
> Apr  9 08:39:49 host kernel:  [<ffffffff814064f7>] ? =
ip_local_deliver_finish+0xf2/0x1c7
> Apr  9 08:39:49 host kernel:  [<ffffffff81406126>] ? =
ip_rcv_finish+0x326/0x340
> Apr  9 08:39:49 host kernel:  [<ffffffff814063c0>] ? =
ip_rcv+0x280/0x2c5
> Apr  9 08:39:49 host kernel:  [<ffffffff8135d39d>] ? =
xennet_poll+0xb06/0xc68
> Apr  9 08:39:49 host kernel:  [<ffffffff8100e39f>] ? =
xen_restore_fl_direct_end+0x0/0x1
> Apr  9 08:39:49 host kernel:  [<ffffffff81334afc>] ? =
do_blkif_request+0x341/0x350
> Apr  9 08:39:49 host kernel:  [<ffffffff8100e3b2>] ? =
check_events+0x12/0x20
> Apr  9 08:39:49 host kernel:  [<ffffffff813c469b>] ? =
net_rx_action+0x77/0x179
> Apr  9 08:39:49 host kernel:  [<ffffffff810462b4>] ? =
__do_softirq+0xdd/0x1a1
> Apr  9 08:39:49 host kernel:  [<ffffffff81011bec>] ? =
call_softirq+0x1c/0x30
> Apr  9 08:39:49 host kernel:  [<ffffffff81012edf>] ? =
do_softirq+0x3f/0x7c
> Apr  9 08:39:49 host kernel:  [<ffffffff812ff975>] ? =
xen_evtchn_do_upcall+0x173/0x190
> Apr  9 08:39:49 host kernel:  [<ffffffff81011c3e>] ? =
xen_do_hypervisor_callback+0x1e/0x30
> Apr  9 08:39:49 host kernel:  <EOI>  [<ffffffff8100e3b2>] ? =
check_events+0x12/0x20
> Apr  9 08:39:49 host kernel:  [<ffffffff8100922a>] ? =
hypercall_page+0x22a/0x1001
> Apr  9 08:39:49 host kernel:  [<ffffffff8100922a>] ? =
hypercall_page+0x22a/0x1001
> Apr  9 08:39:49 host kernel:  [<ffffffff8100dd39>] ? =
xen_force_evtchn_callback+0x9/0xa
> Apr  9 08:39:49 host kernel:  [<ffffffff8100e3b2>] ? =
check_events+0x12/0x20
> Apr  9 08:39:49 host kernel:  [<ffffffff8100e3b2>] ? =
check_events+0x12/0x20
> Apr  9 08:39:49 host kernel:  [<ffffffff8100e359>] ? =
xen_irq_enable_direct_end+0x0/0x7
> Apr  9 08:39:49 host kernel:  [<ffffffff8103b070>] ? =
finish_task_switch+0x27/0x6e
> Apr  9 08:39:49 host kernel:  [<ffffffff814d6a77>] ? =
thread_return+0x3e/0x109
> Apr  9 08:39:49 host kernel:  [<ffffffff8103d336>] ? =
default_wake_function+0x0/0x9
> Apr  9 08:39:49 host kernel:  [<ffffffff81261668>] ? =
cap_socket_recvmsg+0x0/0x3
> Apr  9 08:39:49 host kernel:  [<ffffffff81261668>] ? =
cap_socket_recvmsg+0x0/0x3
> Apr  9 08:39:49 host kernel:  [<ffffffff814d6e59>] ? =
schedule_timeout+0x23/0x1d9
> Apr  9 08:39:49 host kernel:  [<ffffffff8100dd39>] ? =
xen_force_evtchn_callback+0x9/0xa
> Apr  9 08:39:49 host kernel:  [<ffffffff8100e3b2>] ? =
check_events+0x12/0x20
> Apr  9 08:39:49 host kernel:  [<ffffffff81261668>] ? =
cap_socket_recvmsg+0x0/0x3
> Apr  9 08:39:49 host kernel:  [<ffffffff814d824c>] ? =
_spin_unlock_irqrestore+0xc/0xd
> Apr  9 08:39:49 host kernel:  [<ffffffff81456ebe>] ? =
unix_stream_recvmsg+0x2c4/0x644
> Apr  9 08:39:49 host kernel:  [<ffffffff810561b8>] ? =
autoremove_wake_function+0x0/0x2e
> Apr  9 08:39:49 host kernel:  [<ffffffff81457231>] ? =
unix_stream_recvmsg+0x637/0x644
> Apr  9 08:39:49 host kernel:  [<ffffffff813b6303>] ? =
sock_aio_read+0x103/0x117
> Apr  9 08:39:49 host kernel:  [<ffffffff810bac91>] ? =
do_sync_read+0xce/0x113
> Apr  9 08:39:49 host kernel:  [<ffffffff81011c1b>] ? =
xen_hypervisor_callback+0x1b/0x20
> Apr  9 08:39:49 host kernel:  [<ffffffff810561b8>] ? =
autoremove_wake_function+0x0/0x2e
> Apr  9 08:39:49 host kernel:  [<ffffffff81261668>] ? =
cap_socket_recvmsg+0x0/0x3
> Apr  9 08:39:49 host kernel:  [<ffffffff814d824c>] ? =
_spin_unlock_irqrestore+0xc/0xd
> Apr  9 08:39:49 host kernel:  [<ffffffff81011c1b>] ? =
xen_hypervisor_callback+0x1b/0x20
> Apr  9 08:39:49 host kernel:  [<ffffffff810bb6f0>] ? =
vfs_read+0xbd/0x146
> Apr  9 08:39:49 host kernel:  [<ffffffff8100922a>] ? =
hypercall_page+0x22a/0x1001
> Apr  9 08:39:49 host kernel:  [<ffffffff810bb835>] ? =
sys_read+0x45/0x6e
> Apr  9 08:39:49 host kernel:  [<ffffffff81010ac2>] ? =
system_call_fastpath+0x16/0x1b
> Apr  9 09:11:36 host kernel: BUG: soft lockup - CPU#1 stuck for 1776s! =
[swapper:0]
> Apr  9 09:11:36 host kernel: Modules linked in:
> Apr  9 09:11:36 host kernel: CPU 1:
> Apr  9 09:11:36 host kernel: Modules linked in:
> Apr  9 09:11:36 host kernel: Pid: 0, comm: swapper Not tainted =
2.6.32.9xls-domU #2=20
> Apr  9 09:11:36 host kernel: RIP: e030:[<ffffffff810093aa>]  =
[<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
> Apr  9 09:11:36 host kernel: RSP: e02b:ffff88007f8a1f08  EFLAGS: =
00000246
> Apr  9 09:11:36 host kernel: RAX: 0000000000000000 RBX: =
ffff88007f8a0000 RCX: ffffffff810093aa
> Apr  9 09:11:36 host kernel: RDX: 0000000000000000 RSI: =
0000000000000000 RDI: 0000000000000001
> Apr  9 09:11:36 host kernel: RBP: ffffffff81896d30 R08: =
0000000000000000 R09: ffffffff8100e3b2
> Apr  9 09:11:36 host kernel: R10: 0000000000000001 R11: =
0000000000000246 R12: 0000000000000000
> Apr  9 09:11:36 host kernel: R13: 0000000000000000 R14: =
0000000000000000 R15: 0000000000000000
> Apr  9 09:11:36 host kernel: FS:  00007f44bc51f710(0000) =
GS:ffff8800022c7000(0000) knlGS:0000000000000000
> Apr  9 09:11:36 host kernel: CS:  e033 DS: 002b ES: 002b CR0: =
000000008005003b
> Apr  9 09:11:36 host kernel: CR2: 00007f44bc52c000 CR3: =
0000000071c86000 CR4: 0000000000002660
> Apr  9 09:11:36 host kernel: DR0: 0000000000000000 DR1: =
0000000000000000 DR2: 0000000000000000
> Apr  9 09:11:36 host kernel: DR3: 0000000000000000 DR6: =
00000000ffff0ff0 DR7: 0000000000000400
> Apr  9 09:11:36 host kernel: Call Trace:
> Apr  9 09:11:36 host kernel:  [<ffffffff8100ddb7>] ? =
xen_safe_halt+0xc/0x15
> Apr  9 09:11:36 host kernel:  [<ffffffff8100bdcf>] ? =
xen_idle+0x37/0x40
> Apr  9 09:11:36 host kernel:  [<ffffffff8100fe2e>] ? =
cpu_idle+0x4f/0x82
> Apr  9 09:11:36 host kernel: BUG: soft lockup - CPU#0 stuck for 1776s! =
[swapper:0]
> Apr  9 09:11:36 host kernel: Modules linked in:
> Apr  9 09:11:36 host kernel: CPU 0:
> Apr  9 09:11:36 host kernel: Modules linked in:
> Apr  9 09:11:36 host kernel: Pid: 0, comm: swapper Not tainted =
2.6.32.9xls-domU #2=20
> Apr  9 09:11:36 host kernel: RIP: e030:[<ffffffff810093aa>]  =
[<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
> Apr  9 09:11:36 host kernel: RSP: e02b:ffffffff81691f70  EFLAGS: =
00000246
> Apr  9 09:11:36 host kernel: RAX: 0000000000000000 RBX: =
ffffffff81690000 RCX: ffffffff810093aa
> Apr  9 09:11:36 host kernel: RDX: 0000000000000000 RSI: =
0000000000000000 RDI: 0000000000000001
> Apr  9 09:11:36 host kernel: RBP: ffffffff81896d30 R08: =
0000000000000000 R09: ffffffff8100e3b2
> Apr  9 09:11:36 host kernel: R10: 0000000000000001 R11: =
0000000000000246 R12: ffffffffffffffff
> Apr  9 09:11:36 host kernel: R13: ffffffff818ebf20 R14: =
ffffffff818eec70 R15: 0000000000000000
> Apr  9 09:11:36 host kernel: FS:  00007f07704fe6e0(0000) =
GS:ffff8800022ac000(0000) knlGS:0000000000000000
> Apr  9 09:11:36 host kernel: CS:  e033 DS: 0000 ES: 0000 CR0: =
000000008005003b
> Apr  9 09:11:36 host kernel: CR2: 00000000004884b2 CR3: =
0000000071c86000 CR4: 0000000000002660
> Apr  9 09:11:36 host kernel: DR0: 0000000000000000 DR1: =
0000000000000000 DR2: 0000000000000000
> Apr  9 09:11:36 host kernel: DR3: 0000000000000000 DR6: =
00000000ffff0ff0 DR7: 0000000000000400
> Apr  9 09:11:36 host kernel: Call Trace:
> Apr  9 09:11:36 host kernel:  [<ffffffff8100ddb7>] ? =
xen_safe_halt+0xc/0x15
> Apr  9 09:11:36 host kernel:  [<ffffffff8100bdcf>] ? =
xen_idle+0x37/0x40
> Apr  9 09:11:36 host kernel:  [<ffffffff8100fe2e>] ? =
cpu_idle+0x4f/0x82
> Apr  9 09:11:36 host kernel:  [<ffffffff818b6c42>] ? =
start_kernel+0x353/0x35f
> Apr  9 09:20:24 host kernel: BUG: soft lockup - CPU#1 stuck for 492s! =
[ssh:2324]
> Apr  9 09:20:24 host kernel: Modules linked in:
> Apr  9 09:20:24 host kernel: CPU 1:
> Apr  9 09:20:24 host kernel: Modules linked in:
> Apr  9 09:20:24 host kernel: Pid: 2324, comm: ssh Not tainted =
2.6.32.9xls-domU #2=20
> Apr  9 09:20:24 host kernel: RIP: e033:[<00007f1b0e259f20>]  =
[<00007f1b0e259f20>] 0x7f1b0e259f20
> Apr  9 09:20:24 host kernel: RSP: e02b:00007fffa45a7338  EFLAGS: =
00000206
> Apr  9 09:20:24 host kernel: RAX: 00007f1b11f32b20 RBX: =
00007f1b0e5399c0 RCX: 00000000000000a0
> Apr  9 09:20:24 host kernel: RDX: 00007f1b11f32b80 RSI: =
0000000000000004 RDI: 00007f1b0e539c30
> Apr  9 09:20:24 host kernel: RBP: 0000000000000064 R08: =
0000000000000001 R09: 0000000000000001
> Apr  9 09:20:24 host kernel: R10: 00007f1b0e539a30 R11: =
00007f1b0e25bfb0 R12: 0000000000000000
> Apr  9 09:20:24 host kernel: R13: 00007f1b0fcd2aec R14: =
0000000000000111 R15: 0000000000000071
> Apr  9 09:20:24 host kernel: FS:  00007f1b10528710(0000) =
GS:ffff8800022c7000(0000) knlGS:0000000000000000
> Apr  9 09:20:24 host kernel: CS:  e033 DS: 0000 ES: 0000 CR0: =
000000008005003b
> Apr  9 09:20:24 host kernel: CR2: 00007fa913f53000 CR3: =
000000006ad44000 CR4: 0000000000002660
> Apr  9 09:20:24 host kernel: DR0: 0000000000000000 DR1: =
0000000000000000 DR2: 0000000000000000
> Apr  9 09:20:24 host kernel: DR3: 0000000000000000 DR6: =
00000000ffff0ff0 DR7: 0000000000000400
> Apr  9 09:20:24 host kernel: Call Trace:
> Apr  9 09:20:24 host kernel: BUG: soft lockup - CPU#0 stuck for 492s! =
[swapper:0]
> Apr  9 09:20:24 host kernel: Modules linked in:
> Apr  9 09:20:24 host kernel: CPU 0:
> Apr  9 09:20:24 host kernel: Modules linked in:
> Apr  9 09:20:24 host kernel: Pid: 0, comm: swapper Not tainted =
2.6.32.9xls-domU #2=20
> Apr  9 09:20:24 host kernel: RIP: e030:[<ffffffff810093aa>]  =
[<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
> Apr  9 09:20:24 host kernel: RSP: e02b:ffffffff81691f70  EFLAGS: =
00000246
> Apr  9 09:20:24 host kernel: RAX: 0000000000000000 RBX: =
ffffffff81690000 RCX: ffffffff810093aa
> Apr  9 09:20:24 host kernel: RDX: 0000000000000000 RSI: =
0000000000000000 RDI: 0000000000000001
> Apr  9 09:20:24 host kernel: RBP: ffffffff81896d30 R08: =
0000000000000000 R09: ffffffff8100e3b2
> Apr  9 09:20:24 host kernel: R10: 0000000000000001 R11: =
0000000000000246 R12: ffffffffffffffff
> Apr  9 09:20:24 host kernel: R13: ffffffff818ebf20 R14: =
ffffffff818eec70 R15: 0000000000000000
> Apr  9 09:20:24 host kernel: FS:  00007ff39e495710(0000) =
GS:ffff8800022ac000(0000) knlGS:0000000000000000
> Apr  9 09:20:24 host kernel: CS:  e033 DS: 0000 ES: 0000 CR0: =
000000008005003b
> Apr  9 09:20:24 host kernel: CR2: 00007fff892a1e30 CR3: =
000000006acb1000 CR4: 0000000000002660
> Apr  9 09:20:24 host kernel: DR0: 0000000000000000 DR1: =
0000000000000000 DR2: 0000000000000000
> Apr  9 09:20:24 host kernel: DR3: 0000000000000000 DR6: =
00000000ffff0ff0 DR7: 0000000000000400
> Apr  9 09:20:24 host kernel: Call Trace:
> Apr  9 09:20:24 host kernel:  [<ffffffff8100ddb7>] ? =
xen_safe_halt+0xc/0x15
> Apr  9 09:20:24 host kernel:  [<ffffffff8100bdcf>] ? =
xen_idle+0x37/0x40
> Apr  9 09:20:24 host kernel:  [<ffffffff8100fe2e>] ? =
cpu_idle+0x4f/0x82
> Apr  9 09:20:24 host kernel:  [<ffffffff818b6c42>] ? =
start_kernel+0x353/0x35f
> Apr  9 09:24:36 host kernel: BUG: soft lockup - CPU#1 stuck for 234s! =
[gzip:4496]
> Apr  9 09:24:36 host kernel: Modules linked in:
> Apr  9 09:24:36 host kernel: CPU 1:
> Apr  9 09:24:36 host kernel: Modules linked in:
> Apr  9 09:24:36 host kernel: Pid: 4496, comm: gzip Not tainted =
2.6.32.9xls-domU #2=20
> Apr  9 09:24:36 host kernel: RIP: e033:[<0000000000409804>]  =
[<0000000000409804>] 0x409804
> Apr  9 09:24:36 host kernel: RSP: e02b:00007fff87e009f8  EFLAGS: =
00000202
> Apr  9 09:24:36 host kernel: RAX: 00000000f6019e6d RBX: =
0000000000008000 RCX: 00007f529598c610
> Apr  9 09:24:36 host kernel: RDX: 0000000000f6019e RSI: =
00000000000045a6 RDI: 00000000006534fb
> Apr  9 09:24:36 host kernel: RBP: 0000000000005800 R08: =
0000000000004000 R09: 7464697720223333
> Apr  9 09:24:36 host kernel: R10: 6c61202236223d68 R11: =
0000000000000246 R12: 00000000006522a0
> Apr  9 09:24:36 host kernel: R13: 0000000000000000 R14: =
0000000000000000 R15: 0000000000000000
> Apr  9 09:24:36 host kernel: FS:  00007f5295e2a6e0(0000) =
GS:ffff8800022c7000(0000) knlGS:0000000000000000
> Apr  9 09:24:36 host kernel: CS:  e033 DS: 0000 ES: 0000 CR0: =
000000008005003b
> Apr  9 09:24:36 host kernel: CR2: 0000000000d550c8 CR3: =
0000000026474000 CR4: 0000000000002660
> Apr  9 09:24:37 host kernel: DR0: 0000000000000000 DR1: =
0000000000000000 DR2: 0000000000000000
> Apr  9 09:24:37 host kernel: DR3: 0000000000000000 DR6: =
00000000ffff0ff0 DR7: 0000000000000400
> Apr  9 09:24:37 host kernel: Call Trace:
> Apr  9 09:24:37 host kernel: BUG: soft lockup - CPU#0 stuck for 234s! =
[swapper:0]
> Apr  9 09:24:37 host kernel: Modules linked in:
> Apr  9 09:24:37 host kernel: CPU 0:
> Apr  9 09:24:37 host kernel: Modules linked in:
> Apr  9 09:24:37 host kernel: Pid: 0, comm: swapper Not tainted =
2.6.32.9xls-domU #2=20
> Apr  9 09:24:37 host kernel: RIP: e030:[<ffffffff810093aa>]  =
[<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
> Apr  9 09:24:38 host kernel: RSP: e02b:ffffffff81691f70  EFLAGS: =
00000246
> Apr  9 09:24:38 host kernel: RAX: 0000000000000000 RBX: =
ffffffff81690000 RCX: ffffffff810093aa
> Apr  9 09:24:38 host kernel: RDX: 0000000000000000 RSI: =
0000000000000000 RDI: 0000000000000001
> Apr  9 09:24:38 host kernel: RBP: ffffffff81896d30 R08: =
0000000000000000 R09: ffffffff8100e3b2
> Apr  9 09:24:38 host kernel: R10: 0000000000000001 R11: =
0000000000000246 R12: ffffffffffffffff
> Apr  9 09:24:38 host kernel: R13: ffffffff818ebf20 R14: =
ffffffff818eec70 R15: 0000000000000000
> Apr  9 09:24:38 host kernel: FS:  00007f2f6a31a710(0000) =
GS:ffff8800022ac000(0000) knlGS:0000000000000000
> Apr  9 09:24:38 host kernel: CS:  e033 DS: 0000 ES: 0000 CR0: =
000000008005003b
> Apr  9 09:24:38 host kernel: CR2: 00007f2f6a327000 CR3: =
000000003f62e000 CR4: 0000000000002660
> Apr  9 09:24:38 host kernel: DR0: 0000000000000000 DR1: =
0000000000000000 DR2: 0000000000000000
> Apr  9 09:24:38 host kernel: DR3: 0000000000000000 DR6: =
00000000ffff0ff0 DR7: 0000000000000400
> Apr  9 09:24:38 host kernel: Call Trace:
> Apr  9 09:24:38 host kernel:  [<ffffffff8100ddb7>] ? =
xen_safe_halt+0xc/0x15
> Apr  9 09:24:38 host kernel:  [<ffffffff8100bdcf>] ? =
xen_idle+0x37/0x40
> Apr  9 09:24:38 host kernel:  [<ffffffff8100fe2e>] ? =
cpu_idle+0x4f/0x82
> Apr  9 09:24:38 host kernel:  [<ffffffff818b6c42>] ? =
start_kernel+0x353/0x35f
> Apr  9 10:05:51 host kernel: BUG: soft lockup - CPU#1 stuck for 2305s! =
[swapper:0]
> Apr  9 10:05:51 host kernel: Modules linked in:
> Apr  9 10:05:51 host kernel: CPU 1:
> Apr  9 10:05:51 host kernel: Modules linked in:
> Apr  9 10:05:51 host kernel: Pid: 0, comm: swapper Not tainted =
2.6.32.9xls-domU #2=20
> Apr  9 10:05:51 host kernel: RIP: e030:[<ffffffff810093aa>]  =
[<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
> Apr  9 10:05:51 host kernel: RSP: e02b:ffff88007f8a1f08  EFLAGS: =
00000246
> Apr  9 10:05:51 host kernel: RAX: 0000000000000000 RBX: =
ffff88007f8a0000 RCX: ffffffff810093aa
> Apr  9 10:05:51 host kernel: RDX: 0000000000000000 RSI: =
0000000000000000 RDI: 0000000000000001
> Apr  9 10:05:51 host kernel: RBP: ffffffff81896d30 R08: =
0000000000000000 R09: ffffffff8100e3b2
> Apr  9 10:05:51 host kernel: R10: 0000000000000001 R11: =
0000000000000246 R12: 0000000000000000
> Apr  9 10:05:51 host kernel: R13: 0000000000000000 R14: =
0000000000000000 R15: 0000000000000000
> Apr  9 10:05:51 host kernel: FS:  00007ff39e495710(0000) =
GS:ffff8800022c7000(0000) knlGS:0000000000000000
> Apr  9 10:05:51 host kernel: CS:  e033 DS: 002b ES: 002b CR0: =
000000008005003b
> Apr  9 10:05:51 host kernel: CR2: 0000000004786158 CR3: =
000000007e2a6000 CR4: 0000000000002660
> Apr  9 10:05:51 host kernel: DR0: 0000000000000000 DR1: =
0000000000000000 DR2: 0000000000000000
> Apr  9 10:05:51 host kernel: DR3: 0000000000000000 DR6: =
00000000ffff0ff0 DR7: 0000000000000400
> Apr  9 10:05:51 host kernel: Call Trace:
> Apr  9 10:05:51 host kernel:  [<ffffffff8100ddb7>] ? =
xen_safe_halt+0xc/0x15
> Apr  9 10:05:51 host kernel:  [<ffffffff8100bdcf>] ? =
xen_idle+0x37/0x40
> Apr  9 10:05:51 host kernel:  [<ffffffff8100fe2e>] ? =
cpu_idle+0x4f/0x82
> Apr  9 10:05:51 host kernel: BUG: soft lockup - CPU#0 stuck for 2305s! =
[httpd:9922]
> Apr  9 10:05:51 host kernel: Modules linked in:
> Apr  9 10:05:51 host kernel: CPU 0:
> Apr  9 10:05:51 host kernel: Modules linked in:
> Apr  9 10:05:51 host kernel: Pid: 9922, comm: httpd Not tainted =
2.6.32.9xls-domU #2=20
> Apr  9 10:05:51 host kernel: RIP: e033:[<00007ff39a44a36d>]  =
[<00007ff39a44a36d>] 0x7ff39a44a36d
> Apr  9 10:05:51 host kernel: RSP: e02b:00007fff4b9ad118  EFLAGS: =
00000a86
> Apr  9 10:05:51 host kernel: RAX: c0c176e403831574 RBX: =
0000000003643b60 RCX: c0c176e403831574
> Apr  9 10:05:51 host kernel: RDX: 0000000000000000 RSI: =
0000000000000004 RDI: 000000000420e43b
> Apr  9 10:05:51 host kernel: RBP: 0000000003643b60 R08: =
0000000000000004 R09: 00007fff4b9ad270
> Apr  9 10:05:51 host kernel: R10: 00007ff39aaa9fa0 R11: =
00007fff4b9ad188 R12: 000000000320e6d0
> Apr  9 10:05:51 host kernel: R13: 0000000000000000 R14: =
00007fff4b9ad368 R15: 0000000000000000
> Apr  9 10:05:51 host kernel: FS:  00007ff39e495710(0000) =
GS:ffff8800022ac000(0000) knlGS:0000000000000000
> Apr  9 10:05:51 host kernel: CS:  e033 DS: 0000 ES: 0000 CR0: =
000000008005003b
> Apr  9 10:05:51 host kernel: CR2: 00000000047a4018 CR3: =
000000007972c000 CR4: 0000000000002660
> Apr  9 10:05:51 host kernel: DR0: 0000000000000000 DR1: =
0000000000000000 DR2: 0000000000000000
> Apr  9 10:05:51 host kernel: DR3: 0000000000000000 DR6: =
00000000ffff0ff0 DR7: 0000000000000400
> Apr  9 10:05:51 host kernel: Call Trace:
> Apr  9 10:50:04 host kernel: BUG: soft lockup - CPU#1 stuck for 2471s! =
[swapper:0]
> Apr  9 10:50:04 host kernel: Modules linked in:
> Apr  9 10:50:04 host kernel: CPU 1:
> Apr  9 10:50:04 host kernel: Modules linked in:
> Apr  9 10:50:04 host kernel: Pid: 0, comm: swapper Not tainted =
2.6.32.9xls-domU #2=20
> Apr  9 10:50:04 host kernel: RIP: e030:[<ffffffff810093aa>]  =
[<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
> Apr  9 10:50:04 host kernel: RSP: e02b:ffff88007f8a1f08  EFLAGS: =
00000246
> Apr  9 10:50:04 host kernel: RAX: 0000000000000000 RBX: =
ffff88007f8a0000 RCX: ffffffff810093aa
> Apr  9 10:50:04 host kernel: RDX: 0000000000000000 RSI: =
0000000000000000 RDI: 0000000000000001
> Apr  9 10:50:04 host kernel: RBP: ffffffff81896d30 R08: =
0000000000000000 R09: ffffffff8100e3b2
> Apr  9 10:50:04 host kernel: R10: 0000000000000001 R11: =
0000000000000246 R12: 0000000000000000
> Apr  9 10:50:04 host kernel: R13: 0000000000000000 R14: =
0000000000000000 R15: 0000000000000000
> Apr  9 10:50:04 host kernel: FS:  00007ff39e495710(0000) =
GS:ffff8800022c7000(0000) knlGS:0000000000000000
> Apr  9 10:50:04 host kernel: CS:  e033 DS: 002b ES: 002b CR0: =
000000008005003b
> Apr  9 10:50:04 host kernel: CR2: 00007ff398a0ce08 CR3: =
000000004ddb4000 CR4: 0000000000002660
> Apr  9 10:50:04 host kernel: DR0: 0000000000000000 DR1: =
0000000000000000 DR2: 0000000000000000
> Apr  9 10:50:04 host kernel: DR3: 0000000000000000 DR6: =
00000000ffff0ff0 DR7: 0000000000000400
> Apr  9 10:50:04 host kernel: Call Trace:
> Apr  9 10:50:04 host kernel:  [<ffffffff8100ddb7>] ? =
xen_safe_halt+0xc/0x15
> Apr  9 10:50:04 host kernel:  [<ffffffff8100bdcf>] ? =
xen_idle+0x37/0x40
> Apr  9 10:50:04 host kernel:  [<ffffffff8100fe2e>] ? =
cpu_idle+0x4f/0x82
> Apr  9 10:50:04 host kernel: BUG: soft lockup - CPU#0 stuck for 2471s! =
[httpd:16797]
> Apr  9 10:50:04 host kernel: Modules linked in:
> Apr  9 10:50:04 host kernel: CPU 0:
> Apr  9 10:50:04 host kernel: Modules linked in:
> Apr  9 10:50:04 host kernel: Pid: 16797, comm: httpd Not tainted =
2.6.32.9xls-domU #2=20
> Apr  9 10:50:04 host kernel: RIP: e033:[<00007ff39a49ae55>]  =
[<00007ff39a49ae55>] 0x7ff39a49ae55
> Apr  9 10:50:04 host kernel: RSP: e02b:00007fff4b9b10b8  EFLAGS: =
00000246
> Apr  9 10:50:04 host kernel: RAX: 000000000368dd40 RBX: =
00000000036868d0 RCX: 222879726575713e
> Apr  9 10:50:05 host kernel: RDX: 000000000000000c RSI: =
0000000003686a6c RDI: 00007fff4b9b1d00
> Apr  9 10:50:05 host kernel: RBP: 00007fff4b9b1da0 R08: =
0000000000000001 R09: 0101010101010101
> Apr  9 10:50:05 host kernel: R10: 0000000000000001 R11: =
00007ff39c4ac0be R12: 00007fff4b9b1d00
> Apr  9 10:50:05 host kernel: R13: 00007ff39aaa9fa0 R14: =
00007ff39aaa9fa0 R15: 0000000003695df0
> Apr  9 10:50:05 host kernel: FS:  00007ff39e495710(0000) =
GS:ffff8800022ac000(0000) knlGS:0000000000000000
> Apr  9 10:50:05 host kernel: CS:  e033 DS: 0000 ES: 0000 CR0: =
000000008005003b
> Apr  9 10:50:05 host kernel: CR2: 00000000032068c0 CR3: =
000000004ddb4000 CR4: 0000000000002660
> Apr  9 10:50:05 host kernel: DR0: 0000000000000000 DR1: =
0000000000000000 DR2: 0000000000000000
> Apr  9 10:50:05 host kernel: DR3: 0000000000000000 DR6: =
00000000ffff0ff0 DR7: 0000000000000400
> Apr  9 10:50:05 host kernel: Call Trace:
> Apr  9 10:57:01 host kernel: BUG: soft lockup - CPU#1 stuck for 389s! =
[exim:17977]
> Apr  9 10:57:01 host kernel: Modules linked in:
> Apr  9 10:57:01 host kernel: CPU 1:
> Apr  9 10:57:01 host kernel: Modules linked in:
> Apr  9 10:57:01 host kernel: Pid: 17977, comm: exim Not tainted =
2.6.32.9xls-domU #2=20
> Apr  9 10:57:01 host kernel: RIP: e030:[<ffffffff8100922a>]  =
[<ffffffff8100922a>] hypercall_page+0x22a/0x1001
> Apr  9 10:57:01 host kernel: RSP: e02b:ffff880056611ea0  EFLAGS: =
00000246
> Apr  9 10:57:01 host kernel: RAX: 0000000000030004 RBX: =
0000000000000000 RCX: ffffffff8100922a
> Apr  9 10:57:01 host kernel: RDX: 00007fd75a6d614e RSI: =
0000000000000000 RDI: 0000000000000000
> Apr  9 10:57:01 host kernel: RBP: 00007fd75a6d614e R08: =
00007fd75c048710 R09: 0000000000000003
> Apr  9 10:57:01 host kernel: R10: 00007fd75a909a30 R11: =
0000000000000246 R12: 0000000000000004
> Apr  9 10:57:01 host kernel: R13: ffff880056611f58 R14: =
ffff88005aa996d0 R15: ffff88005a51b3c0
> Apr  9 10:57:01 host kernel: FS:  00007fd75c048710(0000) =
GS:ffff8800022c7000(0000) knlGS:0000000000000000
> Apr  9 10:57:01 host kernel: CS:  e033 DS: 0000 ES: 0000 CR0: =
000000008005003b
> Apr  9 10:57:01 host kernel: CR2: 00007fd75a6d614e CR3: =
000000007eb9a000 CR4: 0000000000002660
> Apr  9 10:57:01 host kernel: DR0: 0000000000000000 DR1: =
0000000000000000 DR2: 0000000000000000
> Apr  9 10:57:01 host kernel: DR3: 0000000000000000 DR6: =
00000000ffff0ff0 DR7: 0000000000000400
> Apr  9 10:57:01 host kernel: Call Trace:
> Apr  9 10:57:01 host kernel:  [<ffffffff8100dd39>] ? =
xen_force_evtchn_callback+0x9/0xa
> Apr  9 10:57:01 host kernel:  [<ffffffff8100e3b2>] ? =
check_events+0x12/0x20
> Apr  9 10:57:01 host kernel:  [<ffffffff8100e359>] ? =
xen_irq_enable_direct_end+0x0/0x7
> Apr  9 10:57:01 host kernel:  [<ffffffff814da477>] ? =
do_page_fault+0xfb/0x28d
> Apr  9 10:57:01 host kernel:  [<ffffffff814d85b5>] ? =
page_fault+0x25/0x30
> Apr  9 10:57:01 host kernel: BUG: soft lockup - CPU#0 stuck for 389s! =
[netstat:17966]
> Apr  9 10:57:01 host kernel: Modules linked in:
> Apr  9 10:57:01 host kernel: CPU 0:
> Apr  9 10:57:01 host kernel: Modules linked in:
> Apr  9 10:57:01 host kernel: Pid: 17966, comm: netstat Not tainted =
2.6.32.9xls-domU #2=20
> Apr  9 10:57:01 host kernel: RIP: e030:[<ffffffff814d8226>]  =
[<ffffffff814d8226>] _spin_lock+0xa/0x15
> Apr  9 10:57:01 host kernel: RSP: e02b:ffff88006ad4dce0  EFLAGS: =
00000202
> Apr  9 10:57:01 host kernel: RAX: 0000000000001c1c RBX: =
ffff88007ff77180 RCX: 0000000000000012
> Apr  9 10:57:01 host kernel: RDX: 018721dfffc35c9f RSI: =
ffff88006ad4ddc8 RDI: ffff88007ff77188
> Apr  9 10:57:01 host kernel: RBP: ffff88007ff77198 R08: =
0000000000000000 R09: 0000000000300000
> Apr  9 10:57:01 host kernel: R10: 00000000fffffff2 R11: =
ffffffff8126158e R12: ffff88000ff859c0
> Apr  9 10:57:01 host kernel: R13: ffff88006ad4ddc8 R14: =
0000000000034b38 R15: 0000000000000002
> Apr  9 10:57:01 host kernel: FS:  00007fb435545710(0000) =
GS:ffff8800022ac000(0000) knlGS:0000000000000000
> Apr  9 10:57:01 host kernel: CS:  e033 DS: 0000 ES: 0000 CR0: =
000000008005003b
> Apr  9 10:57:01 host kernel: CR2: 00007f0e429d6000 CR3: =
000000007ee3d000 CR4: 0000000000002660
> Apr  9 10:57:01 host kernel: DR0: 0000000000000000 DR1: =
0000000000000000 DR2: 0000000000000000
> Apr  9 10:57:01 host kernel: DR3: 0000000000000000 DR6: =
00000000ffff0ff0 DR7: 0000000000000400
> Apr  9 10:57:01 host kernel: Call Trace:
> Apr  9 10:57:01 host kernel:  [<ffffffff810cb01b>] ? =
__d_lookup+0x7a/0x105
> Apr  9 10:57:01 host kernel:  [<ffffffff810c251e>] ? =
do_lookup+0x2a/0x1b3
> Apr  9 10:57:01 host kernel:  [<ffffffff810c4498>] ? =
__link_path_walk+0x348/0xdcd
> Apr  9 10:57:01 host kernel:  [<ffffffff8100dd39>] ? =
xen_force_evtchn_callback+0x9/0xa
> Apr  9 10:57:01 host kernel:  [<ffffffff8100e3b2>] ? =
check_events+0x12/0x20
> Apr  9 10:57:01 host kernel:  [<ffffffff810c514b>] ? =
path_walk+0x66/0xca
> Apr  9 10:57:01 host kernel:  [<ffffffff810c51cf>] ? =
do_path_lookup+0x20/0x77
> Apr  9 10:57:01 host kernel:  [<ffffffff810c5ae9>] ? =
user_path_at+0x48/0x79
> Apr  9 10:57:01 host kernel:  [<ffffffff8100e39f>] ? =
xen_restore_fl_direct_end+0x0/0x1
> Apr  9 10:57:01 host kernel:  [<ffffffff810cfd6b>] ? =
mntput_no_expire+0x23/0xed
> Apr  9 10:57:01 host kernel:  [<ffffffff8129b703>] ? =
_atomic_dec_and_lock+0x33/0x50
> Apr  9 10:57:01 host kernel:  [<ffffffff810cfd6b>] ? =
mntput_no_expire+0x23/0xed
> Apr  9 10:57:01 host kernel:  [<ffffffff810bdf4d>] ? =
sys_readlinkat+0x25/0x8d
> Apr  9 10:57:01 host kernel:  [<ffffffff81010ac2>] ? =
system_call_fastpath+0x16/0x1b
> Apr  9 11:00:36 host kernel: BUG: soft lockup - CPU#1 stuck for 200s! =
[swapper:0]
> Apr  9 11:00:36 host kernel: Modules linked in:
> Apr  9 11:00:36 host kernel: CPU 1:
> Apr  9 11:00:36 host kernel: Modules linked in:
> Apr  9 11:00:36 host kernel: Pid: 0, comm: swapper Not tainted =
2.6.32.9xls-domU #2=20
> Apr  9 11:00:36 host kernel: RIP: e030:[<ffffffff810093aa>]  =
[<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
> Apr  9 11:00:36 host kernel: RSP: e02b:ffff88007f8a1f08  EFLAGS: =
00000246
> Apr  9 11:00:36 host kernel: RAX: 0000000000000000 RBX: =
ffff88007f8a0000 RCX: ffffffff810093aa
> Apr  9 11:00:36 host kernel: RDX: 0000000000000000 RSI: =
0000000000000000 RDI: 0000000000000001
> Apr  9 11:00:36 host kernel: RBP: ffffffff81896d30 R08: =
0000000000000000 R09: ffffffff8100e3b2
> Apr  9 11:00:36 host kernel: R10: 0000000000000001 R11: =
0000000000000246 R12: 0000000000000000
> Apr  9 11:00:36 host kernel: R13: 0000000000000000 R14: =
0000000000000000 R15: 0000000000000000
> Apr  9 11:00:36 host kernel: FS:  00007f719f5fa6e0(0000) =
GS:ffff8800022c7000(0000) knlGS:0000000000000000
> Apr  9 11:00:36 host kernel: CS:  e033 DS: 002b ES: 002b CR0: =
000000008005003b
> Apr  9 11:00:36 host kernel: CR2: 00007ff39c43cfa8 CR3: =
000000007d0db000 CR4: 0000000000002660
> Apr  9 11:00:36 host kernel: DR0: 0000000000000000 DR1: =
0000000000000000 DR2: 0000000000000000
> Apr  9 11:00:36 host kernel: DR3: 0000000000000000 DR6: =
00000000ffff0ff0 DR7: 0000000000000400
> Apr  9 11:00:37 host kernel: Call Trace:
> Apr  9 11:00:37 host kernel:  [<ffffffff8100ddb7>] ? =
xen_safe_halt+0xc/0x15
> Apr  9 11:00:37 host kernel:  [<ffffffff8100bdcf>] ? =
xen_idle+0x37/0x40
> Apr  9 11:00:37 host kernel:  [<ffffffff8100fe2e>] ? =
cpu_idle+0x4f/0x82
> Apr  9 11:00:37 host kernel: BUG: soft lockup - CPU#0 stuck for 200s! =
[swapper:0]
> Apr  9 11:00:37 host kernel: Modules linked in:
> Apr  9 11:00:37 host kernel: CPU 0:
> Apr  9 11:00:37 host kernel: Modules linked in:
> Apr  9 11:00:37 host kernel: Pid: 0, comm: swapper Not tainted =
2.6.32.9xls-domU #2=20
> Apr  9 11:00:37 host kernel: RIP: e030:[<ffffffff810093aa>]  =
[<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
> Apr  9 11:00:37 host kernel: RSP: e02b:ffffffff81691f70  EFLAGS: =
00000246
> Apr  9 11:00:37 host kernel: RAX: 0000000000000000 RBX: =
ffffffff81690000 RCX: ffffffff810093aa
> Apr  9 11:00:37 host kernel: RDX: 0000000000000000 RSI: =
0000000000000000 RDI: 0000000000000001
> Apr  9 11:00:37 host kernel: RBP: ffffffff81896d30 R08: =
0000000000000000 R09: ffffffff8100e3b2
> Apr  9 11:00:37 host kernel: R10: 0000000000000001 R11: =
0000000000000246 R12: ffffffffffffffff
> Apr  9 11:00:37 host kernel: R13: ffffffff818ebf20 R14: =
ffffffff818eec70 R15: 0000000000000000
> Apr  9 11:00:37 host kernel: FS:  00007f07704fe6e0(0000) =
GS:ffff8800022ac000(0000) knlGS:0000000000000000
> Apr  9 11:00:37 host kernel: CS:  e033 DS: 0000 ES: 0000 CR0: =
000000008005003b
> Apr  9 11:00:37 host kernel: CR2: 00000000035750c8 CR3: =
000000007ea99000 CR4: 0000000000002660
> Apr  9 11:00:37 host kernel: DR0: 0000000000000000 DR1: =
0000000000000000 DR2: 0000000000000000
> Apr  9 11:00:37 host kernel: DR3: 0000000000000000 DR6: =
00000000ffff0ff0 DR7: 0000000000000400
> Apr  9 11:00:37 host kernel: Call Trace:
> Apr  9 11:00:37 host kernel:  [<ffffffff8100ddb7>] ? =
xen_safe_halt+0xc/0x15
> Apr  9 11:00:37 host kernel:  [<ffffffff8100bdcf>] ? =
xen_idle+0x37/0x40
> Apr  9 11:00:37 host kernel:  [<ffffffff8100fe2e>] ? =
cpu_idle+0x4f/0x82
> Apr  9 11:00:37 host kernel:  [<ffffffff818b6c42>] ? =
start_kernel+0x353/0x35f
> Apr  9 11:08:50 host kernel: BUG: soft lockup - CPU#0 stuck for 460s! =
[httpd:19360]
> Apr  9 11:08:50 host kernel: Modules linked in:
> Apr  9 11:08:50 host kernel: CPU 0:
> Apr  9 11:08:50 host kernel: Modules linked in:
> Apr  9 11:08:50 host kernel: Pid: 19360, comm: httpd Not tainted =
2.6.32.9xls-domU #2=20
> Apr  9 11:08:50 host kernel: RIP: e030:[<ffffffff813b995e>]  =
[<ffffffff813b995e>] sock_alloc_send_skb+0x3/0xc
> Apr  9 11:08:50 host kernel: RSP: e02b:ffff88005675dc80  EFLAGS: =
00000246
> Apr  9 11:08:50 host kernel: RAX: 0000000000003e00 RBX: =
ffff88005675dcb8 RCX: ffff88005675dce4
> Apr  9 11:08:50 host kernel: RDX: 0000000000000000 RSI: =
0000000000000057 RDI: ffff88002678c900
> Apr  9 11:08:50 host kernel: RBP: ffff88007f740b80 R08: =
ffff88005675dce4 R09: 75646f72705f6d76
> Apr  9 11:08:50 host kernel: R10: 5245485720607463 R11: =
ffffffff81261683 R12: 0000000000000057
> Apr  9 11:08:50 host kernel: R13: 0000000000000000 R14: =
ffff88006add3c80 R15: ffff88005675dd60
> Apr  9 11:08:50 host kernel: FS:  00007ff39e495710(0000) =
GS:ffff8800022ac000(0000) knlGS:0000000000000000
> Apr  9 11:08:50 host kernel: CS:  e033 DS: 0000 ES: 0000 CR0: =
000000008005003b
> Apr  9 11:08:50 host kernel: CR2: 00000000045cf000 CR3: =
000000007f97d000 CR4: 0000000000002660
> Apr  9 11:08:50 host kernel: DR0: 0000000000000000 DR1: =
0000000000000000 DR2: 0000000000000000
> Apr  9 11:08:50 host kernel: DR3: 0000000000000000 DR6: =
00000000ffff0ff0 DR7: 0000000000000400
> Apr  9 11:08:50 host kernel: Call Trace:
> Apr  9 11:08:50 host kernel:  [<ffffffff814573c8>] ? =
unix_stream_sendmsg+0x18a/0x31f
> Apr  9 11:08:50 host kernel:  [<ffffffff813b61ec>] ? =
sock_aio_write+0xf6/0x10a
> Apr  9 11:08:50 host kernel:  [<ffffffff8100dd39>] ? =
xen_force_evtchn_callback+0x9/0xa
> Apr  9 11:08:50 host kernel:  [<ffffffff8100e39f>] ? =
xen_restore_fl_direct_end+0x0/0x1
> Apr  9 11:08:50 host kernel:  [<ffffffff810bab7e>] ? =
do_sync_write+0xce/0x113
> Apr  9 11:08:50 host kernel:  [<ffffffff810c5af3>] ? =
user_path_at+0x52/0x79
> Apr  9 11:08:50 host kernel:  [<ffffffff810be0b0>] ? =
cp_new_stat+0xe9/0xfc
> Apr  9 11:08:50 host kernel:  [<ffffffff810561b8>] ? =
autoremove_wake_function+0x0/0x2e
> Apr  9 11:08:50 host kernel:  [<ffffffff8100e3b2>] ? =
check_events+0x12/0x20
> Apr  9 11:08:50 host kernel:  [<ffffffff8100dd39>] ? =
xen_force_evtchn_callback+0x9/0xa
> Apr  9 11:08:50 host kernel:  [<ffffffff8100e3b2>] ? =
check_events+0x12/0x20
> Apr  9 11:08:50 host kernel:  [<ffffffff810bb4c5>] ? =
vfs_write+0xc0/0x149
> Apr  9 11:08:50 host kernel:  [<ffffffff810bb60a>] ? =
sys_write+0x45/0x6e
> Apr  9 11:08:50 host kernel:  [<ffffffff81010ac2>] ? =
system_call_fastpath+0x16/0x1b
> Apr  9 11:08:50 host kernel: BUG: soft lockup - CPU#1 stuck for 460s! =
[swapper:0]
> Apr  9 11:08:50 host kernel: Modules linked in:
> Apr  9 11:08:50 host kernel: CPU 1:
> Apr  9 11:08:50 host kernel: Modules linked in:
> Apr  9 11:08:50 host kernel: Pid: 0, comm: swapper Not tainted =
2.6.32.9xls-domU #2=20
> Apr  9 11:08:50 host kernel: RIP: e030:[<ffffffff810093aa>]  =
[<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
> Apr  9 11:08:50 host kernel: RSP: e02b:ffff88007f8a1f08  EFLAGS: =
00000246
> Apr  9 11:08:50 host kernel: RAX: 0000000000000000 RBX: =
ffff88007f8a0000 RCX: ffffffff810093aa
> Apr  9 11:08:50 host kernel: RDX: 0000000000000000 RSI: =
0000000000000000 RDI: 0000000000000001
> Apr  9 11:08:50 host kernel: RBP: ffffffff81896d30 R08: =
0000000000000000 R09: ffffffff8100e3b2
> Apr  9 11:08:50 host kernel: R10: 0000000000000001 R11: =
0000000000000246 R12: 0000000000000000
> Apr  9 11:08:50 host kernel: R13: 0000000000000000 R14: =
0000000000000000 R15: 0000000000000000
> Apr  9 11:08:50 host kernel: FS:  00007f0e429cb6e0(0000) =
GS:ffff8800022c7000(0000) knlGS:0000000000000000
> Apr  9 11:08:50 host kernel: CS:  e033 DS: 002b ES: 002b CR0: =
000000008005003b
> Apr  9 11:08:50 host kernel: CR2: 0000000004598000 CR3: =
000000007e2a6000 CR4: 0000000000002660
> Apr  9 11:08:50 host kernel: DR0: 0000000000000000 DR1: =
0000000000000000 DR2: 0000000000000000
> Apr  9 11:08:50 host kernel: DR3: 0000000000000000 DR6: =
00000000ffff0ff0 DR7: 0000000000000400
> Apr  9 11:08:50 host kernel: Call Trace:
> Apr  9 11:08:50 host kernel:  [<ffffffff8100ddb7>] ? =
xen_safe_halt+0xc/0x15
> Apr  9 11:08:50 host kernel:  [<ffffffff8100bdcf>] ? =
xen_idle+0x37/0x40
> Apr  9 11:08:50 host kernel:  [<ffffffff8100fe2e>] ? =
cpu_idle+0x4f/0x82
> Apr  9 12:00:40 host kernel: BUG: soft lockup - CPU#1 stuck for 2897s! =
[httpd:26903]
> Apr  9 12:00:40 host kernel: Modules linked in:
> Apr  9 12:00:40 host kernel: CPU 1:
> Apr  9 12:00:40 host kernel: Modules linked in:
> Apr  9 12:00:40 host kernel: Pid: 26903, comm: httpd Not tainted =
2.6.32.9xls-domU #2=20
> Apr  9 12:00:40 host kernel: RIP: e033:[<00007ff39a427cd2>]  =
[<00007ff39a427cd2>] 0x7ff39a427cd2
> Apr  9 12:00:40 host kernel: RSP: e02b:00007fff4b9afa20  EFLAGS: =
00000202
> Apr  9 12:00:40 host kernel: RAX: 0000000000000020 RBX: =
00007fff4b9aff30 RCX: 0000000000000013
> Apr  9 12:00:40 host kernel: RDX: 00007ff39a77be20 RSI: =
00007fff4b9aff50 RDI: 0000000003bd6418
> Apr  9 12:00:40 host kernel: RBP: 0000000000000000 R08: =
00007ff39a782e60 R09: 00007ff39a783d80
> Apr  9 12:00:40 host kernel: R10: 0000000000000000 R11: =
00007ff39c4ac0a2 R12: 00007ff39aaaa3e0
> Apr  9 12:00:40 host kernel: R13: 00007fff4b9aff50 R14: =
00007fff4b9aff50 R15: 000000000000015e
> Apr  9 12:00:40 host kernel: FS:  00007ff39e495710(0000) =
GS:ffff8800022c7000(0000) knlGS:0000000000000000
> Apr  9 12:00:40 host kernel: CS:  e033 DS: 0000 ES: 0000 CR0: =
000000008005003b
> Apr  9 12:00:40 host kernel: CR2: 0000000003bdc000 CR3: =
0000000056566000 CR4: 0000000000002660
> Apr  9 12:00:40 host kernel: DR0: 0000000000000000 DR1: =
0000000000000000 DR2: 0000000000000000
> Apr  9 12:00:40 host kernel: DR3: 0000000000000000 DR6: =
00000000ffff0ff0 DR7: 0000000000000400
> Apr  9 12:00:40 host kernel: Call Trace:
> Apr  9 12:00:40 host kernel: BUG: soft lockup - CPU#0 stuck for 2897s! =
[swapper:0]
> Apr  9 12:00:40 host kernel: Modules linked in:
> Apr  9 12:00:40 host kernel: CPU 0:
> Apr  9 12:00:40 host kernel: Modules linked in:
> Apr  9 12:00:40 host kernel: Pid: 0, comm: swapper Not tainted =
2.6.32.9xls-domU #2=20
> Apr  9 12:00:40 host kernel: RIP: e030:[<ffffffff810093aa>]  =
[<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
> Apr  9 12:00:40 host kernel: RSP: e02b:ffffffff81691f70  EFLAGS: =
00000246
> Apr  9 12:00:40 host kernel: RAX: 0000000000000000 RBX: =
ffffffff81690000 RCX: ffffffff810093aa
> Apr  9 12:00:40 host kernel: RDX: 0000000000000000 RSI: =
0000000000000000 RDI: 0000000000000001
> Apr  9 12:00:40 host kernel: RBP: ffffffff81896d30 R08: =
0000000000000000 R09: ffffffff8100e3b2
> Apr  9 12:00:40 host kernel: R10: 0000000000000001 R11: =
0000000000000246 R12: ffffffffffffffff
> Apr  9 12:00:40 host kernel: R13: ffffffff818ebf20 R14: =
ffffffff818eec70 R15: 0000000000000000
> Apr  9 12:00:40 host kernel: FS:  00007f74fde746e0(0000) =
GS:ffff8800022ac000(0000) knlGS:0000000000000000
> Apr  9 12:00:40 host kernel: CS:  e033 DS: 0000 ES: 0000 CR0: =
000000008005003b
> Apr  9 12:00:40 host kernel: CR2: 00007ff39aec11a7 CR3: =
000000007e2a6000 CR4: 0000000000002660
> Apr  9 12:00:40 host kernel: DR0: 0000000000000000 DR1: =
0000000000000000 DR2: 0000000000000000
> Apr  9 12:00:40 host kernel: DR3: 0000000000000000 DR6: =
00000000ffff0ff0 DR7: 0000000000000400
> Apr  9 12:00:40 host kernel: Call Trace:
> Apr  9 12:00:40 host kernel:  [<ffffffff8100ddb7>] ? =
xen_safe_halt+0xc/0x15
> Apr  9 12:00:40 host kernel:  [<ffffffff8100bdcf>] ? =
xen_idle+0x37/0x40
> Apr  9 12:00:40 host kernel:  [<ffffffff8100fe2e>] ? =
cpu_idle+0x4f/0x82
> Apr  9 12:00:40 host kernel:  [<ffffffff818b6c42>] ? =
start_kernel+0x353/0x35f
> Apr  9 12:19:45 host kernel: BUG: soft lockup - CPU#1 stuck for 1065s! =
[swapper:0]
> Apr  9 12:19:45 host kernel: Modules linked in:
> Apr  9 12:19:45 host kernel: CPU 1:
> Apr  9 12:19:45 host kernel: Modules linked in:
> Apr  9 12:19:45 host kernel: Pid: 0, comm: swapper Not tainted =
2.6.32.9xls-domU #2=20
> Apr  9 12:19:45 host kernel: RIP: e030:[<ffffffff810093aa>]  =
[<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
> Apr  9 12:19:45 host kernel: RSP: e02b:ffff88007f8a1f08  EFLAGS: =
00000246
> Apr  9 12:19:45 host kernel: RAX: 0000000000000000 RBX: =
ffff88007f8a0000 RCX: ffffffff810093aa
> Apr  9 12:19:45 host kernel: RDX: 0000000000000000 RSI: =
0000000000000000 RDI: 0000000000000001
> Apr  9 12:19:45 host kernel: RBP: ffffffff81896d30 R08: =
0000000000000000 R09: ffffffff8100e3b2
> Apr  9 12:19:45 host kernel: R10: 0000000000000001 R11: =
0000000000000246 R12: 0000000000000000
> Apr  9 12:19:45 host kernel: R13: 0000000000000000 R14: =
0000000000000000 R15: 0000000000000000
> Apr  9 12:19:45 host kernel: FS:  00007ff39e495710(0000) =
GS:ffff8800022c7000(0000) knlGS:0000000000000000
> Apr  9 12:19:45 host kernel: CS:  e033 DS: 002b ES: 002b CR0: =
000000008005003b
> Apr  9 12:19:45 host kernel: CR2: 00007ff39c5002b0 CR3: =
0000000079588000 CR4: 0000000000002660
> Apr  9 12:19:45 host kernel: DR0: 0000000000000000 DR1: =
0000000000000000 DR2: 0000000000000000
> Apr  9 12:19:45 host kernel: DR3: 0000000000000000 DR6: =
00000000ffff0ff0 DR7: 0000000000000400
> Apr  9 12:19:45 host kernel: Call Trace:
> Apr  9 12:19:45 host kernel:  [<ffffffff8100ddb7>] ? =
xen_safe_halt+0xc/0x15
> Apr  9 12:19:45 host kernel:  [<ffffffff8100bdcf>] ? =
xen_idle+0x37/0x40
> Apr  9 12:19:45 host kernel:  [<ffffffff8100fe2e>] ? =
cpu_idle+0x4f/0x82
> Apr  9 12:19:45 host kernel: BUG: soft lockup - CPU#0 stuck for 1065s! =
[swapper:0]
> Apr  9 12:19:45 host kernel: Modules linked in:
> Apr  9 12:19:45 host kernel: CPU 0:
> Apr  9 12:19:45 host kernel: Modules linked in:
> Apr  9 12:19:45 host kernel: Pid: 0, comm: swapper Not tainted =
2.6.32.9xls-domU #2=20
> Apr  9 12:19:45 host kernel: RIP: e030:[<ffffffff810093aa>]  =
[<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
> Apr  9 12:19:45 host kernel: RSP: e02b:ffffffff81691f70  EFLAGS: =
00000246
> Apr  9 12:19:45 host kernel: RAX: 0000000000000000 RBX: =
ffffffff81690000 RCX: ffffffff810093aa
> Apr  9 12:19:45 host kernel: RDX: 0000000000000000 RSI: =
0000000000000000 RDI: 0000000000000001
> Apr  9 12:19:45 host kernel: RBP: ffffffff81896d30 R08: =
0000000000000000 R09: ffffffff8100e3b2
> Apr  9 12:19:45 host kernel: R10: 0000000000000001 R11: =
0000000000000246 R12: ffffffffffffffff
> Apr  9 12:19:45 host kernel: R13: ffffffff818ebf20 R14: =
ffffffff818eec70 R15: 0000000000000000
> Apr  9 12:19:45 host kernel: FS:  00007ff39e495710(0000) =
GS:ffff8800022ac000(0000) knlGS:0000000000000000
> Apr  9 12:19:45 host kernel: CS:  e033 DS: 0000 ES: 0000 CR0: =
000000008005003b
> Apr  9 12:19:45 host kernel: CR2: 00007ff39e404000 CR3: =
000000007e2a6000 CR4: 0000000000002660
> Apr  9 12:19:45 host kernel: DR0: 0000000000000000 DR1: =
0000000000000000 DR2: 0000000000000000
> Apr  9 12:19:45 host kernel: DR3: 0000000000000000 DR6: =
00000000ffff0ff0 DR7: 0000000000000400
> Apr  9 12:19:45 host kernel: Call Trace:
> Apr  9 12:19:45 host kernel:  [<ffffffff8100ddb7>] ? =
xen_safe_halt+0xc/0x15
> Apr  9 12:19:45 host kernel:  [<ffffffff8100bdcf>] ? =
xen_idle+0x37/0x40
> Apr  9 12:19:45 host kernel:  [<ffffffff8100fe2e>] ? =
cpu_idle+0x4f/0x82
> Apr  9 12:19:45 host kernel:  [<ffffffff818b6c42>] ? =
start_kernel+0x353/0x35f
> Apr  9 12:37:26 host kernel: BUG: soft lockup - CPU#1 stuck for 989s! =
[swapper:0]
> Apr  9 12:37:26 host kernel: Modules linked in:
> Apr  9 12:37:26 host kernel: CPU 1:
> Apr  9 12:37:26 host kernel: Modules linked in:
> Apr  9 12:37:26 host kernel: Pid: 0, comm: swapper Not tainted =
2.6.32.9xls-domU #2=20
> Apr  9 12:37:26 host kernel: RIP: e030:[<ffffffff810093aa>]  =
[<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
> Apr  9 12:37:26 host kernel: RSP: e02b:ffff88007f8a1f08  EFLAGS: =
00000246
> Apr  9 12:37:26 host kernel: RAX: 0000000000000000 RBX: =
ffff88007f8a0000 RCX: ffffffff810093aa
> Apr  9 12:37:26 host kernel: RDX: 0000000000000000 RSI: =
0000000000000000 RDI: 0000000000000001
> Apr  9 12:37:26 host kernel: RBP: ffffffff81896d30 R08: =
0000000000000000 R09: ffffffff8100e3b2
> Apr  9 12:37:26 host kernel: R10: 0000000000000001 R11: =
0000000000000246 R12: 0000000000000000
> Apr  9 12:37:26 host kernel: R13: 0000000000000000 R14: =
0000000000000000 R15: 0000000000000000
> Apr  9 12:37:26 host kernel: FS:  00007ff39e495710(0000) =
GS:ffff8800022c7000(0000) knlGS:0000000000000000
> Apr  9 12:37:26 host kernel: CS:  e033 DS: 002b ES: 002b CR0: =
000000008005003b
> Apr  9 12:37:26 host kernel: CR2: 00000000039f7330 CR3: =
000000007e2a6000 CR4: 0000000000002660
> Apr  9 12:37:26 host kernel: DR0: 0000000000000000 DR1: =
0000000000000000 DR2: 0000000000000000
> Apr  9 12:37:26 host kernel: DR3: 0000000000000000 DR6: =
00000000ffff0ff0 DR7: 0000000000000400
> Apr  9 12:37:26 host kernel: Call Trace:
> Apr  9 12:37:26 host kernel:  [<ffffffff8100ddb7>] ? =
xen_safe_halt+0xc/0x15
> Apr  9 12:37:26 host kernel:  [<ffffffff8100bdcf>] ? =
xen_idle+0x37/0x40
> Apr  9 12:37:26 host kernel:  [<ffffffff8100fe2e>] ? =
cpu_idle+0x4f/0x82
> Apr  9 12:37:26 host kernel: BUG: soft lockup - CPU#0 stuck for 989s! =
[httpd:31684]
> Apr  9 12:37:26 host kernel: Modules linked in:
> Apr  9 12:37:26 host kernel: CPU 0:
> Apr  9 12:37:26 host kernel: Modules linked in:
> Apr  9 12:37:26 host kernel: Pid: 31684, comm: httpd Not tainted =
2.6.32.9xls-domU #2=20
> Apr  9 12:37:26 host kernel: RIP: e033:[<00007ff39a41f7d3>]  =
[<00007ff39a41f7d3>] 0x7ff39a41f7d3
> Apr  9 12:37:26 host kernel: RSP: e02b:00007fff4b9ad330  EFLAGS: =
00000246
> Apr  9 12:37:26 host kernel: RAX: 00007ff39a41f7ac RBX: =
00007ff39aaaa3e0 RCX: 0000000000000001
> Apr  9 12:37:26 host kernel: RDX: 00007ff39a781bdc RSI: =
000000000000003e RDI: 0000000000000000
> Apr  9 12:37:26 host kernel: RBP: 00007ff39aaa9e60 R08: =
00007ff39a782e60 R09: 0000000000000003
> Apr  9 12:37:26 host kernel: R10: 0000000000000000 R11: =
00007ff39c4ac0a2 R12: 0000000003a2b297
> Apr  9 12:37:26 host kernel: R13: 00007ff39aaaa3e0 R14: =
00007ff39a784da0 R15: 00007ff39a783e80
> Apr  9 12:37:26 host kernel: FS:  00007ff39e495710(0000) =
GS:ffff8800022ac000(0000) knlGS:0000000000000000
> Apr  9 12:37:26 host kernel: CS:  e033 DS: 0000 ES: 0000 CR0: =
000000008005003b
> Apr  9 12:37:26 host kernel: CR2: 0000000003aa1000 CR3: =
000000004debb000 CR4: 0000000000002660
> Apr  9 12:37:26 host kernel: DR0: 0000000000000000 DR1: =
0000000000000000 DR2: 0000000000000000
> Apr  9 12:37:26 host kernel: DR3: 0000000000000000 DR6: =
00000000ffff0ff0 DR7: 0000000000000400
> Apr  9 12:37:26 host kernel: Call Trace:
> Apr  9 13:44:52 host kernel: BUG: soft lockup - CPU#1 stuck for 3768s! =
[swapper:0]
> Apr  9 13:44:52 host kernel: Modules linked in:
> Apr  9 13:44:52 host kernel: CPU 1:
> Apr  9 13:44:52 host kernel: Modules linked in:
> Apr  9 13:44:52 host kernel: Pid: 0, comm: swapper Not tainted =
2.6.32.9xls-domU #2=20
> Apr  9 13:44:52 host kernel: RIP: e030:[<ffffffff810093aa>]  =
[<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
> Apr  9 13:44:52 host kernel: RSP: e02b:ffff88007f8a1f08  EFLAGS: =
00000246
> Apr  9 13:44:52 host kernel: RAX: 0000000000000000 RBX: =
ffff88007f8a0000 RCX: ffffffff810093aa
> Apr  9 13:44:52 host kernel: RDX: 0000000000000000 RSI: =
0000000000000000 RDI: 0000000000000001
> Apr  9 13:44:52 host kernel: RBP: ffffffff81896d30 R08: =
0000000000000000 R09: ffffffff8100e3b2
> Apr  9 13:44:52 host kernel: R10: 0000000000000001 R11: =
0000000000000246 R12: 0000000000000000
> Apr  9 13:44:52 host kernel: R13: 0000000000000000 R14: =
0000000000000000 R15: 0000000000000000
> Apr  9 13:44:52 host kernel: FS:  00007fd75c048710(0000) =
GS:ffff8800022c7000(0000) knlGS:0000000000000000
> Apr  9 13:44:52 host kernel: CS:  e033 DS: 002b ES: 002b CR0: =
000000008005003b
> Apr  9 13:44:52 host kernel: CR2: 00007fff892a1e30 CR3: =
0000000071c86000 CR4: 0000000000002660
> Apr  9 13:44:52 host kernel: DR0: 0000000000000000 DR1: =
0000000000000000 DR2: 0000000000000000
> Apr  9 13:44:52 host kernel: DR3: 0000000000000000 DR6: =
00000000ffff0ff0 DR7: 0000000000000400
> Apr  9 13:44:52 host kernel: Call Trace:
> Apr  9 13:44:52 host kernel:  [<ffffffff8100ddb7>] ? =
xen_safe_halt+0xc/0x15
> Apr  9 13:44:52 host kernel:  [<ffffffff8100bdcf>] ? =
xen_idle+0x37/0x40
> Apr  9 13:44:52 host kernel:  [<ffffffff8100fe2e>] ? =
cpu_idle+0x4f/0x82
> Apr  9 13:44:52 host kernel: BUG: soft lockup - CPU#0 stuck for 3768s! =
[swapper:0]
> Apr  9 13:44:52 host kernel: Modules linked in:
> Apr  9 13:44:52 host kernel: CPU 0:
> Apr  9 13:44:52 host kernel: Modules linked in:
> Apr  9 13:44:52 host kernel: Pid: 0, comm: swapper Not tainted =
2.6.32.9xls-domU #2=20
> Apr  9 13:44:52 host kernel: RIP: e030:[<ffffffff810093aa>]  =
[<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
> Apr  9 13:44:52 host kernel: RSP: e02b:ffffffff81691f70  EFLAGS: =
00000246
> Apr  9 13:44:52 host kernel: RAX: 0000000000000000 RBX: =
ffffffff81690000 RCX: ffffffff810093aa
> Apr  9 13:44:52 host kernel: RDX: 0000000000000000 RSI: =
0000000000000000 RDI: 0000000000000001
> Apr  9 13:44:52 host kernel: RBP: ffffffff81896d30 R08: =
0000000000000000 R09: ffffffff8100e3b2
> Apr  9 13:44:52 host kernel: R10: 0000000000000001 R11: =
0000000000000246 R12: ffffffffffffffff
> Apr  9 13:44:52 host kernel: R13: ffffffff818ebf20 R14: =
ffffffff818eec70 R15: 0000000000000000
> Apr  9 13:44:52 host kernel: FS:  00007f0e429cb6e0(0000) =
GS:ffff8800022ac000(0000) knlGS:0000000000000000
> Apr  9 13:44:52 host kernel: CS:  e033 DS: 0000 ES: 0000 CR0: =
000000008005003b
> Apr  9 13:44:52 host kernel: CR2: 00007f0e429d6000 CR3: =
000000007d0de000 CR4: 0000000000002660
> Apr  9 13:44:52 host kernel: DR0: 0000000000000000 DR1: =
0000000000000000 DR2: 0000000000000000
> Apr  9 13:44:52 host kernel: DR3: 0000000000000000 DR6: =
00000000ffff0ff0 DR7: 0000000000000400
> Apr  9 13:44:52 host kernel: Call Trace:
> Apr  9 13:44:52 host kernel:  [<ffffffff8100ddb7>] ? =
xen_safe_halt+0xc/0x15
> Apr  9 13:44:52 host kernel:  [<ffffffff8100bdcf>] ? =
xen_idle+0x37/0x40
> Apr  9 13:44:52 host kernel:  [<ffffffff8100fe2e>] ? =
cpu_idle+0x4f/0x82
> Apr  9 13:44:52 host kernel:  [<ffffffff818b6c42>] ? =
start_kernel+0x353/0x35f
> Apr  9 13:55:43 host kernel: BUG: soft lockup - CPU#1 stuck for 607s! =
[swapper:0]
> Apr  9 13:55:43 host kernel: Modules linked in:
> Apr  9 13:55:43 host kernel: CPU 1:
> Apr  9 13:55:43 host kernel: Modules linked in:
> Apr  9 13:55:43 host kernel: Pid: 0, comm: swapper Not tainted =
2.6.32.9xls-domU #2=20
> Apr  9 13:55:43 host kernel: RIP: e030:[<ffffffff810093aa>]  =
[<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
> Apr  9 13:55:43 host kernel: RSP: e02b:ffff88007f8a1f08  EFLAGS: =
00000246
> Apr  9 13:55:43 host kernel: RAX: 0000000000000000 RBX: =
ffff88007f8a0000 RCX: ffffffff810093aa
> Apr  9 13:55:43 host kernel: RDX: 0000000000000000 RSI: =
0000000000000000 RDI: 0000000000000001
> Apr  9 13:55:43 host kernel: RBP: ffffffff81896d30 R08: =
0000000000000000 R09: ffffffff8100e3b2
> Apr  9 13:55:43 host kernel: R10: 0000000000000001 R11: =
0000000000000246 R12: 0000000000000000
> Apr  9 13:55:43 host kernel: R13: 0000000000000000 R14: =
0000000000000000 R15: 0000000000000000
> Apr  9 13:55:43 host kernel: FS:  00007ff39e495710(0000) =
GS:ffff8800022c7000(0000) knlGS:0000000000000000
> Apr  9 13:55:43 host kernel: CS:  e033 DS: 002b ES: 002b CR0: =
000000008005003b
> Apr  9 13:55:43 host kernel: CR2: 0000000003c7b27f CR3: =
00000000266e8000 CR4: 0000000000002660
> Apr  9 13:55:43 host kernel: DR0: 0000000000000000 DR1: =
0000000000000000 DR2: 0000000000000000
> Apr  9 13:55:43 host kernel: DR3: 0000000000000000 DR6: =
00000000ffff0ff0 DR7: 0000000000000400
> Apr  9 13:55:43 host kernel: Call Trace:
> Apr  9 13:55:43 host kernel:  [<ffffffff8100ddb7>] ? =
xen_safe_halt+0xc/0x15
> Apr  9 13:55:43 host kernel:  [<ffffffff8100bdcf>] ? =
xen_idle+0x37/0x40
> Apr  9 13:55:43 host kernel:  [<ffffffff8100fe2e>] ? =
cpu_idle+0x4f/0x82
> Apr  9 13:55:43 host kernel: BUG: soft lockup - CPU#0 stuck for 607s! =
[swapper:0]
> Apr  9 13:55:43 host kernel: Modules linked in:
> Apr  9 13:55:43 host kernel: CPU 0:
> Apr  9 13:55:43 host kernel: Modules linked in:
> Apr  9 13:55:43 host kernel: Pid: 0, comm: swapper Not tainted =
2.6.32.9xls-domU #2=20
> Apr  9 13:55:43 host kernel: RIP: e030:[<ffffffff810093aa>]  =
[<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
> Apr  9 13:55:43 host kernel: RSP: e02b:ffffffff81691f70  EFLAGS: =
00000246
> Apr  9 13:55:43 host kernel: RAX: 0000000000000000 RBX: =
ffffffff81690000 RCX: ffffffff810093aa
> Apr  9 13:55:43 host kernel: RDX: 0000000000000000 RSI: =
0000000000000000 RDI: 0000000000000001
> Apr  9 13:55:43 host kernel: RBP: ffffffff81896d30 R08: =
0000000000000000 R09: ffffffff8100e3b2
> Apr  9 13:55:43 host kernel: R10: 0000000000000001 R11: =
0000000000000246 R12: ffffffffffffffff
> Apr  9 13:55:43 host kernel: R13: ffffffff818ebf20 R14: =
ffffffff818eec70 R15: 0000000000000000
> Apr  9 13:55:43 host kernel: FS:  00007ff39e495710(0000) =
GS:ffff8800022ac000(0000) knlGS:0000000000000000
> Apr  9 13:55:43 host kernel: CS:  e033 DS: 0000 ES: 0000 CR0: =
000000008005003b
> Apr  9 13:55:43 host kernel: CR2: 00007f2f28bf62e8 CR3: =
000000007e2a6000 CR4: 0000000000002660
> Apr  9 13:55:43 host kernel: DR0: 0000000000000000 DR1: =
0000000000000000 DR2: 0000000000000000
> Apr  9 13:55:43 host kernel: DR3: 0000000000000000 DR6: =
00000000ffff0ff0 DR7: 0000000000000400
> Apr  9 13:55:43 host kernel: Call Trace:
> Apr  9 13:55:43 host kernel:  [<ffffffff8100ddb7>] ? =
xen_safe_halt+0xc/0x15
> Apr  9 13:55:43 host kernel:  [<ffffffff8100bdcf>] ? =
xen_idle+0x37/0x40
> Apr  9 13:55:43 host kernel:  [<ffffffff8100fe2e>] ? =
cpu_idle+0x4f/0x82
> Apr  9 13:55:43 host kernel:  [<ffffffff818b6c42>] ? =
start_kernel+0x353/0x35f
> Apr  9 14:16:21 host kernel: BUG: soft lockup - CPU#1 stuck for 1153s! =
[swapper:0]
> Apr  9 14:16:21 host kernel: Modules linked in:
> Apr  9 14:16:21 host kernel: CPU 1:
> Apr  9 14:16:21 host kernel: Modules linked in:
> Apr  9 14:16:21 host kernel: Pid: 0, comm: swapper Not tainted =
2.6.32.9xls-domU #2=20
> Apr  9 14:16:21 host kernel: RIP: e030:[<ffffffff8100922a>]  =
[<ffffffff8100922a>] hypercall_page+0x22a/0x1001
> Apr  9 14:16:21 host kernel: RSP: e02b:ffff88007f8a1dd8  EFLAGS: =
00000246
> Apr  9 14:16:21 host kernel: RAX: 0000000000030004 RBX: =
ffff88007db55810 RCX: ffffffff8100922a
> Apr  9 14:16:21 host kernel: RDX: 0000000000000000 RSI: =
0000000000000000 RDI: 0000000000000000
> Apr  9 14:16:21 host kernel: RBP: 0000000000000001 R08: =
ffff88007f8a0000 R09: ffffffff8100e3b2
> Apr  9 14:16:21 host kernel: R10: 0000000000000001 R11: =
0000000000000246 R12: 0000000000000000
> Apr  9 14:16:21 host kernel: R13: ffff88007e0040c0 R14: =
0000000000000000 R15: 0000000000001000
> Apr  9 14:16:21 host kernel: FS:  00007ff39e495710(0000) =
GS:ffff8800022c7000(0000) knlGS:0000000000000000
> Apr  9 14:16:21 host kernel: CS:  e033 DS: 002b ES: 002b CR0: =
000000008005003b
> Apr  9 14:16:21 host kernel: CR2: 00007ff39e4a7000 CR3: =
000000007e2a6000 CR4: 0000000000002660
> Apr  9 14:16:21 host kernel: DR0: 0000000000000000 DR1: =
0000000000000000 DR2: 0000000000000000
> Apr  9 14:16:21 host kernel: DR3: 0000000000000000 DR6: =
00000000ffff0ff0 DR7: 0000000000000400
> Apr  9 14:16:21 host kernel: Call Trace:
> Apr  9 14:16:21 host kernel:  [<ffffffff8100dd39>] ? =
xen_force_evtchn_callback+0x9/0xa
> Apr  9 14:16:21 host kernel:  [<ffffffff8100e3b2>] ? =
check_events+0x12/0x20
> Apr  9 14:16:21 host kernel:  [<ffffffff8100e3b2>] ? =
check_events+0x12/0x20
> Apr  9 14:16:21 host kernel:  [<ffffffff8100e359>] ? =
xen_irq_enable_direct_end+0x0/0x7
> Apr  9 14:16:21 host kernel:  [<ffffffff8103b070>] ? =
finish_task_switch+0x27/0x6e
> Apr  9 14:16:21 host kernel:  [<ffffffff814d6a77>] ? =
thread_return+0x3e/0x109
> Apr  9 14:16:21 host kernel:  [<ffffffff814d87ba>] ? =
error_exit+0x2a/0x60
> Apr  9 14:16:21 host kernel:  [<ffffffff8101149d>] ? =
retint_restore_args+0x5/0x6
> Apr  9 14:16:21 host kernel:  [<ffffffff8100fe5f>] ? =
cpu_idle+0x80/0x82
> Apr  9 14:16:21 host kernel: BUG: soft lockup - CPU#0 stuck for 1153s! =
[swapper:0]
> Apr  9 14:16:21 host kernel: Modules linked in:
> Apr  9 14:16:21 host kernel: CPU 0:
> Apr  9 14:16:21 host kernel: Modules linked in:
> Apr  9 14:16:21 host kernel: Pid: 0, comm: swapper Not tainted =
2.6.32.9xls-domU #2=20
> Apr  9 14:16:21 host kernel: RIP: e030:[<ffffffff810093aa>]  =
[<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
> Apr  9 14:16:21 host kernel: RSP: e02b:ffffffff81691f70  EFLAGS: =
00000246
> Apr  9 14:16:21 host kernel: RAX: 0000000000000000 RBX: =
ffffffff81690000 RCX: ffffffff810093aa
> Apr  9 14:16:21 host kernel: RDX: 0000000000000000 RSI: =
0000000000000000 RDI: 0000000000000001
> Apr  9 14:16:21 host kernel: RBP: ffffffff81896d30 R08: =
0000000000000000 R09: ffffffff8100e3b2
> Apr  9 14:16:21 host kernel: R10: 0000000000000001 R11: =
0000000000000246 R12: ffffffffffffffff
> Apr  9 14:16:21 host kernel: R13: ffffffff818ebf20 R14: =
ffffffff818eec70 R15: 0000000000000000
> Apr  9 14:16:21 host kernel: FS:  00007ff39e495710(0000) =
GS:ffff8800022ac000(0000) knlGS:0000000000000000
> Apr  9 14:16:21 host kernel: CS:  e033 DS: 0000 ES: 0000 CR0: =
000000008005003b
> Apr  9 14:16:21 host kernel: CR2: 00007ff39e4a7000 CR3: =
000000007e2a6000 CR4: 0000000000002660
> Apr  9 14:16:21 host kernel: DR0: 0000000000000000 DR1: =
0000000000000000 DR2: 0000000000000000
> Apr  9 14:16:21 host kernel: DR3: 0000000000000000 DR6: =
00000000ffff0ff0 DR7: 0000000000000400
> Apr  9 14:16:21 host kernel: Call Trace:
> Apr  9 14:16:21 host kernel:  [<ffffffff8100ddb7>] ? =
xen_safe_halt+0xc/0x15
> Apr  9 14:16:21 host kernel:  [<ffffffff8100bdcf>] ? =
xen_idle+0x37/0x40
> Apr  9 14:16:21 host kernel:  [<ffffffff8100fe2e>] ? =
cpu_idle+0x4f/0x82
> Apr  9 14:16:21 host kernel:  [<ffffffff818b6c42>] ? =
start_kernel+0x353/0x35f
> Apr  9 15:40:21 host kernel: BUG: soft lockup - CPU#1 stuck for 4104s! =
[swapper:0]
> Apr  9 15:40:21 host kernel: Modules linked in:
> Apr  9 15:40:21 host kernel: CPU 1:
> Apr  9 15:40:21 host kernel: Modules linked in:
> Apr  9 15:40:21 host kernel: Pid: 0, comm: swapper Not tainted =
2.6.32.9xls-domU #2=20
> Apr  9 15:40:21 host kernel: RIP: e030:[<ffffffff810093aa>]  =
[<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
> Apr  9 15:40:21 host kernel: RSP: e02b:ffff88007f8a1f08  EFLAGS: =
00000246
> Apr  9 15:40:21 host kernel: RAX: 0000000000000000 RBX: =
ffff88007f8a0000 RCX: ffffffff810093aa
> Apr  9 15:40:21 host kernel: RDX: 0000000000000000 RSI: =
0000000000000000 RDI: 0000000000000001
> Apr  9 15:40:21 host kernel: RBP: ffffffff81896d30 R08: =
0000000000000000 R09: ffffffff8100e3b2
> Apr  9 15:40:21 host kernel: R10: 0000000000000001 R11: =
0000000000000246 R12: 0000000000000000
> Apr  9 15:40:21 host kernel: R13: 0000000000000000 R14: =
0000000000000000 R15: 0000000000000000
> Apr  9 15:40:21 host kernel: FS:  00007fd75c048710(0000) =
GS:ffff8800022c7000(0000) knlGS:0000000000000000
> Apr  9 15:40:21 host kernel: CS:  e033 DS: 002b ES: 002b CR0: =
000000008005003b
> Apr  9 15:40:21 host kernel: CR2: 000000000040d0e2 CR3: =
0000000071c86000 CR4: 0000000000002660
> Apr  9 15:40:21 host kernel: DR0: 0000000000000000 DR1: =
0000000000000000 DR2: 0000000000000000
> Apr  9 15:40:21 host kernel: DR3: 0000000000000000 DR6: =
00000000ffff0ff0 DR7: 0000000000000400
> Apr  9 15:40:21 host kernel: Call Trace:
> Apr  9 15:40:21 host kernel:  [<ffffffff8100ddb7>] ? =
xen_safe_halt+0xc/0x15
> Apr  9 15:40:21 host kernel:  [<ffffffff8100bdcf>] ? =
xen_idle+0x37/0x40
> Apr  9 15:40:21 host kernel:  [<ffffffff8100fe2e>] ? =
cpu_idle+0x4f/0x82
> Apr  9 15:40:21 host kernel: BUG: soft lockup - CPU#0 stuck for 4104s! =
[swapper:0]
> Apr  9 15:40:21 host kernel: Modules linked in:
> Apr  9 15:40:21 host kernel: CPU 0:
> Apr  9 15:40:21 host kernel: Modules linked in:
> Apr  9 15:40:21 host kernel: Pid: 0, comm: swapper Not tainted =
2.6.32.9xls-domU #2=20
> Apr  9 15:40:21 host kernel: RIP: e030:[<ffffffff810093aa>]  =
[<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
> Apr  9 15:40:21 host kernel: RSP: e02b:ffffffff81691f70  EFLAGS: =
00000246
> Apr  9 15:40:21 host kernel: RAX: 0000000000000000 RBX: =
ffffffff81690000 RCX: ffffffff810093aa
> Apr  9 15:40:21 host kernel: RDX: 0000000000000000 RSI: =
0000000000000000 RDI: 0000000000000001
> Apr  9 15:40:21 host kernel: RBP: ffffffff81896d30 R08: =
0000000000000000 R09: ffffffff8100e3b2
> Apr  9 15:40:21 host kernel: R10: 0000000000000001 R11: =
0000000000000246 R12: ffffffffffffffff
> Apr  9 15:40:21 host kernel: R13: ffffffff818ebf20 R14: =
ffffffff818eec70 R15: 0000000000000000
> Apr  9 15:40:21 host kernel: FS:  00007f07704fe6e0(0000) =
GS:ffff8800022ac000(0000) knlGS:0000000000000000
> Apr  9 15:40:21 host kernel: CS:  e033 DS: 0000 ES: 0000 CR0: =
000000008005003b
> Apr  9 15:40:21 host kernel: CR2: 00000000004c04a8 CR3: =
0000000071c86000 CR4: 0000000000002660
> Apr  9 15:40:21 host kernel: DR0: 0000000000000000 DR1: =
0000000000000000 DR2: 0000000000000000
> Apr  9 15:40:21 host kernel: DR3: 0000000000000000 DR6: =
00000000ffff0ff0 DR7: 0000000000000400
> Apr  9 15:40:21 host kernel: Call Trace:
> Apr  9 15:40:21 host kernel:  [<ffffffff8100ddb7>] ? =
xen_safe_halt+0xc/0x15
> Apr  9 15:40:21 host kernel:  [<ffffffff8100bdcf>] ? =
xen_idle+0x37/0x40
> Apr  9 15:40:21 host kernel:  [<ffffffff8100fe2e>] ? =
cpu_idle+0x4f/0x82
> Apr  9 15:40:21 host kernel:  [<ffffffff818b6c42>] ? =
start_kernel+0x353/0x35f
> Apr  9 16:06:21 host kernel: BUG: soft lockup - CPU#1 stuck for 1452s! =
[swapper:0]
> Apr  9 16:06:21 host kernel: Modules linked in:
> Apr  9 16:06:21 host kernel: CPU 1:
> Apr  9 16:06:21 host kernel: Modules linked in:
> Apr  9 16:06:21 host kernel: Pid: 0, comm: swapper Not tainted =
2.6.32.9xls-domU #2=20
> Apr  9 16:06:21 host kernel: RIP: e030:[<ffffffff810093aa>]  =
[<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
> Apr  9 16:06:21 host kernel: RSP: e02b:ffff88007f8a1f08  EFLAGS: =
00000246
> Apr  9 16:06:21 host kernel: RAX: 0000000000000000 RBX: =
ffff88007f8a0000 RCX: ffffffff810093aa
> Apr  9 16:06:21 host kernel: RDX: 0000000000000000 RSI: =
0000000000000000 RDI: 0000000000000001
> Apr  9 16:06:21 host kernel: RBP: ffffffff81896d30 R08: =
0000000000000000 R09: ffffffff8100e3b2
> Apr  9 16:06:21 host kernel: R10: 0000000000000001 R11: =
0000000000000246 R12: 0000000000000000
> Apr  9 16:06:21 host kernel: R13: 0000000000000000 R14: =
0000000000000000 R15: 0000000000000000
> Apr  9 16:06:21 host kernel: FS:  00007ff39e495710(0000) =
GS:ffff8800022c7000(0000) knlGS:0000000000000000
> Apr  9 16:06:25 host kernel: CS:  e033 DS: 002b ES: 002b CR0: =
000000008005003b
> Apr  9 16:06:25 host kernel: CR2: 00007fbd3c012909 CR3: =
0000000071c86000 CR4: 0000000000002660
> Apr  9 16:06:26 host kernel: DR0: 0000000000000000 DR1: =
0000000000000000 DR2: 0000000000000000
> Apr  9 16:06:26 host kernel: DR3: 0000000000000000 DR6: =
00000000ffff0ff0 DR7: 0000000000000400
> Apr  9 16:06:26 host kernel: Call Trace:
> Apr  9 16:06:26 host kernel:  [<ffffffff8100ddb7>] ? =
xen_safe_halt+0xc/0x15
> Apr  9 16:06:26 host kernel:  [<ffffffff8100bdcf>] ? =
xen_idle+0x37/0x40
> Apr  9 16:06:26 host kernel:  [<ffffffff8100fe2e>] ? =
cpu_idle+0x4f/0x82
> Apr  9 16:06:26 host kernel: BUG: soft lockup - CPU#0 stuck for 1452s! =
[swapper:0]
> Apr  9 16:06:26 host kernel: Modules linked in:
> Apr  9 16:06:26 host kernel: CPU 0:
> Apr  9 16:06:26 host kernel: Modules linked in:
> Apr  9 16:06:26 host kernel: Pid: 0, comm: swapper Not tainted =
2.6.32.9xls-domU #2=20
> Apr  9 16:06:26 host kernel: RIP: e030:[<ffffffff810093aa>]  =
[<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
> Apr  9 16:06:26 host kernel: RSP: e02b:ffffffff81691f70  EFLAGS: =
00000246
> Apr  9 16:06:26 host kernel: RAX: 0000000000000000 RBX: =
ffffffff81690000 RCX: ffffffff810093aa
> Apr  9 16:06:26 host kernel: RDX: 0000000000000000 RSI: =
0000000000000000 RDI: 0000000000000001
> Apr  9 16:06:26 host kernel: RBP: ffffffff81896d30 R08: =
0000000000000000 R09: ffffffff8100e3b2
> Apr  9 16:06:26 host kernel: R10: 0000000000000001 R11: =
0000000000000246 R12: ffffffffffffffff
> Apr  9 16:06:26 host kernel: R13: ffffffff818ebf20 R14: =
ffffffff818eec70 R15: 0000000000000000
> Apr  9 16:06:26 host kernel: FS:  00007f07704fe6e0(0000) =
GS:ffff8800022ac000(0000) knlGS:0000000000000000
> Apr  9 16:06:26 host kernel: CS:  e033 DS: 0000 ES: 0000 CR0: =
000000008005003b
> Apr  9 16:06:26 host kernel: CR2: 00007f09a4413000 CR3: =
000000007ea99000 CR4: 0000000000002660
> Apr  9 16:06:26 host kernel: DR0: 0000000000000000 DR1: =
0000000000000000 DR2: 0000000000000000
> Apr  9 16:06:26 host kernel: DR3: 0000000000000000 DR6: =
00000000ffff0ff0 DR7: 0000000000000400
> Apr  9 16:06:26 host kernel: Call Trace:
> Apr  9 16:06:26 host kernel:  [<ffffffff8100ddb7>] ? =
xen_safe_halt+0xc/0x15
> Apr  9 16:06:26 host kernel:  [<ffffffff8100bdcf>] ? =
xen_idle+0x37/0x40
> Apr  9 16:06:26 host kernel:  [<ffffffff8100fe2e>] ? =
cpu_idle+0x4f/0x82
> Apr  9 16:06:26 host kernel:  [<ffffffff818b6c42>] ? =
start_kernel+0x353/0x35f
> Apr  9 16:17:43 host kernel: BUG: soft lockup - CPU#1 stuck for 637s! =
[httpd:32342]
> Apr  9 16:17:43 host kernel: Modules linked in:
> Apr  9 16:17:43 host kernel: CPU 1:
> Apr  9 16:17:43 host kernel: Modules linked in:
> Apr  9 16:17:43 host kernel: Pid: 32342, comm: httpd Not tainted =
2.6.32.9xls-domU #2=20
> Apr  9 16:17:43 host kernel: RIP: e030:[<ffffffff8100922a>]  =
[<ffffffff8100922a>] hypercall_page+0x22a/0x1001
> Apr  9 16:17:43 host kernel: RSP: e02b:ffff88007955fea0  EFLAGS: =
00000246
> Apr  9 16:17:43 host kernel: RAX: 0000000000030004 RBX: =
0000000000000000 RCX: ffffffff8100922a
> Apr  9 16:17:43 host kernel: RDX: 000000000002ba30 RSI: =
0000000000000000 RDI: 0000000000000000
> Apr  9 16:17:43 host kernel: RBP: 000000000394ca98 R08: =
0000000000033240 R09: 000000310f6de059
> Apr  9 16:17:43 host kernel: R10: 0000000000000001 R11: =
0000000000000246 R12: 0000000000000006
> Apr  9 16:17:43 host kernel: R13: ffff88007955ff58 R14: =
ffff88007f9a8ab0 R15: ffff88007d093b40
> Apr  9 16:17:43 host kernel: FS:  00007ff39e495710(0000) =
GS:ffff8800022c7000(0000) knlGS:0000000000000000
> Apr  9 16:17:43 host kernel: CS:  e033 DS: 0000 ES: 0000 CR0: =
000000008005003b
> Apr  9 16:17:43 host kernel: CR2: 000000000394ca98 CR3: =
000000006adde000 CR4: 0000000000002660
> Apr  9 16:17:43 host kernel: DR0: 0000000000000000 DR1: =
0000000000000000 DR2: 0000000000000000
> Apr  9 16:17:43 host kernel: DR3: 0000000000000000 DR6: =
00000000ffff0ff0 DR7: 0000000000000400
> Apr  9 16:17:43 host kernel: Call Trace:
> Apr  9 16:17:43 host kernel:  [<ffffffff8100dd39>] ? =
xen_force_evtchn_callback+0x9/0xa
> Apr  9 16:17:43 host kernel:  [<ffffffff8100e3b2>] ? =
check_events+0x12/0x20
> Apr  9 16:17:43 host kernel:  [<ffffffff8100e359>] ? =
xen_irq_enable_direct_end+0x0/0x7
> Apr  9 16:17:43 host kernel:  [<ffffffff814da477>] ? =
do_page_fault+0xfb/0x28d
> Apr  9 16:17:43 host kernel:  [<ffffffff814d85b5>] ? =
page_fault+0x25/0x30
> Apr  9 16:17:43 host kernel: BUG: soft lockup - CPU#0 stuck for 637s! =
[swapper:0]
> Apr  9 16:17:43 host kernel: Modules linked in:
> Apr  9 16:17:43 host kernel: CPU 0:
> Apr  9 16:17:43 host kernel: Modules linked in:
> Apr  9 16:17:43 host kernel: Pid: 0, comm: swapper Not tainted =
2.6.32.9xls-domU #2=20
> Apr  9 16:17:43 host kernel: RIP: e030:[<ffffffff810093aa>]  =
[<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
> Apr  9 16:17:44 host kernel: RSP: e02b:ffffffff81691f70  EFLAGS: =
00000246
> Apr  9 16:17:44 host kernel: RAX: 0000000000000000 RBX: =
ffffffff81690000 RCX: ffffffff810093aa
> Apr  9 16:17:44 host kernel: RDX: 0000000000000000 RSI: =
0000000000000000 RDI: 0000000000000001
> Apr  9 16:17:44 host kernel: RBP: ffffffff81896d30 R08: =
0000000000000000 R09: ffffffff8100e3b2
> Apr  9 16:17:44 host kernel: R10: 0000000000000001 R11: =
0000000000000246 R12: ffffffffffffffff
> Apr  9 16:17:44 host kernel: R13: ffffffff818ebf20 R14: =
ffffffff818eec70 R15: 0000000000000000
> Apr  9 16:17:44 host kernel: FS:  00007f07704fe6e0(0000) =
GS:ffff8800022ac000(0000) knlGS:0000000000000000
> Apr  9 16:17:44 host kernel: CS:  e033 DS: 0000 ES: 0000 CR0: =
000000008005003b
> Apr  9 16:17:44 host kernel: CR2: 00000000004505b0 CR3: =
000000007ea99000 CR4: 0000000000002660
> Apr  9 16:17:44 host kernel: DR0: 0000000000000000 DR1: =
0000000000000000 DR2: 0000000000000000
> Apr  9 16:17:44 host kernel: DR3: 0000000000000000 DR6: =
00000000ffff0ff0 DR7: 0000000000000400
> Apr  9 16:17:44 host kernel: Call Trace:
> Apr  9 16:17:44 host kernel:  [<ffffffff8100ddb7>] ? =
xen_safe_halt+0xc/0x15
> Apr  9 16:17:44 host kernel:  [<ffffffff8100bdcf>] ? =
xen_idle+0x37/0x40
> Apr  9 16:17:44 host kernel:  [<ffffffff8100fe2e>] ? =
cpu_idle+0x4f/0x82
> Apr  9 16:17:44 host kernel:  [<ffffffff818b6c42>] ? =
start_kernel+0x353/0x35f
> Apr  9 16:20:22 host kernel: BUG: soft lockup - CPU#1 stuck for 147s! =
[swapper:0]
> Apr  9 16:20:22 host kernel: Modules linked in:
> Apr  9 16:20:22 host kernel: CPU 1:
> Apr  9 16:20:22 host kernel: Modules linked in:
> Apr  9 16:20:22 host kernel: Pid: 0, comm: swapper Not tainted =
2.6.32.9xls-domU #2=20
> Apr  9 16:20:22 host kernel: RIP: e030:[<ffffffff810093aa>]  =
[<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
> Apr  9 16:20:22 host kernel: RSP: e02b:ffff88007f8a1f08  EFLAGS: =
00000246
> Apr  9 16:20:22 host kernel: RAX: 0000000000000000 RBX: =
ffff88007f8a0000 RCX: ffffffff810093aa
> Apr  9 16:20:22 host kernel: RDX: 0000000000000000 RSI: =
0000000000000000 RDI: 0000000000000001
> Apr  9 16:20:22 host kernel: RBP: ffffffff81896d30 R08: =
0000000000000000 R09: ffffffff8100e3b2
> Apr  9 16:20:22 host kernel: R10: 0000000000000001 R11: =
0000000000000246 R12: 0000000000000000
> Apr  9 16:20:22 host kernel: R13: 0000000000000000 R14: =
0000000000000000 R15: 0000000000000000
> Apr  9 16:20:22 host kernel: FS:  00007ff39e495710(0000) =
GS:ffff8800022c7000(0000) knlGS:0000000000000000
> Apr  9 16:20:22 host kernel: CS:  e033 DS: 002b ES: 002b CR0: =
000000008005003b
> Apr  9 16:20:22 host kernel: CR2: 00007ff39aec11a7 CR3: =
0000000071c86000 CR4: 0000000000002660
> Apr  9 16:20:22 host kernel: DR0: 0000000000000000 DR1: =
0000000000000000 DR2: 0000000000000000
> Apr  9 16:20:22 host kernel: DR3: 0000000000000000 DR6: =
00000000ffff0ff0 DR7: 0000000000000400
> Apr  9 16:20:22 host kernel: Call Trace:
> Apr  9 16:20:22 host kernel:  [<ffffffff8100ddb7>] ? =
xen_safe_halt+0xc/0x15
> Apr  9 16:20:22 host kernel:  [<ffffffff8100bdcf>] ? =
xen_idle+0x37/0x40
> Apr  9 16:20:22 host kernel:  [<ffffffff8100fe2e>] ? =
cpu_idle+0x4f/0x82
> Apr  9 16:20:22 host kernel: BUG: soft lockup - CPU#0 stuck for 147s! =
[swapper:0]
> Apr  9 16:20:22 host kernel: Modules linked in:
> Apr  9 16:20:22 host kernel: CPU 0:
> Apr  9 16:20:22 host kernel: Modules linked in:
> Apr  9 16:20:22 host kernel: Pid: 0, comm: swapper Not tainted =
2.6.32.9xls-domU #2=20
> Apr  9 16:20:22 host kernel: RIP: e030:[<ffffffff810093aa>]  =
[<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
> Apr  9 16:20:22 host kernel: RSP: e02b:ffffffff81691f70  EFLAGS: =
00000246
> Apr  9 16:20:22 host kernel: RAX: 0000000000000000 RBX: =
ffffffff81690000 RCX: ffffffff810093aa
> Apr  9 16:20:22 host kernel: RDX: 0000000000000000 RSI: =
0000000000000000 RDI: 0000000000000001
> Apr  9 16:20:22 host kernel: RBP: ffffffff81896d30 R08: =
0000000000000000 R09: ffffffff8100e3b2
> Apr  9 16:20:22 host kernel: R10: 0000000000000001 R11: =
0000000000000246 R12: ffffffffffffffff
> Apr  9 16:20:22 host kernel: R13: ffffffff818ebf20 R14: =
ffffffff818eec70 R15: 0000000000000000
> Apr  9 16:20:22 host kernel: FS:  00007f07704fe6e0(0000) =
GS:ffff8800022ac000(0000) knlGS:0000000000000000
> Apr  9 16:20:22 host kernel: CS:  e033 DS: 0000 ES: 0000 CR0: =
000000008005003b
> Apr  9 16:20:22 host kernel: CR2: 00000000006ddc20 CR3: =
0000000071c86000 CR4: 0000000000002660
> Apr  9 16:20:22 host kernel: DR0: 0000000000000000 DR1: =
0000000000000000 DR2: 0000000000000000
> Apr  9 16:20:22 host kernel: DR3: 0000000000000000 DR6: =
00000000ffff0ff0 DR7: 0000000000000400
> Apr  9 16:20:22 host kernel: Call Trace:
> Apr  9 16:20:22 host kernel:  [<ffffffff8100ddb7>] ? =
xen_safe_halt+0xc/0x15
> Apr  9 16:20:22 host kernel:  [<ffffffff8100bdcf>] ? =
xen_idle+0x37/0x40
> Apr  9 16:20:22 host kernel:  [<ffffffff8100fe2e>] ? =
cpu_idle+0x4f/0x82
> Apr  9 16:20:22 host kernel:  [<ffffffff818b6c42>] ? =
start_kernel+0x353/0x35f
> Apr  9 16:22:29 host kernel: BUG: soft lockup - CPU#1 stuck for 119s! =
[swapper:0]
> Apr  9 16:22:29 host kernel: Modules linked in:
> Apr  9 16:22:29 host kernel: CPU 1:
> Apr  9 16:22:29 host kernel: Modules linked in:
> Apr  9 16:22:29 host kernel: Pid: 0, comm: swapper Not tainted =
2.6.32.9xls-domU #2=20
> Apr  9 16:22:29 host kernel: RIP: e030:[<ffffffff810093aa>]  =
[<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
> Apr  9 16:22:29 host kernel: RSP: e02b:ffff88007f8a1f08  EFLAGS: =
00000246
> Apr  9 16:22:29 host kernel: RAX: 0000000000000000 RBX: =
ffff88007f8a0000 RCX: ffffffff810093aa
> Apr  9 16:22:29 host kernel: RDX: 0000000000000000 RSI: =
0000000000000000 RDI: 0000000000000001
> Apr  9 16:22:29 host kernel: RBP: ffffffff81896d30 R08: =
0000000000000000 R09: ffffffff8100e3b2
> Apr  9 16:22:29 host kernel: R10: 0000000000000001 R11: =
0000000000000246 R12: 0000000000000000
> Apr  9 16:22:29 host kernel: R13: 0000000000000000 R14: =
0000000000000000 R15: 0000000000000000
> Apr  9 16:22:29 host kernel: FS:  00007ff39e495710(0000) =
GS:ffff8800022c7000(0000) knlGS:0000000000000000
> Apr  9 16:22:29 host kernel: CS:  e033 DS: 002b ES: 002b CR0: =
000000008005003b
> Apr  9 16:22:29 host kernel: CR2: 00007ff39c506460 CR3: =
0000000026469000 CR4: 0000000000002660
> Apr  9 16:22:29 host kernel: DR0: 0000000000000000 DR1: =
0000000000000000 DR2: 0000000000000000
> Apr  9 16:22:29 host kernel: DR3: 0000000000000000 DR6: =
00000000ffff0ff0 DR7: 0000000000000400
> Apr  9 16:22:29 host kernel: Call Trace:
> Apr  9 16:22:29 host kernel:  [<ffffffff8100ddb7>] ? =
xen_safe_halt+0xc/0x15
> Apr  9 16:22:29 host kernel:  [<ffffffff8100bdcf>] ? =
xen_idle+0x37/0x40
> Apr  9 16:22:29 host kernel:  [<ffffffff8100fe2e>] ? =
cpu_idle+0x4f/0x82
> Apr  9 16:22:29 host kernel: BUG: soft lockup - CPU#0 stuck for 119s! =
[syslogd:1848]
> Apr  9 16:22:29 host kernel: Modules linked in:
> Apr  9 16:22:29 host kernel: CPU 0:
> Apr  9 16:22:29 host kernel: Modules linked in:
> Apr  9 16:22:29 host kernel: Pid: 1848, comm: syslogd Not tainted =
2.6.32.9xls-domU #2=20
> Apr  9 16:22:29 host kernel: RIP: 0033:[<00007f74fd9e4ad5>]  =
[<00007f74fd9e4ad5>] 0x7f74fd9e4ad5
> Apr  9 16:22:29 host kernel: RSP: 002b:00007fff22fa00d0  EFLAGS: =
00000246
> Apr  9 16:22:29 host kernel: RAX: 000000000000002e RBX: =
0000000000000000 RCX: ffffffff810092eb
> Apr  9 16:22:29 host kernel: RDX: 00000000000003fe RSI: =
00007fff22fa01b0 RDI: 0000000000000000
> Apr  9 16:22:30 host kernel: RBP: 00000000000003fe R08: =
0000000000000000 R09: 0000000000000000
> Apr  9 16:22:30 host kernel: R10: 0000000000000000 R11: =
0000000000000246 R12: 00007fff22fa01b0
> Apr  9 16:22:30 host kernel: R13: 0000000000000000 R14: =
00007fff22fa0120 R15: 00007f74fde74690
> Apr  9 16:22:30 host kernel: FS:  00007f74fde746e0(0000) =
GS:ffff8800022ac000(0000) knlGS:0000000000000000
> Apr  9 16:22:30 host kernel: CS:  e033 DS: 0000 ES: 0000 CR0: =
000000008005003b
> Apr  9 16:22:30 host kernel: CR2: 00007f0e429d6000 CR3: =
000000007d03a000 CR4: 0000000000002660
> Apr  9 16:22:30 host kernel: DR0: 0000000000000000 DR1: =
0000000000000000 DR2: 0000000000000000
> Apr  9 16:22:30 host kernel: DR3: 0000000000000000 DR6: =
00000000ffff0ff0 DR7: 0000000000000400
> Apr  9 16:22:30 host kernel: Call Trace:


On Apr 9, 2010, at 15:22 , Pim van Riezen wrote:

> Good day,
>=20
> I'm getting lots of these from a couple of pvops DomU guests that are =
pretty heavy in their i/o and cpu-load. The issue reproduces on pvops =
2.6.31 and 2.6.32 trees under Xen 3.3 as well as 3.4.1 (in the process =
of trying out 3.4.2 but I'm not holding my breath). Is there anything we =
can do be more helpful towards solving this issue?
>=20
> BUG: soft lockup - CPU#0 stuck for 561s! [swapper:0]
> Modules linked in:
> CPU 0:
> Modules linked in:
> Pid: 0, comm: swapper Not tainted 2.6.32.9xls-domU #2=20
> RIP: e030:[<ffffffff810093aa>]  [<ffffffff810093aa>] =
hypercall_page+0x3aa/0x1001
> RSP: e02b:ffffffff81691f70  EFLAGS: 00000246
> RAX: 0000000000000000 RBX: ffffffff81690000 RCX: ffffffff810093aa
> RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000001
> RBP: ffffffff81896d30 R08: 0000000000000000 R09: ffffffff8100e3b2
> R10: 0000000000000001 R11: 0000000000000246 R12: ffffffffffffffff
> R13: ffffffff818ebf20 R14: ffffffff818eec70 R15: 0000000000000000
> FS:  00007f8ac7a9c6e0(0000) GS:ffff8800022ac000(0000) =
knlGS:0000000000000000
> CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
> CR2: 00007f291b4c9000 CR3: 000000007d8c1000 CR4: 0000000000002660
> DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> Call Trace:
>  [<ffffffff8100ddb7>] ? xen_safe_halt+0xc/0x15
>  [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40
>  [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82
>  [<ffffffff818b6c42>] ? start_kernel+0x353/0x35f
>=20
> Kind regards,
> Pim van Riezen
>=20
>=20
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel