* several domU crashes after 4.1->4.8 live migration
@ 2017-02-02 7:58 Vincent Legout
2017-02-02 11:22 ` George Dunlap
2017-02-02 12:05 ` Andrew Cooper
0 siblings, 2 replies; 8+ messages in thread
From: Vincent Legout @ 2017-02-02 7:58 UTC (permalink / raw)
To: xen-devel
[-- Attachment #1: Type: text/plain, Size: 1207 bytes --]
Hello,
We had some issues after live migrating several domU from xen 4.1 to xen
4.8. We migrated around 200 domU and 5 crashed, from a few hours up to
several days after the migration. All the domU had more than 1 year of
uptime, and for example one crashed several days after the migration
during a high load period.
All 5 domU are running a 3.10 kernel (from 3.10.44 to 3.10.103). They
have between 2GB and 16GB of RAM, and between 1 and 4 vCPUS.
We use 3 types of machines (several PowerEdge C6100 (Intel L5640) and
R710 (Intel L5520), and one C8220 (Intel E5-2650)). The C6100 and R710
have 24 logical cores with HT enabled. The PowerEdge C8220 is only used
for Xen 4.8, and has 32 logical cores with HT enabled. The C6100 and the
R710 have 50GB of RAM, and the C8220 128GB. The xen 4.1 dom0 is running
a 3.4.69 kernel, and the xen 4.8 one a 4.1.37 kernel.
I've attached the most relevant parts of the domU kernel logs we could
get. It seems the crashes came from different components of the kernel,
though most of them seem to be related to memory.
Would anyone have any idea if that's something that could be fixed? Or
is it just that migrating from 4.1 to 4.8 is not supported?
Thanks,
Vincent
[-- Attachment #2: domu1 --]
[-- Type: text/plain, Size: 24103 bytes --]
xen:grant_table: Grant tables using version 2 layout
PM: noirq restore of devices complete after 0.034 msecs
PM: early restore of devices complete after 0.012 msecs
BUG: Bad page state in process xenwatch pfn:0afb5
page:ffffea0000266f98 count:0 mapcount:0 mapping: (null) index:0x0
page flags: 0x4000000000090028(uptodate|lru|swapcache|swapbacked)
Modules linked in: crc32c_intel tcp_diag inet_diag
CPU: 0 PID: 76 Comm: xenwatch Not tainted 3.10.103-xenU-corp-33-adcc8c8-x86_64 #6
ffffffff813d282e ffffffff810d26b3 ffffea0000266fd0 ffffffff810d27a5
ffffea0000266f98 ffff880074cd8000 ffff8800751647a0 ffffffff810d4528
ffff880074d41ce0 ffff880074cd8000 ffff8800751647a0 ffff880074cdc6b8
Call Trace:
[<ffffffff813d282e>] ? dump_stack+0xc/0x15
[<ffffffff810d26b3>] ? bad_page+0xa3/0xf0
[<ffffffff810d27a5>] ? free_pages_prepare+0xa5/0xd0
[<ffffffff810d4528>] ? free_hot_cold_page+0x18/0x170
[<ffffffff812b0e17>] ? blkif_free+0xf7/0x190
[<ffffffff812b12f3>] ? blkfront_resume+0x23/0x30
[<ffffffff8127ea75>] ? xenbus_dev_resume+0x45/0xf0
[<ffffffff8127ea30>] ? xenbus_dev_remove+0x70/0x70
[<ffffffff812ab026>] ? dpm_run_callback.isra.11+0x16/0x50
[<ffffffff812aba62>] ? device_resume+0x82/0x120
[<ffffffff812ac09c>] ? dpm_resume+0xdc/0x200
[<ffffffff812ac378>] ? dpm_resume_end+0x8/0x10
[<ffffffff81276482>] ? do_suspend+0xf2/0x1c0
[<ffffffff81276370>] ? xen_post_suspend+0x10/0x10
[<ffffffff81276360>] ? xen_suspend+0xb0/0xb0
[<ffffffff8127627f>] ? shutdown_handler+0x11f/0x150
[<ffffffff8127d7a0>] ? xenwatch_thread+0xb0/0x160
[<ffffffff81068b40>] ? wake_up_bit+0x20/0x20
[<ffffffff8127d6f0>] ? unregister_xenbus_watch+0x200/0x200
[<ffffffff81067f42>] ? kthread+0xc2/0xd0
[<ffffffff81067e80>] ? kthread_create_on_node+0x120/0x120
[<ffffffff813d6618>] ? ret_from_fork+0x58/0x90
[<ffffffff81067e80>] ? kthread_create_on_node+0x120/0x120
Disabling lock debugging due to kernel taint
BUG: Bad page state in process xenwatch pfn:64171
page:ffffea00015e50b8 count:0 mapcount:0 mapping: (null) index:0x0
page flags: 0x4000000000090028(uptodate|lru|swapcache|swapbacked)
Modules linked in: crc32c_intel tcp_diag inet_diag
CPU: 0 PID: 76 Comm: xenwatch Tainted: G B 3.10.103-xenU-corp-33-adcc8c8-x86_64 #6
ffffffff813d282e ffffffff810d26b3 ffffea00015e50f0 ffffffff810d27a5
ffffea00015e50b8 ffff880074cd8000 ffff8800751648c0 ffffffff810d4528
ffff8800751647a0 ffff880074cd8000 ffff8800751648c0 ffff880074cdc6b8
Call Trace:
[<ffffffff813d282e>] ? dump_stack+0xc/0x15
[<ffffffff810d26b3>] ? bad_page+0xa3/0xf0
[<ffffffff810d27a5>] ? free_pages_prepare+0xa5/0xd0
[<ffffffff810d4528>] ? free_hot_cold_page+0x18/0x170
[<ffffffff812b0e17>] ? blkif_free+0xf7/0x190
[<ffffffff812b12f3>] ? blkfront_resume+0x23/0x30
[<ffffffff8127ea75>] ? xenbus_dev_resume+0x45/0xf0
[<ffffffff8127ea30>] ? xenbus_dev_remove+0x70/0x70
[<ffffffff812ab026>] ? dpm_run_callback.isra.11+0x16/0x50
[<ffffffff812aba62>] ? device_resume+0x82/0x120
[<ffffffff812ac09c>] ? dpm_resume+0xdc/0x200
[<ffffffff812ac378>] ? dpm_resume_end+0x8/0x10
[<ffffffff81276482>] ? do_suspend+0xf2/0x1c0
[<ffffffff81276370>] ? xen_post_suspend+0x10/0x10
[<ffffffff81276360>] ? xen_suspend+0xb0/0xb0
[<ffffffff8127627f>] ? shutdown_handler+0x11f/0x150
[<ffffffff8127d7a0>] ? xenwatch_thread+0xb0/0x160
[<ffffffff81068b40>] ? wake_up_bit+0x20/0x20
[<ffffffff8127d6f0>] ? unregister_xenbus_watch+0x200/0x200
[<ffffffff81067f42>] ? kthread+0xc2/0xd0
[<ffffffff81067e80>] ? kthread_create_on_node+0x120/0x120
[<ffffffff813d6618>] ? ret_from_fork+0x58/0x90
[<ffffffff81067e80>] ? kthread_create_on_node+0x120/0x120
BUG: Bad page state in process xenwatch pfn:449c2
page:ffffea0000f02270 count:0 mapcount:0 mapping: (null) index:0x0
page flags: 0x4000000000090028(uptodate|lru|swapcache|swapbacked)
Modules linked in: crc32c_intel tcp_diag inet_diag
CPU: 0 PID: 76 Comm: xenwatch Tainted: G B 3.10.103-xenU-corp-33-adcc8c8-x86_64 #6
ffffffff813d282e ffffffff810d26b3 ffffea0000f022a8 ffffffff810d27a5
ffffea0000f02270 ffff880074cd8000 ffff8800751647e0 ffffffff810d4528
ffff8800751648c0 ffff880074cd8000 ffff8800751647e0 ffff880074cdc6b8
Call Trace:
[<ffffffff813d282e>] ? dump_stack+0xc/0x15
[<ffffffff810d26b3>] ? bad_page+0xa3/0xf0
[<ffffffff810d27a5>] ? free_pages_prepare+0xa5/0xd0
[<ffffffff810d4528>] ? free_hot_cold_page+0x18/0x170
[<ffffffff812b0e17>] ? blkif_free+0xf7/0x190
[<ffffffff812b12f3>] ? blkfront_resume+0x23/0x30
[<ffffffff8127ea75>] ? xenbus_dev_resume+0x45/0xf0
[<ffffffff8127ea30>] ? xenbus_dev_remove+0x70/0x70
[<ffffffff812ab026>] ? dpm_run_callback.isra.11+0x16/0x50
[<ffffffff812aba62>] ? device_resume+0x82/0x120
[<ffffffff812ac09c>] ? dpm_resume+0xdc/0x200
[<ffffffff812ac378>] ? dpm_resume_end+0x8/0x10
[<ffffffff81276482>] ? do_suspend+0xf2/0x1c0
[<ffffffff81276370>] ? xen_post_suspend+0x10/0x10
[<ffffffff81276360>] ? xen_suspend+0xb0/0xb0
[<ffffffff8127627f>] ? shutdown_handler+0x11f/0x150
[<ffffffff8127d7a0>] ? xenwatch_thread+0xb0/0x160
[<ffffffff81068b40>] ? wake_up_bit+0x20/0x20
[<ffffffff8127d6f0>] ? unregister_xenbus_watch+0x200/0x200
[<ffffffff81067f42>] ? kthread+0xc2/0xd0
[<ffffffff81067e80>] ? kthread_create_on_node+0x120/0x120
[<ffffffff813d6618>] ? ret_from_fork+0x58/0x90
[<ffffffff81067e80>] ? kthread_create_on_node+0x120/0x120
BUG: Bad page state in process xenwatch pfn:2a61d
page:ffffea0000945658 count:0 mapcount:0 mapping: (null) index:0x0
page flags: 0x4000000000090028(uptodate|lru|swapcache|swapbacked)
Modules linked in: crc32c_intel tcp_diag inet_diag
CPU: 0 PID: 76 Comm: xenwatch Tainted: G B 3.10.103-xenU-corp-33-adcc8c8-x86_64 #6
ffffffff813d282e ffffffff810d26b3 ffffea0000945690 ffffffff810d27a5
ffffea0000945658 ffff880074cd8000 ffff880075164bc0 ffffffff810d4528
ffff8800751647e0 ffff880074cd8000 ffff880075164bc0 ffff880074cdc6b8
Call Trace:
[<ffffffff813d282e>] ? dump_stack+0xc/0x15
[<ffffffff810d26b3>] ? bad_page+0xa3/0xf0
[<ffffffff810d27a5>] ? free_pages_prepare+0xa5/0xd0
[<ffffffff810d4528>] ? free_hot_cold_page+0x18/0x170
[<ffffffff812b0e17>] ? blkif_free+0xf7/0x190
[<ffffffff812b12f3>] ? blkfront_resume+0x23/0x30
[<ffffffff8127ea75>] ? xenbus_dev_resume+0x45/0xf0
[<ffffffff8127ea30>] ? xenbus_dev_remove+0x70/0x70
[<ffffffff812ab026>] ? dpm_run_callback.isra.11+0x16/0x50
[<ffffffff812aba62>] ? device_resume+0x82/0x120
[<ffffffff812ac09c>] ? dpm_resume+0xdc/0x200
[<ffffffff812ac378>] ? dpm_resume_end+0x8/0x10
[<ffffffff81276482>] ? do_suspend+0xf2/0x1c0
[<ffffffff81276370>] ? xen_post_suspend+0x10/0x10
[<ffffffff81276360>] ? xen_suspend+0xb0/0xb0
[<ffffffff8127627f>] ? shutdown_handler+0x11f/0x150
[<ffffffff8127d7a0>] ? xenwatch_thread+0xb0/0x160
[<ffffffff81068b40>] ? wake_up_bit+0x20/0x20
[<ffffffff8127d6f0>] ? unregister_xenbus_watch+0x200/0x200
[<ffffffff81067f42>] ? kthread+0xc2/0xd0
[<ffffffff81067e80>] ? kthread_create_on_node+0x120/0x120
[<ffffffff813d6618>] ? ret_from_fork+0x58/0x90
[<ffffffff81067e80>] ? kthread_create_on_node+0x120/0x120
BUG: Bad page state in process xenwatch pfn:1910f
page:ffffea000057bb48 count:0 mapcount:1 mapping: (null) index:0x7f767c003
page flags: 0x4000000000080038(uptodate|dirty|lru|swapbacked)
Modules linked in: crc32c_intel tcp_diag inet_diag
CPU: 0 PID: 76 Comm: xenwatch Tainted: G B 3.10.103-xenU-corp-33-adcc8c8-x86_64 #6
ffffffff813d282e ffffffff810d26b3 ffffea000057bb80 ffffffff810d27a5
ffffea000057bb48 ffff880074cd8000 ffff8800751643c0 ffffffff810d4528
ffff880075164bc0 ffff880074cd8000 ffff8800751643c0 ffff880074cdc6b8
Call Trace:
[<ffffffff813d282e>] ? dump_stack+0xc/0x15
[<ffffffff810d26b3>] ? bad_page+0xa3/0xf0
[<ffffffff810d27a5>] ? free_pages_prepare+0xa5/0xd0
[<ffffffff810d4528>] ? free_hot_cold_page+0x18/0x170
[<ffffffff812b0e17>] ? blkif_free+0xf7/0x190
[<ffffffff812b12f3>] ? blkfront_resume+0x23/0x30
[<ffffffff8127ea75>] ? xenbus_dev_resume+0x45/0xf0
[<ffffffff8127ea30>] ? xenbus_dev_remove+0x70/0x70
[<ffffffff812ab026>] ? dpm_run_callback.isra.11+0x16/0x50
[<ffffffff812aba62>] ? device_resume+0x82/0x120
[<ffffffff812ac09c>] ? dpm_resume+0xdc/0x200
[<ffffffff812ac378>] ? dpm_resume_end+0x8/0x10
[<ffffffff81276482>] ? do_suspend+0xf2/0x1c0
[<ffffffff81276370>] ? xen_post_suspend+0x10/0x10
[<ffffffff81276360>] ? xen_suspend+0xb0/0xb0
[<ffffffff8127627f>] ? shutdown_handler+0x11f/0x150
[<ffffffff8127d7a0>] ? xenwatch_thread+0xb0/0x160
[<ffffffff81068b40>] ? wake_up_bit+0x20/0x20
[<ffffffff8127d6f0>] ? unregister_xenbus_watch+0x200/0x200
[<ffffffff81067f42>] ? kthread+0xc2/0xd0
[<ffffffff81067e80>] ? kthread_create_on_node+0x120/0x120
[<ffffffff813d6618>] ? ret_from_fork+0x58/0x90
[<ffffffff81067e80>] ? kthread_create_on_node+0x120/0x120
BUG: Bad page state in process xenwatch pfn:493a9
page:ffffea0001004cf8 count:0 mapcount:0 mapping: (null) index:0x0
page flags: 0x4000000000090028(uptodate|lru|swapcache|swapbacked)
Modules linked in: crc32c_intel tcp_diag inet_diag
CPU: 0 PID: 76 Comm: xenwatch Tainted: G B 3.10.103-xenU-corp-33-adcc8c8-x86_64 #6
ffffffff813d282e ffffffff810d26b3 ffffea0001004d30 ffffffff810d27a5
ffffea0001004cf8 ffff880074cd8000 ffff8800751643e0 ffffffff810d4528
ffff8800751643c0 ffff880074cd8000 ffff8800751643e0 ffff880074cdc6b8
Call Trace:
[<ffffffff813d282e>] ? dump_stack+0xc/0x15
[<ffffffff810d26b3>] ? bad_page+0xa3/0xf0
[<ffffffff810d27a5>] ? free_pages_prepare+0xa5/0xd0
[<ffffffff810d4528>] ? free_hot_cold_page+0x18/0x170
[<ffffffff812b0e17>] ? blkif_free+0xf7/0x190
[<ffffffff812b12f3>] ? blkfront_resume+0x23/0x30
[<ffffffff8127ea75>] ? xenbus_dev_resume+0x45/0xf0
[<ffffffff8127ea30>] ? xenbus_dev_remove+0x70/0x70
[<ffffffff812ab026>] ? dpm_run_callback.isra.11+0x16/0x50
[<ffffffff812aba62>] ? device_resume+0x82/0x120
[<ffffffff812ac09c>] ? dpm_resume+0xdc/0x200
[<ffffffff812ac378>] ? dpm_resume_end+0x8/0x10
[<ffffffff81276482>] ? do_suspend+0xf2/0x1c0
[<ffffffff81276370>] ? xen_post_suspend+0x10/0x10
[<ffffffff81276360>] ? xen_suspend+0xb0/0xb0
[<ffffffff8127627f>] ? shutdown_handler+0x11f/0x150
[<ffffffff8127d7a0>] ? xenwatch_thread+0xb0/0x160
[<ffffffff81068b40>] ? wake_up_bit+0x20/0x20
[<ffffffff8127d6f0>] ? unregister_xenbus_watch+0x200/0x200
[<ffffffff81067f42>] ? kthread+0xc2/0xd0
[<ffffffff81067e80>] ? kthread_create_on_node+0x120/0x120
[<ffffffff813d6618>] ? ret_from_fork+0x58/0x90
[<ffffffff81067e80>] ? kthread_create_on_node+0x120/0x120
BUG: Bad page state in process xenwatch pfn:3d6f2
page:ffffea0000d704f0 count:0 mapcount:0 mapping: (null) index:0x0
page flags: 0x4000000000090028(uptodate|lru|swapcache|swapbacked)
Modules linked in: crc32c_intel tcp_diag inet_diag
CPU: 0 PID: 76 Comm: xenwatch Tainted: G B 3.10.103-xenU-corp-33-adcc8c8-x86_64 #6
ffffffff813d282e ffffffff810d26b3 ffffea0000d70528 ffffffff810d27a5
ffffea0000d704f0 ffff880074cd8000 ffff880075164380 ffffffff810d4528
ffff8800751643e0 ffff880074cd8000 ffff880075164380 ffff880074cdc6b8
Call Trace:
[<ffffffff813d282e>] ? dump_stack+0xc/0x15
[<ffffffff810d26b3>] ? bad_page+0xa3/0xf0
[<ffffffff810d27a5>] ? free_pages_prepare+0xa5/0xd0
[<ffffffff810d4528>] ? free_hot_cold_page+0x18/0x170
[<ffffffff812b0e17>] ? blkif_free+0xf7/0x190
[<ffffffff812b12f3>] ? blkfront_resume+0x23/0x30
[<ffffffff8127ea75>] ? xenbus_dev_resume+0x45/0xf0
[<ffffffff8127ea30>] ? xenbus_dev_remove+0x70/0x70
[<ffffffff812ab026>] ? dpm_run_callback.isra.11+0x16/0x50
[<ffffffff812aba62>] ? device_resume+0x82/0x120
[<ffffffff812ac09c>] ? dpm_resume+0xdc/0x200
[<ffffffff812ac378>] ? dpm_resume_end+0x8/0x10
[<ffffffff81276482>] ? do_suspend+0xf2/0x1c0
[<ffffffff81276370>] ? xen_post_suspend+0x10/0x10
[<ffffffff81276360>] ? xen_suspend+0xb0/0xb0
[<ffffffff8127627f>] ? shutdown_handler+0x11f/0x150
[<ffffffff8127d7a0>] ? xenwatch_thread+0xb0/0x160
[<ffffffff81068b40>] ? wake_up_bit+0x20/0x20
[<ffffffff8127d6f0>] ? unregister_xenbus_watch+0x200/0x200
[<ffffffff81067f42>] ? kthread+0xc2/0xd0
[<ffffffff81067e80>] ? kthread_create_on_node+0x120/0x120
[<ffffffff813d6618>] ? ret_from_fork+0x58/0x90
[<ffffffff81067e80>] ? kthread_create_on_node+0x120/0x120
BUG: Bad page state in process xenwatch pfn:2ac94
page:ffffea000095c060 count:0 mapcount:0 mapping: (null) index:0x0
page flags: 0x4000000000090028(uptodate|lru|swapcache|swapbacked)
Modules linked in: crc32c_intel tcp_diag inet_diag
CPU: 0 PID: 76 Comm: xenwatch Tainted: G B 3.10.103-xenU-corp-33-adcc8c8-x86_64 #6
ffffffff813d282e ffffffff810d26b3 ffffea000095c098 ffffffff810d27a5
ffffea000095c060 ffff880074cd8000 ffff8800752d6a80 ffffffff810d4528
ffff880075164380 ffff880074cd8000 ffff8800752d6a80 ffff880074cdc6b8
Call Trace:
[<ffffffff813d282e>] ? dump_stack+0xc/0x15
[<ffffffff810d26b3>] ? bad_page+0xa3/0xf0
[<ffffffff810d27a5>] ? free_pages_prepare+0xa5/0xd0
[<ffffffff810d4528>] ? free_hot_cold_page+0x18/0x170
[<ffffffff812b0e17>] ? blkif_free+0xf7/0x190
[<ffffffff812b12f3>] ? blkfront_resume+0x23/0x30
[<ffffffff8127ea75>] ? xenbus_dev_resume+0x45/0xf0
[<ffffffff8127ea30>] ? xenbus_dev_remove+0x70/0x70
[<ffffffff812ab026>] ? dpm_run_callback.isra.11+0x16/0x50
[<ffffffff812aba62>] ? device_resume+0x82/0x120
[<ffffffff812ac09c>] ? dpm_resume+0xdc/0x200
[<ffffffff812ac378>] ? dpm_resume_end+0x8/0x10
[<ffffffff81276482>] ? do_suspend+0xf2/0x1c0
[<ffffffff81276370>] ? xen_post_suspend+0x10/0x10
[<ffffffff81276360>] ? xen_suspend+0xb0/0xb0
[<ffffffff8127627f>] ? shutdown_handler+0x11f/0x150
[<ffffffff8127d7a0>] ? xenwatch_thread+0xb0/0x160
[<ffffffff81068b40>] ? wake_up_bit+0x20/0x20
[<ffffffff8127d6f0>] ? unregister_xenbus_watch+0x200/0x200
[<ffffffff81067f42>] ? kthread+0xc2/0xd0
[<ffffffff81067e80>] ? kthread_create_on_node+0x120/0x120
[<ffffffff813d6618>] ? ret_from_fork+0x58/0x90
[<ffffffff81067e80>] ? kthread_create_on_node+0x120/0x120
PM: restore of devices complete after 402.018 msecs
Setting capacity to 41943040
INFO: rcu_sched self-detected stall on CPU { 0} (t=15000 jiffies g=99356660 c=99356659 q=22)
sending NMI to all CPUs:
NMI backtrace for cpu 0
CPU: 0 PID: 1464 Comm: in:imklog Tainted: G B 3.10.103-xenU-corp-33-adcc8c8-x86_64 #6
task: ffff880074c91180 ti: ffff880043496000 task.ti: ffff880043496000
RIP: e030:[<ffffffff8100130a>] [<ffffffff8100130a>] xen_hypercall_vcpu_op+0xa/0x20
RSP: e02b:ffff88007ee03cd0 EFLAGS: 00000046
RAX: 0000000000000000 RBX: 0000000000000000 RCX: ffffffff8100130a
RDX: 0000000000000000 RSI: 0000000000000000 RDI: 000000000000000b
RBP: ffffffff81658620 R08: ffffffff81658620 R09: 0000000000000000
R10: ffff88007ee0d1c0 R11: 0000000000000246 R12: ffffffff81658620
R13: 0000000000000005 R14: 0000000000000016 R15: ffff88007ee0d1c0
FS: 00007f768b268700(0000) GS:ffff88007ee00000(0000) knlGS:0000000000000000
CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007f768b249000 CR3: 0000000075066000 CR4: 0000000000002660
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
0720072007200720 0000000000000000 ffffffff8127a1ef ffffffff8100919a
0000000000002710 ffffffff81658628 ffffffff81628a00 ffffffff81628a00
ffffffff810277a7 ffff880043496000 ffffffff810bbfdd 0000000000000000
Call Trace:
<IRQ>
[<ffffffff8127a1ef>] ? xen_send_IPI_one+0x3f/0x60
[<ffffffff8100919a>] ? __xen_send_IPI_mask+0x2a/0x50
[<ffffffff810277a7>] ? arch_trigger_all_cpu_backtrace+0x47/0x80
[<ffffffff810bbfdd>] ? rcu_check_callbacks+0x2ed/0x580
[<ffffffff8108edf0>] ? tick_sched_handle.isra.17+0x30/0x30
[<ffffffff8105613b>] ? update_process_times+0x3b/0x70
[<ffffffff8108ee24>] ? tick_sched_timer+0x34/0x50
[<ffffffff8106b213>] ? __run_hrtimer.isra.32+0x43/0xd0
[<ffffffff8106b87f>] ? hrtimer_interrupt+0xef/0x250
[<ffffffff8100793a>] ? xen_timer_interrupt+0x2a/0x150
[<ffffffff8125b1dc>] ? radix_tree_lookup_element+0x3c/0x90
[<ffffffff810b5549>] ? handle_irq_event_percpu+0x89/0x140
[<ffffffff810b8403>] ? handle_percpu_irq+0x33/0x50
[<ffffffff81278e04>] ? __xen_evtchn_do_upcall+0x354/0x360
[<ffffffff8104fa6a>] ? __do_softirq+0x17a/0x1f0
[<ffffffff8127a692>] ? xen_evtchn_do_upcall+0x22/0x40
[<ffffffff813d7c3e>] ? xen_do_hypervisor_callback+0x1e/0x30
<EOI>
[<ffffffff8125b1d0>] ? radix_tree_lookup_element+0x30/0x90
[<ffffffff810cc374>] ? find_get_page+0x14/0x80
[<ffffffff810ff15f>] ? lookup_swap_cache+0x1f/0x40
[<ffffffff810ed557>] ? handle_pte_fault+0x107/0xa40
[<ffffffff81004659>] ? __raw_callee_save_xen_pmd_val+0x11/0x1e
[<ffffffff810ef601>] ? handle_mm_fault+0x391/0x430
[<ffffffff8102f7f2>] ? __do_page_fault+0x172/0x4a0
[<ffffffff8100440a>] ? xen_mc_flush+0x10a/0x140
[<ffffffff81003fbb>] ? xen_write_msr_safe+0x7b/0xc0
[<ffffffff8100a628>] ? __switch_to+0x1e8/0x500
[<ffffffff81261615>] ? vsnprintf+0x315/0x690
[<ffffffff813d6198>] ? page_fault+0x28/0x30
[<ffffffff81261e00>] ? copy_user_generic_string+0x30/0x40
[<ffffffff8104a89c>] ? do_syslog+0x1bc/0x570
[<ffffffff81068b40>] ? wake_up_bit+0x20/0x20
[<ffffffff81180799>] ? kmsg_read+0x39/0x50
[<ffffffff8117393e>] ? proc_reg_read+0x3e/0x60
[<ffffffff81119cf5>] ? vfs_read+0x95/0x160
[<ffffffff8111a782>] ? SyS_read+0x42/0x90
[<ffffffff813d66cd>] ? system_call_fastpath+0x1a/0x1f
Code: cc 51 41 53 50 b8 17 00 00 00 0f 05 5b 59 c3 cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc 51 41 53 b8 18 00 00 00 0f 05 <41> 5b 59 c3 cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc
NMI backtrace for cpu 1
CPU: 1 PID: 0 Comm: swapper/1 Tainted: G B 3.10.103-xenU-corp-33-adcc8c8-x86_64 #6
task: ffff880076873140 ti: ffff880075524000 task.ti: ffff880075524000
RIP: e030:[<ffffffff810013aa>] [<ffffffff810013aa>] xen_hypercall_sched_op+0xa/0x20
RSP: e02b:ffff880075525f00 EFLAGS: 00000246
RAX: 0000000000000000 RBX: ffffffff81658620 RCX: ffffffff810013aa
RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000001
RBP: ffff880075525fd8 R08: 0000000000000000 R09: 0000000000000000
R10: 0001401440aac14d R11: 0000000000000246 R12: ffff880075525fd8
R13: ffff880075525fd8 R14: ffff880075525fd8 R15: ffff880075525fd8
FS: 00007fca7fc4a700(0000) GS:ffff88007ee20000(0000) knlGS:0000000000000000
CS: e033 DS: 002b ES: 002b CR0: 000000008005003b
CR2: 00007f5c52d45610 CR3: 0000000038605000 CR4: 0000000000002660
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
0000000000000000 0140000000000000 ffffffff810077ec ffffffff810135e5
ffffffff81085da9 0000000000000000 0000000000000000 0000000000000000
0000000000000000 0000000000000000 0000000000000000 0000000000000000
Call Trace:
[<ffffffff810077ec>] ? xen_safe_halt+0xc/0x20
[<ffffffff810135e5>] ? default_idle+0x5/0x10
[<ffffffff81085da9>] ? cpu_startup_entry+0x129/0x180
Code: cc 51 41 53 b8 1c 00 00 00 0f 05 41 5b 59 c3 cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc 51 41 53 b8 1d 00 00 00 0f 05 <41> 5b 59 c3 cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc
INFO: rcu_sched self-detected stall on CPU { 0} (t=60003 jiffies g=99356660 c=99356659 q=24)
sending NMI to all CPUs:
NMI backtrace for cpu 0
CPU: 0 PID: 1464 Comm: in:imklog Tainted: G B 3.10.103-xenU-corp-33-adcc8c8-x86_64 #6
task: ffff880074c91180 ti: ffff880043496000 task.ti: ffff880043496000
RIP: e030:[<ffffffff8100130a>] [<ffffffff8100130a>] xen_hypercall_vcpu_op+0xa/0x20
RSP: e02b:ffff88007ee03cd0 EFLAGS: 00000046
RAX: 0000000000000000 RBX: 0000000000000000 RCX: ffffffff8100130a
RDX: 0000000000000000 RSI: 0000000000000000 RDI: 000000000000000b
RBP: ffffffff81658620 R08: ffffffff81658620 R09: 0000000000000000
R10: ffff88007ee0d1c0 R11: 0000000000000246 R12: ffffffff81658620
R13: 0000000000000005 R14: 0000000000000018 R15: ffff88007ee0d1c0
FS: 00007f768b268700(0000) GS:ffff88007ee00000(0000) knlGS:0000000000000000
CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007f768b249000 CR3: 0000000075066000 CR4: 0000000000002660
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
0720072007200720 0000000000000000 ffffffff8127a1ef ffffffff8100919a
0000000000002710 ffffffff81658628 ffffffff81628a00 ffffffff81628a00
ffffffff810277a7 ffff880043496000 ffffffff810bbfdd 0000000000000000
Call Trace:
<IRQ>
[<ffffffff8127a1ef>] ? xen_send_IPI_one+0x3f/0x60
[<ffffffff8100919a>] ? __xen_send_IPI_mask+0x2a/0x50
[<ffffffff810277a7>] ? arch_trigger_all_cpu_backtrace+0x47/0x80
[<ffffffff810bbfdd>] ? rcu_check_callbacks+0x2ed/0x580
[<ffffffff8108edf0>] ? tick_sched_handle.isra.17+0x30/0x30
[<ffffffff8105613b>] ? update_process_times+0x3b/0x70
[<ffffffff8108ee24>] ? tick_sched_timer+0x34/0x50
[<ffffffff8106b213>] ? __run_hrtimer.isra.32+0x43/0xd0
[<ffffffff8106b87f>] ? hrtimer_interrupt+0xef/0x250
[<ffffffff8100793a>] ? xen_timer_interrupt+0x2a/0x150
[<ffffffff812a07c9>] ? add_interrupt_randomness+0x39/0x190
[<ffffffff810cc37a>] ? find_get_page+0x1a/0x80
[<ffffffff810b5549>] ? handle_irq_event_percpu+0x89/0x140
[<ffffffff810b8403>] ? handle_percpu_irq+0x33/0x50
[<ffffffff81278e04>] ? __xen_evtchn_do_upcall+0x354/0x360
[<ffffffff8104fa6a>] ? __do_softirq+0x17a/0x1f0
[<ffffffff8127a692>] ? xen_evtchn_do_upcall+0x22/0x40
[<ffffffff813d7c3e>] ? xen_do_hypervisor_callback+0x1e/0x30
<EOI>
[<ffffffff810cc38d>] ? find_get_page+0x2d/0x80
[<ffffffff810cc374>] ? find_get_page+0x14/0x80
[<ffffffff810ff15f>] ? lookup_swap_cache+0x1f/0x40
[<ffffffff810ed557>] ? handle_pte_fault+0x107/0xa40
[<ffffffff81004659>] ? __raw_callee_save_xen_pmd_val+0x11/0x1e
[<ffffffff810ef601>] ? handle_mm_fault+0x391/0x430
[<ffffffff8102f7f2>] ? __do_page_fault+0x172/0x4a0
[<ffffffff8100440a>] ? xen_mc_flush+0x10a/0x140
[<ffffffff81003fbb>] ? xen_write_msr_safe+0x7b/0xc0
[<ffffffff8100a628>] ? __switch_to+0x1e8/0x500
[<ffffffff81261615>] ? vsnprintf+0x315/0x690
[<ffffffff813d6198>] ? page_fault+0x28/0x30
[<ffffffff81261e00>] ? copy_user_generic_string+0x30/0x40
[<ffffffff8104a89c>] ? do_syslog+0x1bc/0x570
[<ffffffff81068b40>] ? wake_up_bit+0x20/0x20
[<ffffffff81180799>] ? kmsg_read+0x39/0x50
[<ffffffff8117393e>] ? proc_reg_read+0x3e/0x60
[<ffffffff81119cf5>] ? vfs_read+0x95/0x160
[<ffffffff8111a782>] ? SyS_read+0x42/0x90
[<ffffffff813d66cd>] ? system_call_fastpath+0x1a/0x1f
Code: cc 51 41 53 50 b8 17 00 00 00 0f 05 5b 59 c3 cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc 51 41 53 b8 18 00 00 00 0f 05 <41> 5b 59 c3 cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc
NMI backtrace for cpu 1
CPU: 1 PID: 0 Comm: swapper/1 Tainted: G B 3.10.103-xenU-corp-33-adcc8c8-x86_64 #6
task: ffff880076873140 ti: ffff880075524000 task.ti: ffff880075524000
RIP: e030:[<ffffffff810013aa>] [<ffffffff810013aa>] xen_hypercall_sched_op+0xa/0x20
RSP: e02b:ffff880075525f00 EFLAGS: 00000246
RAX: 0000000000000000 RBX: ffffffff81658620 RCX: ffffffff810013aa
RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000001
RBP: ffff880075525fd8 R08: 0000000000000000 R09: 0000000000000000
R10: 0001403e29bdd24d R11: 0000000000000246 R12: ffff880075525fd8
R13: ffff880075525fd8 R14: ffff880075525fd8 R15: ffff880075525fd8
FS: 00007fca7fc4a700(0000) GS:ffff88007ee20000(0000) knlGS:0000000000000000
CS: e033 DS: 002b ES: 002b CR0: 000000008005003b
CR2: 00007f5c52d45610 CR3: 0000000038605000 CR4: 0000000000002660
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
0000000000000000 0140000000000000 ffffffff810077ec ffffffff810135e5
ffffffff81085da9 0000000000000000 0000000000000000 0000000000000000
0000000000000000 0000000000000000 0000000000000000 0000000000000000
Call Trace:
[<ffffffff810077ec>] ? xen_safe_halt+0xc/0x20
[<ffffffff810135e5>] ? default_idle+0x5/0x10
[<ffffffff81085da9>] ? cpu_startup_entry+0x129/0x180
Code: cc 51 41 53 b8 1c 00 00 00 0f 05 41 5b 59 c3 cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc 51 41 53 b8 1d 00 00 00 0f 05 <41> 5b 59 c3 cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc
[-- Attachment #3: domu2 --]
[-- Type: text/plain, Size: 10127 bytes --]
xen:grant_table: Grant tables using version 2 layout
PM: noirq restore of devices complete after 0.065 msecs
PM: early restore of devices complete after 0.023 msecs
PM: restore of devices complete after 23.617 msecs
installing Xen timer for CPU 3
cpu 3 spinlock event irq 29
ENERGY_PERF_BIAS: Set to 'normal', was 'performance'
ENERGY_PERF_BIAS: View and update with x86_energy_perf_policy(8)
installing Xen timer for CPU 1
cpu 1 spinlock event irq 36
installing Xen timer for CPU 2
cpu 2 spinlock event irq 43
INFO: rcu_sched detected stalls on CPUs/tasks: { 0} (detected by 1, t=15913 jiffies, g=282376928, c=282376927, q=55)
sending NMI to all CPUs:
NMI backtrace for cpu 1
CPU: 1 PID: 0 Comm: swapper/1 Not tainted 3.10.103-xenU-corp-33-adcc8c8-x86_64 #6
task: ffff8803eb873140 ti: ffff8803e7524000 task.ti: ffff8803e7524000
RIP: e030:[<ffffffff8100130a>] [<ffffffff8100130a>] xen_hypercall_vcpu_op+0xa/0x20
RSP: e02b:ffff8803fdc23cd0 EFLAGS: 00000046
RAX: 0000000000000000 RBX: 0000000000000001 RCX: ffffffff8100130a
RDX: 0000000000000000 RSI: 0000000000000001 RDI: 000000000000000b
RBP: ffffffff81658620 R08: ffffffff81658620 R09: 0000000000000000
R10: ffff8803fdc2d1c0 R11: 0000000000000246 R12: ffffffff81658620
R13: 0000000000000005 R14: 0000000000000037 R15: ffff8803fdc2d1c0
FS: 00007f8feb7fe700(0000) GS:ffff8803fdc20000(0000) knlGS:0000000000000000
CS: e033 DS: 002b ES: 002b CR0: 000000008005003b
CR2: 00007f900fdf3f30 CR3: 00000003b0ec4000 CR4: 0000000000002660
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
0720072007200720 0000000000000001 ffffffff8127a1ef ffffffff8100919a
0000000000002710 ffffffff81658628 ffffffff81628a00 ffffffff81628a00
ffffffff810277a7 ffff8803e7524000 ffffffff810bc220 0000000000000001
Call Trace:
<IRQ>
[<ffffffff8127a1ef>] ? xen_send_IPI_one+0x3f/0x60
[<ffffffff8100919a>] ? __xen_send_IPI_mask+0x2a/0x50
[<ffffffff810277a7>] ? arch_trigger_all_cpu_backtrace+0x47/0x80
[<ffffffff810bc220>] ? rcu_check_callbacks+0x530/0x580
[<ffffffff8108edf0>] ? tick_sched_handle.isra.17+0x30/0x30
[<ffffffff8105613b>] ? update_process_times+0x3b/0x70
[<ffffffff8108ee24>] ? tick_sched_timer+0x34/0x50
[<ffffffff8106b213>] ? __run_hrtimer.isra.32+0x43/0xd0
[<ffffffff8106b87f>] ? hrtimer_interrupt+0xef/0x250
[<ffffffff8100793a>] ? xen_timer_interrupt+0x2a/0x150
[<ffffffff812a07c9>] ? add_interrupt_randomness+0x39/0x190
[<ffffffff810013aa>] ? xen_hypercall_sched_op+0xa/0x20
[<ffffffff810b5549>] ? handle_irq_event_percpu+0x89/0x140
[<ffffffff810b8403>] ? handle_percpu_irq+0x33/0x50
[<ffffffff81278e04>] ? __xen_evtchn_do_upcall+0x354/0x360
[<ffffffff8100788c>] ? xen_clocksource_get_cycles+0x1c/0x20
[<ffffffff8108681c>] ? ktime_get+0x3c/0xe0
[<ffffffff8107641e>] ? sched_clock_cpu+0xe/0x100
[<ffffffff8108e458>] ? update_ts_time_stats+0x58/0x90
[<ffffffff8108efeb>] ? tick_check_idle+0xab/0xc0
[<ffffffff8127a692>] ? xen_evtchn_do_upcall+0x22/0x40
[<ffffffff813d7c3e>] ? xen_do_hypervisor_callback+0x1e/0x30
<EOI>
[<ffffffff810013aa>] ? xen_hypercall_sched_op+0xa/0x20
[<ffffffff810013aa>] ? xen_hypercall_sched_op+0xa/0x20
[<ffffffff810077ec>] ? xen_safe_halt+0xc/0x20
[<ffffffff810135e5>] ? default_idle+0x5/0x10
[<ffffffff81085da9>] ? cpu_startup_entry+0x129/0x180
Code: cc 51 41 53 50 b8 17 00 00 00 0f 05 5b 59 c3 cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc 51 41 53 b8 18 00 00 00 0f 05 <41> 5b 59 c3 cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc
NMI backtrace for cpu 0
CPU: 0 PID: 1816 Comm: named Not tainted 3.10.103-xenU-corp-33-adcc8c8-x86_64 #6
task: ffff880365d920c0 ti: ffff8803ac020000 task.ti: ffff8803ac020000
RIP: e030:[<ffffffff810013aa>] [<ffffffff810013aa>] xen_hypercall_sched_op+0xa/0x20
RSP: e02b:ffff8803ac0216c8 EFLAGS: 00000002
RAX: 0000000000000000 RBX: 0000000000000000 RCX: ffffffff810013aa
RDX: 0000000000000000 RSI: ffff8803ac0216e8 RDI: 0000000000000003
RBP: ffff8803e71c0780 R08: 0000000000000000 R09: ffff8803eb85f580
R10: ffff8803aa5d8380 R11: 0000000000000202 R12: 0000000000000000
R13: 0000000000000001 R14: ffff8803e71c1bc0 R15: 0000000000000001
FS: 00007f79d50f7700(0000) GS:ffff8803fdc00000(0000) knlGS:0000000000000000
CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007fcce3692000 CR3: 000000038b60c000 CR4: 0000000000002660
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
0000000000000202 ffff8803eb4000f0 ffffffff8127977b 00000008a64a15a7
ffff8803ac0216e4 0000000300000001 0000000000000000 0000000000000011
ffffffff810099e5 ffff8803e71c0780 0000000000000000 0000000000000000
Call Trace:
[<ffffffff8127977b>] ? xen_poll_irq_timeout+0x3b/0x50
[<ffffffff810099e5>] ? xen_spin_lock_slow+0x65/0x100
[<ffffffff81009afa>] ? xen_spin_lock_flags+0x3a/0x40
[<ffffffff813d5a1a>] ? _raw_spin_lock_irqsave+0x1a/0x30
[<ffffffff812b62f6>] ? xennet_interrupt+0x16/0x80
[<ffffffff810b5549>] ? handle_irq_event_percpu+0x89/0x140
[<ffffffff810b5635>] ? handle_irq_event+0x35/0x50
[<ffffffff810b7d4e>] ? handle_edge_irq+0x5e/0x130
[<ffffffff81278ce7>] ? __xen_evtchn_do_upcall+0x237/0x360
[<ffffffff8127a692>] ? xen_evtchn_do_upcall+0x22/0x40
[<ffffffff813d7c3e>] ? xen_do_hypervisor_callback+0x1e/0x30
[<ffffffff810013aa>] ? xen_hypercall_sched_op+0xa/0x20
[<ffffffff810013aa>] ? xen_hypercall_sched_op+0xa/0x20
[<ffffffff8127977b>] ? xen_poll_irq_timeout+0x3b/0x50
[<ffffffff810099e5>] ? xen_spin_lock_slow+0x65/0x100
[<ffffffff81009aba>] ? xen_spin_lock+0x3a/0x40
[<ffffffff813d59f5>] ? _raw_spin_lock+0x5/0x10
[<ffffffff812b4bd5>] ? xennet_poll+0x45/0xcc0
[<ffffffff813d7c0e>] ? xen_hypervisor_callback+0x1e/0x30
[<ffffffff813d63aa>] ? error_exit+0x2a/0x60
[<ffffffff813d5efb>] ? retint_restore_args+0x5/0x6
[<ffffffff8100122a>] ? xen_hypercall_xen_version+0xa/0x20
[<ffffffff8100122a>] ? xen_hypercall_xen_version+0xa/0x20
[<ffffffff8100122a>] ? xen_hypercall_xen_version+0xa/0x20
[<ffffffff812db6ec>] ? net_rx_action+0x10c/0x1d0
[<ffffffff8104f9f6>] ? __do_softirq+0x106/0x1f0
[<ffffffff813d7bdc>] ? call_softirq+0x1c/0x30
[<ffffffff8100dbe5>] ? do_softirq+0x45/0x80
[<ffffffff8104f439>] ? local_bh_enable_ip+0x99/0xa0
[<ffffffff812d18bd>] ? skb_free_datagram_locked+0x6d/0xc0
[<ffffffff8133587e>] ? udp_recvmsg+0x18e/0x350
[<ffffffff8133f165>] ? inet_recvmsg+0x65/0x80
[<ffffffff812c381a>] ? sock_recvmsg+0xba/0xe0
[<ffffffff8108ffd2>] ? get_futex_key+0x182/0x290
[<ffffffff812c3fbf>] ? ___sys_recvmsg+0xef/0x250
[<ffffffff81090251>] ? futex_wake+0x101/0x120
[<ffffffff81091dcd>] ? do_futex+0x10d/0xaf0
[<ffffffff8107007e>] ? finish_task_switch+0x4e/0xb0
[<ffffffff813d4349>] ? __schedule+0x279/0x7a0
[<ffffffff812c5f39>] ? __sys_recvmsg+0x39/0x70
[<ffffffff813d66cd>] ? system_call_fastpath+0x1a/0x1f
Code: cc 51 41 53 b8 1c 00 00 00 0f 05 41 5b 59 c3 cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc 51 41 53 b8 1d 00 00 00 0f 05 <41> 5b 59 c3 cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc
NMI backtrace for cpu 2
CPU: 2 PID: 0 Comm: swapper/2 Not tainted 3.10.103-xenU-corp-33-adcc8c8-x86_64 #6
task: ffff8803eb875880 ti: ffff8803e7526000 task.ti: ffff8803e7526000
RIP: e030:[<ffffffff810013aa>] [<ffffffff810013aa>] xen_hypercall_sched_op+0xa/0x20
RSP: e02b:ffff8803e7527f00 EFLAGS: 00000246
RAX: 0000000000000000 RBX: ffffffff81658620 RCX: ffffffff810013aa
RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000001
RBP: ffff8803e7527fd8 R08: 0000000000000000 R09: 0000000000000000
R10: 000285e8164e9740 R11: 0000000000000246 R12: ffff8803e7527fd8
R13: ffff8803e7527fd8 R14: ffff8803e7527fd8 R15: ffff8803e7527fd8
FS: 00007f900ddf5700(0000) GS:ffff8803fdc40000(0000) knlGS:0000000000000000
CS: e033 DS: 002b ES: 002b CR0: 000000008005003b
CR2: 00007fa1c00bd000 CR3: 00000003b0ec4000 CR4: 0000000000002660
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
0000000000000000 0140000000000000 ffffffff810077ec ffffffff810135e5
ffffffff81085da9 0000000000000000 0000000000000000 0000000000000000
0000000000000000 0000000000000000 0000000000000000 0000000000000000
Call Trace:
[<ffffffff810077ec>] ? xen_safe_halt+0xc/0x20
[<ffffffff810135e5>] ? default_idle+0x5/0x10
[<ffffffff81085da9>] ? cpu_startup_entry+0x129/0x180
Code: cc 51 41 53 b8 1c 00 00 00 0f 05 41 5b 59 c3 cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc 51 41 53 b8 1d 00 00 00 0f 05 <41> 5b 59 c3 cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc
NMI backtrace for cpu 3
CPU: 3 PID: 0 Comm: swapper/3 Not tainted 3.10.103-xenU-corp-33-adcc8c8-x86_64 #6
task: ffff8803eb875180 ti: ffff8803e752a000 task.ti: ffff8803e752a000
RIP: e030:[<ffffffff810013aa>] [<ffffffff810013aa>] xen_hypercall_sched_op+0xa/0x20
RSP: e02b:ffff8803e752bf00 EFLAGS: 00000246
RAX: 0000000000000000 RBX: ffffffff81658620 RCX: ffffffff810013aa
RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000001
RBP: ffff8803e752bfd8 R08: 0000000000000000 R09: 0000000000000000
R10: 000285e9165e5940 R11: 0000000000000246 R12: ffff8803e752bfd8
R13: ffff8803e752bfd8 R14: ffff8803e752bfd8 R15: ffff8803e752bfd8
FS: 00007f901e3c8700(0000) GS:ffff8803fdc60000(0000) knlGS:0000000000000000
CS: e033 DS: 002b ES: 002b CR0: 000000008005003b
CR2: 00007f2427e6e000 CR3: 00000003b0ec4000 CR4: 0000000000002660
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
0000000000000000 0140000000000000 ffffffff810077ec ffffffff810135e5
ffffffff81085da9 0000000000000000 0000000000000000 0000000000000000
0000000000000000 0000000000000000 0000000000000000 0000000000000000
Call Trace:
[<ffffffff810077ec>] ? xen_safe_halt+0xc/0x20
[<ffffffff810135e5>] ? default_idle+0x5/0x10
[<ffffffff81085da9>] ? cpu_startup_entry+0x129/0x180
Code: cc 51 41 53 b8 1c 00 00 00 0f 05 41 5b 59 c3 cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc 51 41 53 b8 1d 00 00 00 0f 05 <41> 5b 59 c3 cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc
[-- Attachment #4: domu3 --]
[-- Type: text/plain, Size: 50757 bytes --]
active_anon:37611 inactive_anon:37674 isolated_anon:0
active_file:272491 inactive_file:510668 isolated_file:0
unevictable:8 dirty:37 writeback:0 unstable:0
free:7080 slab_reclaimable:76904 slab_unreclaimable:50942
mapped:2421 shmem:7 pagetables:6259 bounce:0
free_cma:0
DMA free:15720kB min:28kB low:32kB high:40kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:120kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15996kB managed:15912kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:12kB slab_unreclaimable:52kB kernel_stack:8kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 3926 3926 3926
DMA32 free:12600kB min:8004kB low:10004kB high:12004kB active_anon:150444kB inactive_anon:150696kB active_file:1089964kB inactive_file:2042552kB unevictable:32kB isolated(anon):0kB isolated(file):0kB present:4177920kB managed:4020308kB mlocked:32kB dirty:148kB writeback:0kB mapped:9684kB shmem:28kB slab_reclaimable:307604kB slab_unreclaimable:203716kB kernel_stack:3224kB pagetables:25036kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
Normal free:0kB min:0kB low:0kB high:0kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:8192kB managed:0kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
lowmem_reserve[]: 0 0 0 0
DMA: 4*4kB (UE) 7*8kB (UEM) 4*16kB (UE) 3*32kB (UE) 0*64kB 3*128kB (UEM) 3*256kB (UEM) 2*512kB (EM) 3*1024kB (UEM) 3*2048kB (UEM) 1*4096kB (R) = 15720kB
DMA32: 2782*4kB (UEM) 0*8kB 1*16kB (R) 0*32kB 1*64kB (R) 1*128kB (R) 1*256kB (R) 0*512kB 1*1024kB (R) 0*2048kB 0*4096kB = 12616kB
Normal: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB
783441 total pagecache pages
275 pages in swap cache
Swap cache stats: add 5339845, delete 5339570, find 5529400458/5529685579
Free swap = 1009716kB
Total swap = 1048572kB
SLAB: Unable to allocate memory on node 0 (gfp=0x20)
cache: TCP, object size: 1664, order: 1
node 0: slabs: 146/146, objs: 584/584, free: 0
kswapd0: page allocation failure: order:1, mode:0x200020
CPU: 0 PID: 283 Comm: kswapd0 Not tainted 3.10.44-xenU-corp-13-930d568-x86_64 #6
ffffffff813e78ee 0000000000000038 ffffffff810d45c8 0000000000000000
0000000200000002 0000000000000000 ffffffff810b8384 ffff8800f52875c0
ffff8800f52875c0 0000000000000000 fffffffe00000002 0000000000000000
Call Trace:
<IRQ> [<ffffffff813e78ee>] ? dump_stack+0xd/0x17
[<ffffffff810d45c8>] ? warn_alloc_failed+0x108/0x160
[<ffffffff810b8384>] ? handle_irq_event+0x54/0x80
[<ffffffff810d7918>] ? __alloc_pages_nodemask+0x7e8/0xa50
[<ffffffff813ead65>] ? _raw_spin_unlock_irqrestore+0x15/0x20
[<ffffffff812c3565>] ? xennet_start_xmit+0x805/0x970
[<ffffffff8130679b>] ? sch_direct_xmit+0x8b/0x1d0
[<ffffffff8110a9a6>] ? cache_alloc_refill+0x336/0x6d0
[<ffffffff8110a5ca>] ? kmem_cache_alloc+0xda/0x180
[<ffffffff812d6fd3>] ? sk_prot_alloc+0x43/0x1a0
[<ffffffff812d867a>] ? sk_clone_lock+0x1a/0x370
[<ffffffff81326500>] ? inet_csk_clone_lock+0x10/0xc0
[<ffffffff8133f80b>] ? tcp_create_openreq_child+0x1b/0x420
[<ffffffff8133bf83>] ? tcp_v4_syn_recv_sock+0x33/0x290
[<ffffffff8133f671>] ? tcp_check_req+0x3e1/0x560
[<ffffffff8133bdf2>] ? tcp_v4_do_rcv+0x202/0x360
[<ffffffff8133ef9a>] ? tcp_v4_rcv+0x7ba/0x7f0
[<ffffffff8131b6e0>] ? ip_local_deliver_finish+0xa0/0x200
[<ffffffff812e90b3>] ? __netif_receive_skb_core+0x5e3/0x7d0
[<ffffffff812e94b4>] ? netif_receive_skb+0x24/0x80
[<ffffffff812ea868>] ? napi_gro_receive+0x98/0xd0
[<ffffffff812c4307>] ? xennet_poll+0x797/0xce0
[<ffffffff810b8273>] ? handle_irq_event_percpu+0x83/0x140
[<ffffffff812e97b4>] ? net_rx_action+0x104/0x1c0
[<ffffffff810500e6>] ? __do_softirq+0xd6/0x1b0
[<ffffffff813ecf5c>] ? call_softirq+0x1c/0x30
[<ffffffff8100dba5>] ? do_softirq+0x65/0xa0
[<ffffffff81050306>] ? irq_exit+0x86/0xb0
[<ffffffff81286f87>] ? xen_evtchn_do_upcall+0x27/0x40
[<ffffffff813ecfbe>] ? xen_do_hypervisor_callback+0x1e/0x30
<EOI> [<ffffffff81133360>] ? d_set_d_op+0xc0/0xc0
[<ffffffff81009dbf>] ? xen_spin_lock+0x1f/0x60
[<ffffffff813eacf5>] ? _raw_spin_lock+0x5/0x10
[<ffffffff81133e76>] ? shrink_dentry_list+0x46/0x220
[<ffffffff810d9179>] ? global_dirty_limits+0x29/0x110
[<ffffffff81134f14>] ? prune_dcache_sb+0x154/0x180
[<ffffffff811216e0>] ? prune_super+0x170/0x1b0
[<ffffffff810dd77f>] ? shrink_slab+0x11f/0x1d0
[<ffffffff810dfd3d>] ? balance_pgdat+0x46d/0x560
[<ffffffff813ead26>] ? _raw_spin_lock_irqsave+0x26/0x40
[<ffffffff810dff4e>] ? kswapd+0x11e/0x370
[<ffffffff81069d60>] ? abort_exclusive_wait+0xb0/0xb0
[<ffffffff810dfe30>] ? balance_pgdat+0x560/0x560
[<ffffffff81069203>] ? kthread+0xb3/0xc0
[<ffffffff81070000>] ? set_current_groups+0x10/0x70
[<ffffffff81069150>] ? kthread_freezable_should_stop+0x60/0x60
[<ffffffff813eb97c>] ? ret_from_fork+0x7c/0xb0
[<ffffffff81069150>] ? kthread_freezable_should_stop+0x60/0x60
Mem-Info:
DMA per-cpu:
CPU 0: hi: 0, btch: 1 usd: 0
DMA32 per-cpu:
CPU 0: hi: 186, btch: 31 usd: 139
Normal per-cpu:
CPU 0: hi: 0, btch: 1 usd: 0
active_anon:37611 inactive_anon:37674 isolated_anon:0
active_file:272491 inactive_file:510668 isolated_file:0
unevictable:8 dirty:37 writeback:0 unstable:0
free:7080 slab_reclaimable:76904 slab_unreclaimable:50942
mapped:2421 shmem:7 pagetables:6259 bounce:0
free_cma:0
DMA free:15720kB min:28kB low:32kB high:40kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:120kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15996kB managed:15912kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:12kB slab_unreclaimable:52kB kernel_stack:8kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 3926 3926 3926
DMA32 free:12600kB min:8004kB low:10004kB high:12004kB active_anon:150444kB inactive_anon:150696kB active_file:1089964kB inactive_file:2042552kB unevictable:32kB isolated(anon):0kB isolated(file):0kB present:4177920kB managed:4020308kB mlocked:32kB dirty:148kB writeback:0kB mapped:9684kB shmem:28kB slab_reclaimable:307604kB slab_unreclaimable:203716kB kernel_stack:3224kB pagetables:25036kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
Normal free:0kB min:0kB low:0kB high:0kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:8192kB managed:0kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
lowmem_reserve[]: 0 0 0 0
DMA: 4*4kB (UE) 7*8kB (UEM) 4*16kB (UE) 3*32kB (UE) 0*64kB 3*128kB (UEM) 3*256kB (UEM) 2*512kB (EM) 3*1024kB (UEM) 3*2048kB (UEM) 1*4096kB (R) = 15720kB
DMA32: 2782*4kB (UEM) 0*8kB 1*16kB (R) 0*32kB 1*64kB (R) 1*128kB (R) 1*256kB (R) 0*512kB 1*1024kB (R) 0*2048kB 0*4096kB = 12616kB
Normal: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB
783441 total pagecache pages
275 pages in swap cache
Swap cache stats: add 5339845, delete 5339570, find 5529400458/5529685579
Free swap = 1009716kB
Total swap = 1048572kB
SLAB: Unable to allocate memory on node 0 (gfp=0x20)
cache: TCP, object size: 1664, order: 1
node 0: slabs: 146/146, objs: 584/584, free: 0
kswapd0: page allocation failure: order:1, mode:0x200020
CPU: 0 PID: 283 Comm: kswapd0 Not tainted 3.10.44-xenU-corp-13-930d568-x86_64 #6
ffffffff813e78ee 0000000000000038 ffffffff810d45c8 0000000000000000
0000000200000002 0000000000000000 ffffffff810b8384 ffff8800f52875c0
ffff8800f52875c0 0000000000000000 fffffffe00000002 0000000000000000
Call Trace:
<IRQ> [<ffffffff813e78ee>] ? dump_stack+0xd/0x17
[<ffffffff810d45c8>] ? warn_alloc_failed+0x108/0x160
[<ffffffff810b8384>] ? handle_irq_event+0x54/0x80
[<ffffffff810d7918>] ? __alloc_pages_nodemask+0x7e8/0xa50
[<ffffffff813ead65>] ? _raw_spin_unlock_irqrestore+0x15/0x20
[<ffffffff812c3565>] ? xennet_start_xmit+0x805/0x970
[<ffffffff8130679b>] ? sch_direct_xmit+0x8b/0x1d0
[<ffffffff8110a9a6>] ? cache_alloc_refill+0x336/0x6d0
[<ffffffff8110a5ca>] ? kmem_cache_alloc+0xda/0x180
[<ffffffff812d6fd3>] ? sk_prot_alloc+0x43/0x1a0
[<ffffffff812d867a>] ? sk_clone_lock+0x1a/0x370
[<ffffffff81326500>] ? inet_csk_clone_lock+0x10/0xc0
[<ffffffff8133f80b>] ? tcp_create_openreq_child+0x1b/0x420
[<ffffffff8133bf83>] ? tcp_v4_syn_recv_sock+0x33/0x290
[<ffffffff8133f671>] ? tcp_check_req+0x3e1/0x560
[<ffffffff8133bdf2>] ? tcp_v4_do_rcv+0x202/0x360
[<ffffffff8133ef9a>] ? tcp_v4_rcv+0x7ba/0x7f0
[<ffffffff8131b6e0>] ? ip_local_deliver_finish+0xa0/0x200
[<ffffffff812e90b3>] ? __netif_receive_skb_core+0x5e3/0x7d0
[<ffffffff812e94b4>] ? netif_receive_skb+0x24/0x80
[<ffffffff812ea868>] ? napi_gro_receive+0x98/0xd0
[<ffffffff812c4307>] ? xennet_poll+0x797/0xce0
[<ffffffff810b8273>] ? handle_irq_event_percpu+0x83/0x140
[<ffffffff812e97b4>] ? net_rx_action+0x104/0x1c0
[<ffffffff810500e6>] ? __do_softirq+0xd6/0x1b0
[<ffffffff813ecf5c>] ? call_softirq+0x1c/0x30
[<ffffffff8100dba5>] ? do_softirq+0x65/0xa0
[<ffffffff81050306>] ? irq_exit+0x86/0xb0
[<ffffffff81286f87>] ? xen_evtchn_do_upcall+0x27/0x40
[<ffffffff813ecfbe>] ? xen_do_hypervisor_callback+0x1e/0x30
<EOI> [<ffffffff81133360>] ? d_set_d_op+0xc0/0xc0
[<ffffffff81009d86>] ? xen_spin_unlock+0x6/0x20
[<ffffffff81133fcb>] ? shrink_dentry_list+0x19b/0x220
[<ffffffff810d9179>] ? global_dirty_limits+0x29/0x110
[<ffffffff81134f14>] ? prune_dcache_sb+0x154/0x180
[<ffffffff811216e0>] ? prune_super+0x170/0x1b0
[<ffffffff810dd77f>] ? shrink_slab+0x11f/0x1d0
[<ffffffff810dfd3d>] ? balance_pgdat+0x46d/0x560
[<ffffffff813ead26>] ? _raw_spin_lock_irqsave+0x26/0x40
[<ffffffff810dff4e>] ? kswapd+0x11e/0x370
[<ffffffff81069d60>] ? abort_exclusive_wait+0xb0/0xb0
[<ffffffff810dfe30>] ? balance_pgdat+0x560/0x560
[<ffffffff81069203>] ? kthread+0xb3/0xc0
[<ffffffff81070000>] ? set_current_groups+0x10/0x70
[<ffffffff81069150>] ? kthread_freezable_should_stop+0x60/0x60
[<ffffffff813eb97c>] ? ret_from_fork+0x7c/0xb0
[<ffffffff81069150>] ? kthread_freezable_should_stop+0x60/0x60
Mem-Info:
DMA per-cpu:
CPU 0: hi: 0, btch: 1 usd: 0
DMA32 per-cpu:
CPU 0: hi: 186, btch: 31 usd: 139
Normal per-cpu:
CPU 0: hi: 0, btch: 1 usd: 0
active_anon:37611 inactive_anon:37674 isolated_anon:0
active_file:272491 inactive_file:510668 isolated_file:0
unevictable:8 dirty:37 writeback:0 unstable:0
free:7080 slab_reclaimable:76904 slab_unreclaimable:50942
mapped:2421 shmem:7 pagetables:6259 bounce:0
free_cma:0
DMA free:15720kB min:28kB low:32kB high:40kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:120kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15996kB managed:15912kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:12kB slab_unreclaimable:52kB kernel_stack:8kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 3926 3926 3926
DMA32 free:12600kB min:8004kB low:10004kB high:12004kB active_anon:150444kB inactive_anon:150696kB active_file:1089964kB inactive_file:2042552kB unevictable:32kB isolated(anon):0kB isolated(file):0kB present:4177920kB managed:4020308kB mlocked:32kB dirty:148kB writeback:0kB mapped:9684kB shmem:28kB slab_reclaimable:307604kB slab_unreclaimable:203716kB kernel_stack:3224kB pagetables:25036kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
Normal free:0kB min:0kB low:0kB high:0kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:8192kB managed:0kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
lowmem_reserve[]: 0 0 0 0
DMA: 4*4kB (UE) 7*8kB (UEM) 4*16kB (UE) 3*32kB (UE) 0*64kB 3*128kB (UEM) 3*256kB (UEM) 2*512kB (EM) 3*1024kB (UEM) 3*2048kB (UEM) 1*4096kB (R) = 15720kB
DMA32: 2782*4kB (UEM) 0*8kB 1*16kB (R) 0*32kB 1*64kB (R) 1*128kB (R) 1*256kB (R) 0*512kB 1*1024kB (R) 0*2048kB 0*4096kB = 12616kB
Normal: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB
783441 total pagecache pages
275 pages in swap cache
Swap cache stats: add 5339845, delete 5339570, find 5529400458/5529685579
Free swap = 1009716kB
Total swap = 1048572kB
SLAB: Unable to allocate memory on node 0 (gfp=0x20)
cache: TCP, object size: 1664, order: 1
node 0: slabs: 146/146, objs: 584/584, free: 0
kswapd0: page allocation failure: order:1, mode:0x200020
CPU: 0 PID: 283 Comm: kswapd0 Not tainted 3.10.44-xenU-corp-13-930d568-x86_64 #6
ffffffff813e78ee 0000000000000038 ffffffff810d45c8 0000000000000000
0000000200000002 0000000000000000 ffffffff810b8384 ffff8800f52875c0
ffff8800f52875c0 0000000000000000 fffffffe00000002 0000000000000000
Call Trace:
<IRQ> [<ffffffff813e78ee>] ? dump_stack+0xd/0x17
[<ffffffff810d45c8>] ? warn_alloc_failed+0x108/0x160
[<ffffffff810b8384>] ? handle_irq_event+0x54/0x80
[<ffffffff810d7918>] ? __alloc_pages_nodemask+0x7e8/0xa50
[<ffffffff8100122a>] ? xen_hypercall_xen_version+0xa/0x20
[<ffffffff8100122a>] ? xen_hypercall_xen_version+0xa/0x20
[<ffffffff8110a9a6>] ? cache_alloc_refill+0x336/0x6d0
[<ffffffff8110a5ca>] ? kmem_cache_alloc+0xda/0x180
[<ffffffff812d6fd3>] ? sk_prot_alloc+0x43/0x1a0
[<ffffffff812d867a>] ? sk_clone_lock+0x1a/0x370
[<ffffffff81326500>] ? inet_csk_clone_lock+0x10/0xc0
[<ffffffff8133f80b>] ? tcp_create_openreq_child+0x1b/0x420
[<ffffffff8133bf83>] ? tcp_v4_syn_recv_sock+0x33/0x290
[<ffffffff8133f671>] ? tcp_check_req+0x3e1/0x560
[<ffffffff8133bdf2>] ? tcp_v4_do_rcv+0x202/0x360
[<ffffffff8133ef9a>] ? tcp_v4_rcv+0x7ba/0x7f0
[<ffffffff8131b6e0>] ? ip_local_deliver_finish+0xa0/0x200
[<ffffffff812e90b3>] ? __netif_receive_skb_core+0x5e3/0x7d0
[<ffffffff812e94b4>] ? netif_receive_skb+0x24/0x80
[<ffffffff812ea868>] ? napi_gro_receive+0x98/0xd0
[<ffffffff812c4307>] ? xennet_poll+0x797/0xce0
[<ffffffff810b8273>] ? handle_irq_event_percpu+0x83/0x140
[<ffffffff812e97b4>] ? net_rx_action+0x104/0x1c0
[<ffffffff810500e6>] ? __do_softirq+0xd6/0x1b0
[<ffffffff813ecf5c>] ? call_softirq+0x1c/0x30
[<ffffffff8100dba5>] ? do_softirq+0x65/0xa0
[<ffffffff81050306>] ? irq_exit+0x86/0xb0
[<ffffffff81286f87>] ? xen_evtchn_do_upcall+0x27/0x40
[<ffffffff813ecfbe>] ? xen_do_hypervisor_callback+0x1e/0x30
<EOI> [<ffffffff81133360>] ? d_set_d_op+0xc0/0xc0
[<ffffffff81009dbf>] ? xen_spin_lock+0x1f/0x60
[<ffffffff813eacf5>] ? _raw_spin_lock+0x5/0x10
[<ffffffff81133e76>] ? shrink_dentry_list+0x46/0x220
[<ffffffff810d9179>] ? global_dirty_limits+0x29/0x110
[<ffffffff81134f14>] ? prune_dcache_sb+0x154/0x180
[<ffffffff811216e0>] ? prune_super+0x170/0x1b0
[<ffffffff810dd77f>] ? shrink_slab+0x11f/0x1d0
[<ffffffff810dfd3d>] ? balance_pgdat+0x46d/0x560
[<ffffffff813ead26>] ? _raw_spin_lock_irqsave+0x26/0x40
[<ffffffff810dff4e>] ? kswapd+0x11e/0x370
[<ffffffff81069d60>] ? abort_exclusive_wait+0xb0/0xb0
[<ffffffff810dfe30>] ? balance_pgdat+0x560/0x560
[<ffffffff81069203>] ? kthread+0xb3/0xc0
[<ffffffff81070000>] ? set_current_groups+0x10/0x70
[<ffffffff81069150>] ? kthread_freezable_should_stop+0x60/0x60
[<ffffffff813eb97c>] ? ret_from_fork+0x7c/0xb0
[<ffffffff81069150>] ? kthread_freezable_should_stop+0x60/0x60
Mem-Info:
DMA per-cpu:
CPU 0: hi: 0, btch: 1 usd: 0
DMA32 per-cpu:
CPU 0: hi: 186, btch: 31 usd: 138
Normal per-cpu:
CPU 0: hi: 0, btch: 1 usd: 0
active_anon:37611 inactive_anon:37674 isolated_anon:0
active_file:272491 inactive_file:510668 isolated_file:0
unevictable:8 dirty:37 writeback:0 unstable:0
free:7080 slab_reclaimable:76904 slab_unreclaimable:50942
mapped:2421 shmem:7 pagetables:6259 bounce:0
free_cma:0
DMA free:15720kB min:28kB low:32kB high:40kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:120kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15996kB managed:15912kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:12kB slab_unreclaimable:52kB kernel_stack:8kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 3926 3926 3926
DMA32 free:12600kB min:8004kB low:10004kB high:12004kB active_anon:150444kB inactive_anon:150696kB active_file:1089964kB inactive_file:2042552kB unevictable:32kB isolated(anon):0kB isolated(file):0kB present:4177920kB managed:4020308kB mlocked:32kB dirty:148kB writeback:0kB mapped:9684kB shmem:28kB slab_reclaimable:307604kB slab_unreclaimable:203716kB kernel_stack:3224kB pagetables:25036kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
Normal free:0kB min:0kB low:0kB high:0kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:8192kB managed:0kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
lowmem_reserve[]: 0 0 0 0
DMA: 4*4kB (UE) 7*8kB (UEM) 4*16kB (UE) 3*32kB (UE) 0*64kB 3*128kB (UEM) 3*256kB (UEM) 2*512kB (EM) 3*1024kB (UEM) 3*2048kB (UEM) 1*4096kB (R) = 15720kB
DMA32: 2782*4kB (UEM) 0*8kB 1*16kB (R) 0*32kB 1*64kB (R) 1*128kB (R) 1*256kB (R) 0*512kB 1*1024kB (R) 0*2048kB 0*4096kB = 12616kB
Normal: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB
783441 total pagecache pages
275 pages in swap cache
Swap cache stats: add 5339845, delete 5339570, find 5529400458/5529685579
Free swap = 1009716kB
Total swap = 1048572kB
SLAB: Unable to allocate memory on node 0 (gfp=0x20)
cache: TCP, object size: 1664, order: 1
node 0: slabs: 146/146, objs: 584/584, free: 0
kswapd0: page allocation failure: order:1, mode:0x200020
CPU: 0 PID: 283 Comm: kswapd0 Not tainted 3.10.44-xenU-corp-13-930d568-x86_64 #6
ffffffff813e78ee 0000000000000038 ffffffff810d45c8 0000000000000000
0000000200000002 0000000000000000 ffffffff810b8384 ffff8800f52875c0
ffff8800f52875c0 0000000000000000 fffffffe00000002 0000000000000000
Call Trace:
<IRQ> [<ffffffff813e78ee>] ? dump_stack+0xd/0x17
[<ffffffff810d45c8>] ? warn_alloc_failed+0x108/0x160
[<ffffffff810b8384>] ? handle_irq_event+0x54/0x80
[<ffffffff810d7918>] ? __alloc_pages_nodemask+0x7e8/0xa50
[<ffffffff812c3565>] ? xennet_start_xmit+0x805/0x970
[<ffffffff8130679b>] ? sch_direct_xmit+0x8b/0x1d0
[<ffffffff8110a9a6>] ? cache_alloc_refill+0x336/0x6d0
[<ffffffff8110a5ca>] ? kmem_cache_alloc+0xda/0x180
[<ffffffff812d6fd3>] ? sk_prot_alloc+0x43/0x1a0
[<ffffffff812d867a>] ? sk_clone_lock+0x1a/0x370
[<ffffffff81326500>] ? inet_csk_clone_lock+0x10/0xc0
[<ffffffff8133f80b>] ? tcp_create_openreq_child+0x1b/0x420
[<ffffffff8133bf83>] ? tcp_v4_syn_recv_sock+0x33/0x290
[<ffffffff8133f671>] ? tcp_check_req+0x3e1/0x560
[<ffffffff8133bdf2>] ? tcp_v4_do_rcv+0x202/0x360
[<ffffffff8133ef9a>] ? tcp_v4_rcv+0x7ba/0x7f0
[<ffffffff8131b6e0>] ? ip_local_deliver_finish+0xa0/0x200
[<ffffffff812e90b3>] ? __netif_receive_skb_core+0x5e3/0x7d0
[<ffffffff812e94b4>] ? netif_receive_skb+0x24/0x80
[<ffffffff812ea868>] ? napi_gro_receive+0x98/0xd0
[<ffffffff812c4307>] ? xennet_poll+0x797/0xce0
[<ffffffff810b8273>] ? handle_irq_event_percpu+0x83/0x140
[<ffffffff812e97b4>] ? net_rx_action+0x104/0x1c0
[<ffffffff810500e6>] ? __do_softirq+0xd6/0x1b0
[<ffffffff813ecf5c>] ? call_softirq+0x1c/0x30
[<ffffffff8100dba5>] ? do_softirq+0x65/0xa0
[<ffffffff81050306>] ? irq_exit+0x86/0xb0
[<ffffffff81286f87>] ? xen_evtchn_do_upcall+0x27/0x40
[<ffffffff813ecfbe>] ? xen_do_hypervisor_callback+0x1e/0x30
<EOI> [<ffffffff81133360>] ? d_set_d_op+0xc0/0xc0
[<ffffffff81009da0>] ? xen_spin_unlock+0x20/0x20
[<ffffffff813eacf5>] ? _raw_spin_lock+0x5/0x10
[<ffffffff81133e76>] ? shrink_dentry_list+0x46/0x220
[<ffffffff810d9179>] ? global_dirty_limits+0x29/0x110
[<ffffffff81134f14>] ? prune_dcache_sb+0x154/0x180
[<ffffffff811216e0>] ? prune_super+0x170/0x1b0
[<ffffffff810dd77f>] ? shrink_slab+0x11f/0x1d0
[<ffffffff810dfd3d>] ? balance_pgdat+0x46d/0x560
[<ffffffff813ead26>] ? _raw_spin_lock_irqsave+0x26/0x40
[<ffffffff810dff4e>] ? kswapd+0x11e/0x370
[<ffffffff81069d60>] ? abort_exclusive_wait+0xb0/0xb0
[<ffffffff810dfe30>] ? balance_pgdat+0x560/0x560
[<ffffffff81069203>] ? kthread+0xb3/0xc0
[<ffffffff81070000>] ? set_current_groups+0x10/0x70
[<ffffffff81069150>] ? kthread_freezable_should_stop+0x60/0x60
[<ffffffff813eb97c>] ? ret_from_fork+0x7c/0xb0
[<ffffffff81069150>] ? kthread_freezable_should_stop+0x60/0x60
Mem-Info:
DMA per-cpu:
CPU 0: hi: 0, btch: 1 usd: 0
DMA32 per-cpu:
CPU 0: hi: 186, btch: 31 usd: 137
Normal per-cpu:
CPU 0: hi: 0, btch: 1 usd: 0
active_anon:37611 inactive_anon:37674 isolated_anon:0
active_file:272491 inactive_file:510668 isolated_file:0
unevictable:8 dirty:37 writeback:0 unstable:0
free:7080 slab_reclaimable:76904 slab_unreclaimable:50942
mapped:2421 shmem:7 pagetables:6259 bounce:0
free_cma:0
DMA free:15720kB min:28kB low:32kB high:40kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:120kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15996kB managed:15912kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:12kB slab_unreclaimable:52kB kernel_stack:8kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 3926 3926 3926
DMA32 free:12600kB min:8004kB low:10004kB high:12004kB active_anon:150444kB inactive_anon:150696kB active_file:1089964kB inactive_file:2042552kB unevictable:32kB isolated(anon):0kB isolated(file):0kB present:4177920kB managed:4020308kB mlocked:32kB dirty:148kB writeback:0kB mapped:9684kB shmem:28kB slab_reclaimable:307604kB slab_unreclaimable:203716kB kernel_stack:3224kB pagetables:25036kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
Normal free:0kB min:0kB low:0kB high:0kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:8192kB managed:0kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
lowmem_reserve[]: 0 0 0 0
DMA: 4*4kB (UE) 7*8kB (UEM) 4*16kB (UE) 3*32kB (UE) 0*64kB 3*128kB (UEM) 3*256kB (UEM) 2*512kB (EM) 3*1024kB (UEM) 3*2048kB (UEM) 1*4096kB (R) = 15720kB
DMA32: 2782*4kB (UEM) 0*8kB 1*16kB (R) 0*32kB 1*64kB (R) 1*128kB (R) 1*256kB (R) 0*512kB 1*1024kB (R) 0*2048kB 0*4096kB = 12616kB
Normal: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB
783441 total pagecache pages
275 pages in swap cache
Swap cache stats: add 5339845, delete 5339570, find 5529400458/5529685579
Free swap = 1009716kB
Total swap = 1048572kB
SLAB: Unable to allocate memory on node 0 (gfp=0x20)
cache: TCP, object size: 1664, order: 1
node 0: slabs: 146/146, objs: 584/584, free: 0
kswapd0: page allocation failure: order:1, mode:0x200020
CPU: 0 PID: 283 Comm: kswapd0 Not tainted 3.10.44-xenU-corp-13-930d568-x86_64 #6
ffffffff813e78ee 0000000000000038 ffffffff810d45c8 0000000000000000
0000000200000002 0000000000000000 ffffffff810b8384 ffff8800f52875c0
ffff8800f52875c0 0000000000000000 fffffffe00000002 0000000000000000
Call Trace:
<IRQ> [<ffffffff813e78ee>] ? dump_stack+0xd/0x17
[<ffffffff810d45c8>] ? warn_alloc_failed+0x108/0x160
[<ffffffff810b8384>] ? handle_irq_event+0x54/0x80
[<ffffffff810d7918>] ? __alloc_pages_nodemask+0x7e8/0xa50
[<ffffffff813ead26>] ? _raw_spin_lock_irqsave+0x26/0x40
[<ffffffff8110a9a6>] ? cache_alloc_refill+0x336/0x6d0
[<ffffffff810b8384>] ? handle_irq_event+0x54/0x80
[<ffffffff8110a5ca>] ? kmem_cache_alloc+0xda/0x180
[<ffffffff812d6fd3>] ? sk_prot_alloc+0x43/0x1a0
[<ffffffff812d867a>] ? sk_clone_lock+0x1a/0x370
[<ffffffff81326500>] ? inet_csk_clone_lock+0x10/0xc0
[<ffffffff8133f80b>] ? tcp_create_openreq_child+0x1b/0x420
[<ffffffff8133bf83>] ? tcp_v4_syn_recv_sock+0x33/0x290
[<ffffffff8133f671>] ? tcp_check_req+0x3e1/0x560
[<ffffffff8133bdf2>] ? tcp_v4_do_rcv+0x202/0x360
[<ffffffff8133ef9a>] ? tcp_v4_rcv+0x7ba/0x7f0
[<ffffffff8131b6e0>] ? ip_local_deliver_finish+0xa0/0x200
[<ffffffff812e90b3>] ? __netif_receive_skb_core+0x5e3/0x7d0
[<ffffffff812e94b4>] ? netif_receive_skb+0x24/0x80
[<ffffffff812ea868>] ? napi_gro_receive+0x98/0xd0
[<ffffffff812c4307>] ? xennet_poll+0x797/0xce0
[<ffffffff810b8273>] ? handle_irq_event_percpu+0x83/0x140
[<ffffffff812e97b4>] ? net_rx_action+0x104/0x1c0
[<ffffffff810500e6>] ? __do_softirq+0xd6/0x1b0
[<ffffffff813ecf5c>] ? call_softirq+0x1c/0x30
[<ffffffff8100dba5>] ? do_softirq+0x65/0xa0
[<ffffffff81050306>] ? irq_exit+0x86/0xb0
[<ffffffff81286f87>] ? xen_evtchn_do_upcall+0x27/0x40
[<ffffffff813ecfbe>] ? xen_do_hypervisor_callback+0x1e/0x30
<EOI> [<ffffffff81133360>] ? d_set_d_op+0xc0/0xc0
[<ffffffff81009dbf>] ? xen_spin_lock+0x1f/0x60
[<ffffffff813eacf5>] ? _raw_spin_lock+0x5/0x10
[<ffffffff81133e76>] ? shrink_dentry_list+0x46/0x220
[<ffffffff810d9179>] ? global_dirty_limits+0x29/0x110
[<ffffffff81134f14>] ? prune_dcache_sb+0x154/0x180
[<ffffffff811216e0>] ? prune_super+0x170/0x1b0
[<ffffffff810dd77f>] ? shrink_slab+0x11f/0x1d0
[<ffffffff810dfd3d>] ? balance_pgdat+0x46d/0x560
[<ffffffff813ead26>] ? _raw_spin_lock_irqsave+0x26/0x40
[<ffffffff810dff4e>] ? kswapd+0x11e/0x370
[<ffffffff81069d60>] ? abort_exclusive_wait+0xb0/0xb0
[<ffffffff810dfe30>] ? balance_pgdat+0x560/0x560
[<ffffffff81069203>] ? kthread+0xb3/0xc0
[<ffffffff81070000>] ? set_current_groups+0x10/0x70
[<ffffffff81069150>] ? kthread_freezable_should_stop+0x60/0x60
[<ffffffff813eb97c>] ? ret_from_fork+0x7c/0xb0
[<ffffffff81069150>] ? kthread_freezable_should_stop+0x60/0x60
Mem-Info:
DMA per-cpu:
CPU 0: hi: 0, btch: 1 usd: 0
DMA32 per-cpu:
CPU 0: hi: 186, btch: 31 usd: 138
Normal per-cpu:
CPU 0: hi: 0, btch: 1 usd: 0
active_anon:37611 inactive_anon:37674 isolated_anon:0
active_file:272491 inactive_file:510668 isolated_file:0
unevictable:8 dirty:37 writeback:0 unstable:0
free:7080 slab_reclaimable:76904 slab_unreclaimable:50942
mapped:2421 shmem:7 pagetables:6259 bounce:0
free_cma:0
DMA free:15720kB min:28kB low:32kB high:40kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:120kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15996kB managed:15912kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:12kB slab_unreclaimable:52kB kernel_stack:8kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 3926 3926 3926
DMA32 free:12600kB min:8004kB low:10004kB high:12004kB active_anon:150444kB inactive_anon:150696kB active_file:1089964kB inactive_file:2042552kB unevictable:32kB isolated(anon):0kB isolated(file):0kB present:4177920kB managed:4020308kB mlocked:32kB dirty:148kB writeback:0kB mapped:9684kB shmem:28kB slab_reclaimable:307604kB slab_unreclaimable:203716kB kernel_stack:3224kB pagetables:25036kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
Normal free:0kB min:0kB low:0kB high:0kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:8192kB managed:0kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
lowmem_reserve[]: 0 0 0 0
DMA: 4*4kB (UE) 7*8kB (UEM) 4*16kB (UE) 3*32kB (UE) 0*64kB 3*128kB (UEM) 3*256kB (UEM) 2*512kB (EM) 3*1024kB (UEM) 3*2048kB (UEM) 1*4096kB (R) = 15720kB
DMA32: 2782*4kB (UEM) 0*8kB 1*16kB (R) 0*32kB 1*64kB (R) 1*128kB (R) 1*256kB (R) 0*512kB 1*1024kB (R) 0*2048kB 0*4096kB = 12616kB
Normal: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB
783441 total pagecache pages
275 pages in swap cache
Swap cache stats: add 5339845, delete 5339570, find 5529400458/5529685579
Free swap = 1009716kB
Total swap = 1048572kB
SLAB: Unable to allocate memory on node 0 (gfp=0x20)
cache: TCP, object size: 1664, order: 1
node 0: slabs: 146/146, objs: 584/584, free: 0
kswapd0: page allocation failure: order:1, mode:0x200020
CPU: 0 PID: 283 Comm: kswapd0 Not tainted 3.10.44-xenU-corp-13-930d568-x86_64 #6
ffffffff813e78ee 0000000000000038 ffffffff810d45c8 0000000000000000
0000000200000002 0000000000000000 ffffffff810b8384 ffff8800f52875c0
ffff8800f52875c0 0000000000000000 fffffffe00000002 0000000000000000
Call Trace:
<IRQ> [<ffffffff813e78ee>] ? dump_stack+0xd/0x17
[<ffffffff810d45c8>] ? warn_alloc_failed+0x108/0x160
[<ffffffff810b8384>] ? handle_irq_event+0x54/0x80
[<ffffffff810d7918>] ? __alloc_pages_nodemask+0x7e8/0xa50
[<ffffffff8100122a>] ? xen_hypercall_xen_version+0xa/0x20
[<ffffffff8100122a>] ? xen_hypercall_xen_version+0xa/0x20
[<ffffffff8110a9a6>] ? cache_alloc_refill+0x336/0x6d0
[<ffffffff8110a5ca>] ? kmem_cache_alloc+0xda/0x180
[<ffffffff812d6fd3>] ? sk_prot_alloc+0x43/0x1a0
[<ffffffff812d867a>] ? sk_clone_lock+0x1a/0x370
[<ffffffff81326500>] ? inet_csk_clone_lock+0x10/0xc0
[<ffffffff8133f80b>] ? tcp_create_openreq_child+0x1b/0x420
[<ffffffff8133bf83>] ? tcp_v4_syn_recv_sock+0x33/0x290
[<ffffffff8133f671>] ? tcp_check_req+0x3e1/0x560
[<ffffffff8133bdf2>] ? tcp_v4_do_rcv+0x202/0x360
[<ffffffff8133ef9a>] ? tcp_v4_rcv+0x7ba/0x7f0
[<ffffffff8131b6e0>] ? ip_local_deliver_finish+0xa0/0x200
[<ffffffff812e90b3>] ? __netif_receive_skb_core+0x5e3/0x7d0
[<ffffffff812e94b4>] ? netif_receive_skb+0x24/0x80
[<ffffffff812ea868>] ? napi_gro_receive+0x98/0xd0
[<ffffffff812c4307>] ? xennet_poll+0x797/0xce0
[<ffffffff810b8273>] ? handle_irq_event_percpu+0x83/0x140
[<ffffffff812e97b4>] ? net_rx_action+0x104/0x1c0
[<ffffffff810500e6>] ? __do_softirq+0xd6/0x1b0
[<ffffffff813ecf5c>] ? call_softirq+0x1c/0x30
[<ffffffff8100dba5>] ? do_softirq+0x65/0xa0
[<ffffffff81050306>] ? irq_exit+0x86/0xb0
[<ffffffff81286f87>] ? xen_evtchn_do_upcall+0x27/0x40
[<ffffffff813ecfbe>] ? xen_do_hypervisor_callback+0x1e/0x30
<EOI> [<ffffffff81133360>] ? d_set_d_op+0xc0/0xc0
[<ffffffff81133fcf>] ? shrink_dentry_list+0x19f/0x220
[<ffffffff81133fcb>] ? shrink_dentry_list+0x19b/0x220
[<ffffffff810d9179>] ? global_dirty_limits+0x29/0x110
[<ffffffff81134f14>] ? prune_dcache_sb+0x154/0x180
[<ffffffff811216e0>] ? prune_super+0x170/0x1b0
[<ffffffff810dd77f>] ? shrink_slab+0x11f/0x1d0
[<ffffffff810dfd3d>] ? balance_pgdat+0x46d/0x560
[<ffffffff813ead26>] ? _raw_spin_lock_irqsave+0x26/0x40
[<ffffffff810dff4e>] ? kswapd+0x11e/0x370
[<ffffffff81069d60>] ? abort_exclusive_wait+0xb0/0xb0
[<ffffffff810dfe30>] ? balance_pgdat+0x560/0x560
[<ffffffff81069203>] ? kthread+0xb3/0xc0
[<ffffffff81070000>] ? set_current_groups+0x10/0x70
[<ffffffff81069150>] ? kthread_freezable_should_stop+0x60/0x60
[<ffffffff813eb97c>] ? ret_from_fork+0x7c/0xb0
[<ffffffff81069150>] ? kthread_freezable_should_stop+0x60/0x60
Mem-Info:
DMA per-cpu:
CPU 0: hi: 0, btch: 1 usd: 0
DMA32 per-cpu:
CPU 0: hi: 186, btch: 31 usd: 137
Normal per-cpu:
CPU 0: hi: 0, btch: 1 usd: 0
active_anon:37611 inactive_anon:37674 isolated_anon:0
active_file:272491 inactive_file:510668 isolated_file:0
unevictable:8 dirty:37 writeback:0 unstable:0
free:7080 slab_reclaimable:76904 slab_unreclaimable:50942
mapped:2421 shmem:7 pagetables:6259 bounce:0
free_cma:0
DMA free:15720kB min:28kB low:32kB high:40kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:120kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15996kB managed:15912kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:12kB slab_unreclaimable:52kB kernel_stack:8kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 3926 3926 3926
DMA32 free:12600kB min:8004kB low:10004kB high:12004kB active_anon:150444kB inactive_anon:150696kB active_file:1089964kB inactive_file:2042552kB unevictable:32kB isolated(anon):0kB isolated(file):0kB present:4177920kB managed:4020308kB mlocked:32kB dirty:148kB writeback:0kB mapped:9684kB shmem:28kB slab_reclaimable:307604kB slab_unreclaimable:203716kB kernel_stack:3224kB pagetables:25036kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
Normal free:0kB min:0kB low:0kB high:0kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:8192kB managed:0kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
lowmem_reserve[]: 0 0 0 0
DMA: 4*4kB (UE) 7*8kB (UEM) 4*16kB (UE) 3*32kB (UE) 0*64kB 3*128kB (UEM) 3*256kB (UEM) 2*512kB (EM) 3*1024kB (UEM) 3*2048kB (UEM) 1*4096kB (R) = 15720kB
DMA32: 2782*4kB (UEM) 0*8kB 1*16kB (R) 0*32kB 1*64kB (R) 1*128kB (R) 1*256kB (R) 0*512kB 1*1024kB (R) 0*2048kB 0*4096kB = 12616kB
Normal: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB
783441 total pagecache pages
275 pages in swap cache
Swap cache stats: add 5339845, delete 5339570, find 5529400458/5529685579
Free swap = 1009716kB
Total swap = 1048572kB
SLAB: Unable to allocate memory on node 0 (gfp=0x20)
cache: TCP, object size: 1664, order: 1
node 0: slabs: 146/146, objs: 584/584, free: 0
kswapd0: page allocation failure: order:1, mode:0x200020
CPU: 0 PID: 283 Comm: kswapd0 Not tainted 3.10.44-xenU-corp-13-930d568-x86_64 #6
ffffffff813e78ee 0000000000000038 ffffffff810d45c8 0000000000000000
0000000200000002 0000000000000000 ffffffff810b8384 ffff8800f52875c0
ffff8800f52875c0 0000000000000000 fffffffe00000002 0000000000000000
Call Trace:
<IRQ> [<ffffffff813e78ee>] ? dump_stack+0xd/0x17
[<ffffffff810d45c8>] ? warn_alloc_failed+0x108/0x160
[<ffffffff810b8384>] ? handle_irq_event+0x54/0x80
[<ffffffff810d7918>] ? __alloc_pages_nodemask+0x7e8/0xa50
[<ffffffff812c3565>] ? xennet_start_xmit+0x805/0x970
[<ffffffff8130679b>] ? sch_direct_xmit+0x8b/0x1d0
[<ffffffff8110a9a6>] ? cache_alloc_refill+0x336/0x6d0
[<ffffffff8110a5ca>] ? kmem_cache_alloc+0xda/0x180
[<ffffffff812d6fd3>] ? sk_prot_alloc+0x43/0x1a0
[<ffffffff812d867a>] ? sk_clone_lock+0x1a/0x370
[<ffffffff81326500>] ? inet_csk_clone_lock+0x10/0xc0
[<ffffffff8133f80b>] ? tcp_create_openreq_child+0x1b/0x420
[<ffffffff8133bf83>] ? tcp_v4_syn_recv_sock+0x33/0x290
[<ffffffff8133f671>] ? tcp_check_req+0x3e1/0x560
[<ffffffff8133bdf2>] ? tcp_v4_do_rcv+0x202/0x360
[<ffffffff8133ef9a>] ? tcp_v4_rcv+0x7ba/0x7f0
[<ffffffff8131b6e0>] ? ip_local_deliver_finish+0xa0/0x200
[<ffffffff812e90b3>] ? __netif_receive_skb_core+0x5e3/0x7d0
[<ffffffff812e94b4>] ? netif_receive_skb+0x24/0x80
[<ffffffff812ea868>] ? napi_gro_receive+0x98/0xd0
[<ffffffff812c4307>] ? xennet_poll+0x797/0xce0
[<ffffffff810b8273>] ? handle_irq_event_percpu+0x83/0x140
[<ffffffff812e97b4>] ? net_rx_action+0x104/0x1c0
[<ffffffff810500e6>] ? __do_softirq+0xd6/0x1b0
[<ffffffff813ecf5c>] ? call_softirq+0x1c/0x30
[<ffffffff8100dba5>] ? do_softirq+0x65/0xa0
[<ffffffff81050306>] ? irq_exit+0x86/0xb0
[<ffffffff81286f87>] ? xen_evtchn_do_upcall+0x27/0x40
[<ffffffff813ecfbe>] ? xen_do_hypervisor_callback+0x1e/0x30
<EOI> [<ffffffff81133360>] ? d_set_d_op+0xc0/0xc0
[<ffffffff81009d86>] ? xen_spin_unlock+0x6/0x20
[<ffffffff81133fcb>] ? shrink_dentry_list+0x19b/0x220
[<ffffffff810d9179>] ? global_dirty_limits+0x29/0x110
[<ffffffff81134f14>] ? prune_dcache_sb+0x154/0x180
[<ffffffff811216e0>] ? prune_super+0x170/0x1b0
[<ffffffff810dd77f>] ? shrink_slab+0x11f/0x1d0
[<ffffffff810dfd3d>] ? balance_pgdat+0x46d/0x560
[<ffffffff813ead26>] ? _raw_spin_lock_irqsave+0x26/0x40
[<ffffffff810dff4e>] ? kswapd+0x11e/0x370
[<ffffffff81069d60>] ? abort_exclusive_wait+0xb0/0xb0
[<ffffffff810dfe30>] ? balance_pgdat+0x560/0x560
[<ffffffff81069203>] ? kthread+0xb3/0xc0
[<ffffffff81070000>] ? set_current_groups+0x10/0x70
[<ffffffff81069150>] ? kthread_freezable_should_stop+0x60/0x60
[<ffffffff813eb97c>] ? ret_from_fork+0x7c/0xb0
[<ffffffff81069150>] ? kthread_freezable_should_stop+0x60/0x60
Mem-Info:
DMA per-cpu:
CPU 0: hi: 0, btch: 1 usd: 0
DMA32 per-cpu:
CPU 0: hi: 186, btch: 31 usd: 138
Normal per-cpu:
CPU 0: hi: 0, btch: 1 usd: 0
active_anon:37611 inactive_anon:37674 isolated_anon:0
active_file:272491 inactive_file:510668 isolated_file:0
unevictable:8 dirty:37 writeback:0 unstable:0
free:7080 slab_reclaimable:76904 slab_unreclaimable:50942
mapped:2421 shmem:7 pagetables:6259 bounce:0
free_cma:0
DMA free:15720kB min:28kB low:32kB high:40kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:120kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15996kB managed:15912kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:12kB slab_unreclaimable:52kB kernel_stack:8kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 3926 3926 3926
DMA32 free:12600kB min:8004kB low:10004kB high:12004kB active_anon:150444kB inactive_anon:150696kB active_file:1089964kB inactive_file:2042552kB unevictable:32kB isolated(anon):0kB isolated(file):0kB present:4177920kB managed:4020308kB mlocked:32kB dirty:148kB writeback:0kB mapped:9684kB shmem:28kB slab_reclaimable:307604kB slab_unreclaimable:203716kB kernel_stack:3224kB pagetables:25036kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
Normal free:0kB min:0kB low:0kB high:0kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:8192kB managed:0kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
lowmem_reserve[]: 0 0 0 0
DMA: 4*4kB (UE) 7*8kB (UEM) 4*16kB (UE) 3*32kB (UE) 0*64kB 3*128kB (UEM) 3*256kB (UEM) 2*512kB (EM) 3*1024kB (UEM) 3*2048kB (UEM) 1*4096kB (R) = 15720kB
DMA32: 2782*4kB (UEM) 0*8kB 1*16kB (R) 0*32kB 1*64kB (R) 1*128kB (R) 1*256kB (R) 0*512kB 1*1024kB (R) 0*2048kB 0*4096kB = 12616kB
Normal: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB
783441 total pagecache pages
275 pages in swap cache
Swap cache stats: add 5339845, delete 5339570, find 5529400458/5529685579
Free swap = 1009716kB
Total swap = 1048572kB
SLAB: Unable to allocate memory on node 0 (gfp=0x20)
cache: TCP, object size: 1664, order: 1
node 0: slabs: 146/146, objs: 584/584, free: 0
kswapd0: page allocation failure: order:1, mode:0x200020
CPU: 0 PID: 283 Comm: kswapd0 Not tainted 3.10.44-xenU-corp-13-930d568-x86_64 #6
ffffffff813e78ee 0000000000000038 ffffffff810d45c8 0000000000000000
0000000200000002 0000000000000000 ffffffff810b8384 ffff8800f52875c0
ffff8800f52875c0 0000000000000000 fffffffe00000002 0000000000000000
Call Trace:
<IRQ> [<ffffffff813e78ee>] ? dump_stack+0xd/0x17
[<ffffffff810d45c8>] ? warn_alloc_failed+0x108/0x160
[<ffffffff810b8384>] ? handle_irq_event+0x54/0x80
[<ffffffff810d7918>] ? __alloc_pages_nodemask+0x7e8/0xa50
[<ffffffff813ead65>] ? _raw_spin_unlock_irqrestore+0x15/0x20
[<ffffffff812c3565>] ? xennet_start_xmit+0x805/0x970
[<ffffffff8130679b>] ? sch_direct_xmit+0x8b/0x1d0
[<ffffffff8110a9a6>] ? cache_alloc_refill+0x336/0x6d0
[<ffffffff8110a5ca>] ? kmem_cache_alloc+0xda/0x180
[<ffffffff812d6fd3>] ? sk_prot_alloc+0x43/0x1a0
[<ffffffff812d867a>] ? sk_clone_lock+0x1a/0x370
[<ffffffff81326500>] ? inet_csk_clone_lock+0x10/0xc0
[<ffffffff8133f80b>] ? tcp_create_openreq_child+0x1b/0x420
[<ffffffff8133bf83>] ? tcp_v4_syn_recv_sock+0x33/0x290
[<ffffffff8133f671>] ? tcp_check_req+0x3e1/0x560
[<ffffffff8133bdf2>] ? tcp_v4_do_rcv+0x202/0x360
[<ffffffff8133ef9a>] ? tcp_v4_rcv+0x7ba/0x7f0
[<ffffffff8131b6e0>] ? ip_local_deliver_finish+0xa0/0x200
[<ffffffff812e90b3>] ? __netif_receive_skb_core+0x5e3/0x7d0
[<ffffffff812e94b4>] ? netif_receive_skb+0x24/0x80
[<ffffffff812ea868>] ? napi_gro_receive+0x98/0xd0
[<ffffffff812c4307>] ? xennet_poll+0x797/0xce0
[<ffffffff810b8273>] ? handle_irq_event_percpu+0x83/0x140
[<ffffffff812e97b4>] ? net_rx_action+0x104/0x1c0
[<ffffffff810500e6>] ? __do_softirq+0xd6/0x1b0
[<ffffffff813ecf5c>] ? call_softirq+0x1c/0x30
[<ffffffff8100dba5>] ? do_softirq+0x65/0xa0
[<ffffffff81050306>] ? irq_exit+0x86/0xb0
[<ffffffff81286f87>] ? xen_evtchn_do_upcall+0x27/0x40
[<ffffffff813ecfbe>] ? xen_do_hypervisor_callback+0x1e/0x30
<EOI> [<ffffffff81133360>] ? d_set_d_op+0xc0/0xc0
[<ffffffff81133e61>] ? shrink_dentry_list+0x31/0x220
[<ffffffff81133fcb>] ? shrink_dentry_list+0x19b/0x220
[<ffffffff810d9179>] ? global_dirty_limits+0x29/0x110
[<ffffffff81134f14>] ? prune_dcache_sb+0x154/0x180
[<ffffffff811216e0>] ? prune_super+0x170/0x1b0
[<ffffffff810dd77f>] ? shrink_slab+0x11f/0x1d0
[<ffffffff810dfd3d>] ? balance_pgdat+0x46d/0x560
[<ffffffff813ead26>] ? _raw_spin_lock_irqsave+0x26/0x40
[<ffffffff810dff4e>] ? kswapd+0x11e/0x370
[<ffffffff81069d60>] ? abort_exclusive_wait+0xb0/0xb0
[<ffffffff810dfe30>] ? balance_pgdat+0x560/0x560
[<ffffffff81069203>] ? kthread+0xb3/0xc0
[<ffffffff81070000>] ? set_current_groups+0x10/0x70
[<ffffffff81069150>] ? kthread_freezable_should_stop+0x60/0x60
[<ffffffff813eb97c>] ? ret_from_fork+0x7c/0xb0
[<ffffffff81069150>] ? kthread_freezable_should_stop+0x60/0x60
Mem-Info:
DMA per-cpu:
CPU 0: hi: 0, btch: 1 usd: 0
DMA32 per-cpu:
CPU 0: hi: 186, btch: 31 usd: 137
Normal per-cpu:
CPU 0: hi: 0, btch: 1 usd: 0
active_anon:37611 inactive_anon:37674 isolated_anon:0
active_file:272491 inactive_file:510668 isolated_file:0
unevictable:8 dirty:37 writeback:0 unstable:0
free:7080 slab_reclaimable:76904 slab_unreclaimable:50942
mapped:2421 shmem:7 pagetables:6259 bounce:0
free_cma:0
DMA free:15720kB min:28kB low:32kB high:40kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:120kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15996kB managed:15912kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:12kB slab_unreclaimable:52kB kernel_stack:8kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 3926 3926 3926
DMA32 free:12600kB min:8004kB low:10004kB high:12004kB active_anon:150444kB inactive_anon:150696kB active_file:1089964kB inactive_file:2042552kB unevictable:32kB isolated(anon):0kB isolated(file):0kB present:4177920kB managed:4020308kB mlocked:32kB dirty:148kB writeback:0kB mapped:9684kB shmem:28kB slab_reclaimable:307604kB slab_unreclaimable:203716kB kernel_stack:3224kB pagetables:25036kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
Normal free:0kB min:0kB low:0kB high:0kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:8192kB managed:0kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
lowmem_reserve[]: 0 0 0 0
DMA: 4*4kB (UE) 7*8kB (UEM) 4*16kB (UE) 3*32kB (UE) 0*64kB 3*128kB (UEM) 3*256kB (UEM) 2*512kB (EM) 3*1024kB (UEM) 3*2048kB (UEM) 1*4096kB (R) = 15720kB
DMA32: 2782*4kB (UEM) 0*8kB 1*16kB (R) 0*32kB 1*64kB (R) 1*128kB (R) 1*256kB (R) 0*512kB 1*1024kB (R) 0*2048kB 0*4096kB = 12616kB
Normal: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB
783441 total pagecache pages
275 pages in swap cache
Swap cache stats: add 5339845, delete 5339570, find 5529400458/5529685579
Free swap = 1009716kB
Total swap = 1048572kB
SLAB: Unable to allocate memory on node 0 (gfp=0x20)
cache: TCP, object size: 1664, order: 1
node 0: slabs: 146/146, objs: 584/584, free: 0
kswapd0: page allocation failure: order:1, mode:0x200020
CPU: 0 PID: 283 Comm: kswapd0 Not tainted 3.10.44-xenU-corp-13-930d568-x86_64 #6
ffffffff813e78ee 0000000000000038 ffffffff810d45c8 0000000000000000
0000000200000002 0000000000000000 ffffffff810b8384 ffff8800f52875c0
ffff8800f52875c0 0000000000000000 fffffffe00000002 0000000000000000
Call Trace:
<IRQ> [<ffffffff813e78ee>] ? dump_stack+0xd/0x17
[<ffffffff810d45c8>] ? warn_alloc_failed+0x108/0x160
[<ffffffff810b8384>] ? handle_irq_event+0x54/0x80
[<ffffffff810d7918>] ? __alloc_pages_nodemask+0x7e8/0xa50
[<ffffffff8100122a>] ? xen_hypercall_xen_version+0xa/0x20
[<ffffffff8100122a>] ? xen_hypercall_xen_version+0xa/0x20
[<ffffffff8110a9a6>] ? cache_alloc_refill+0x336/0x6d0
[<ffffffff8110a5ca>] ? kmem_cache_alloc+0xda/0x180
[<ffffffff812d6fd3>] ? sk_prot_alloc+0x43/0x1a0
[<ffffffff812d867a>] ? sk_clone_lock+0x1a/0x370
[<ffffffff81326500>] ? inet_csk_clone_lock+0x10/0xc0
[<ffffffff8133f80b>] ? tcp_create_openreq_child+0x1b/0x420
[<ffffffff8133bf83>] ? tcp_v4_syn_recv_sock+0x33/0x290
[<ffffffff8133f671>] ? tcp_check_req+0x3e1/0x560
[<ffffffff8133bdf2>] ? tcp_v4_do_rcv+0x202/0x360
[<ffffffff8133ef9a>] ? tcp_v4_rcv+0x7ba/0x7f0
[<ffffffff8131b6e0>] ? ip_local_deliver_finish+0xa0/0x200
[<ffffffff812e90b3>] ? __netif_receive_skb_core+0x5e3/0x7d0
[<ffffffff812e94b4>] ? netif_receive_skb+0x24/0x80
[<ffffffff812ea868>] ? napi_gro_receive+0x98/0xd0
[<ffffffff812c4307>] ? xennet_poll+0x797/0xce0
[<ffffffff810b8273>] ? handle_irq_event_percpu+0x83/0x140
[<ffffffff812e97b4>] ? net_rx_action+0x104/0x1c0
[<ffffffff810500e6>] ? __do_softirq+0xd6/0x1b0
[<ffffffff813ecf5c>] ? call_softirq+0x1c/0x30
[<ffffffff8100dba5>] ? do_softirq+0x65/0xa0
[<ffffffff81050306>] ? irq_exit+0x86/0xb0
[<ffffffff81286f87>] ? xen_evtchn_do_upcall+0x27/0x40
[<ffffffff813ecfbe>] ? xen_do_hypervisor_callback+0x1e/0x30
<EOI> [<ffffffff81133360>] ? d_set_d_op+0xc0/0xc0
[<ffffffff81133fcb>] ? shrink_dentry_list+0x19b/0x220
[<ffffffff81133fcb>] ? shrink_dentry_list+0x19b/0x220
[<ffffffff810d9179>] ? global_dirty_limits+0x29/0x110
[<ffffffff81134f14>] ? prune_dcache_sb+0x154/0x180
[<ffffffff811216e0>] ? prune_super+0x170/0x1b0
[<ffffffff810dd77f>] ? shrink_slab+0x11f/0x1d0
[<ffffffff810dfd3d>] ? balance_pgdat+0x46d/0x560
[<ffffffff813ead26>] ? _raw_spin_lock_irqsave+0x26/0x40
[<ffffffff810dff4e>] ? kswapd+0x11e/0x370
[<ffffffff81069d60>] ? abort_exclusive_wait+0xb0/0xb0
[<ffffffff810dfe30>] ? balance_pgdat+0x560/0x560
[<ffffffff81069203>] ? kthread+0xb3/0xc0
[<ffffffff81070000>] ? set_current_groups+0x10/0x70
[<ffffffff81069150>] ? kthread_freezable_should_stop+0x60/0x60
[<ffffffff813eb97c>] ? ret_from_fork+0x7c/0xb0
[<ffffffff81069150>] ? kthread_freezable_should_stop+0x60/0x60
Mem-Info:
DMA per-cpu:
CPU 0: hi: 0, btch: 1 usd: 0
DMA32 per-cpu:
CPU 0: hi: 186, btch: 31 usd: 137
Normal per-cpu:
CPU 0: hi: 0, btch: 1 usd: 0
active_anon:37611 inactive_anon:37674 isolated_anon:0
active_file:272491 inactive_file:510668 isolated_file:0
unevictable:8 dirty:37 writeback:0 unstable:0
free:7080 slab_reclaimable:76904 slab_unreclaimable:50942
mapped:2421 shmem:7 pagetables:6259 bounce:0
free_cma:0
DMA free:15720kB min:28kB low:32kB high:40kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:120kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15996kB managed:15912kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:12kB slab_unreclaimable:52kB kernel_stack:8kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 3926 3926 3926
DMA32 free:12600kB min:8004kB low:10004kB high:12004kB active_anon:150444kB inactive_anon:150696kB active_file:1089964kB inactive_file:2042552kB unevictable:32kB isolated(anon):0kB isolated(file):0kB present:4177920kB managed:4020308kB mlocked:32kB dirty:148kB writeback:0kB mapped:9684kB shmem:28kB slab_reclaimable:307604kB slab_unreclaimable:203716kB kernel_stack:3224kB pagetables:25036kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
Normal free:0kB min:0kB low:0kB high:0kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:8192kB managed:0kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
lowmem_reserve[]: 0 0 0 0
DMA: 4*4kB (UE) 7*8kB (UEM) 4*16kB (UE) 3*32kB (UE) 0*64kB 3*128kB (UEM) 3*256kB (UEM) 2*512kB (EM) 3*1024kB (UEM) 3*2048kB (UEM) 1*4096kB (R) = 15720kB
DMA32: 2782*4kB (UEM) 0*8kB 1*16kB (R) 0*32kB 1*64kB (R) 1*128kB (R) 1*256kB (R) 0*512kB 1*1024kB (R) 0*2048kB 0*4096kB = 12616kB
Normal: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB
783441 total pagecache pages
275 pages in swap cache
Swap cache stats: add 5339845, delete 5339570, find 5529400458/5529685579
Free swap = 1009716kB
Total swap = 1048572kB
SLAB: Unable to allocate memory on node 0 (gfp=0x20)
cache: TCP, object size: 1664, order: 1
node 0: slabs: 146/146, objs: 584/584, free: 0
[-- Attachment #5: domu4 --]
[-- Type: text/plain, Size: 36305 bytes --]
WARNING: at /home/package/git/hosting-paravirt/src/linux-3.10/arch/x86/xen/multicalls.c:129 __xen_pgd_pin+0x100/0x1b0()
Modules linked in: nfnetlink_log nfnetlink crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel aes_x86_64
CPU: 0 PID: 22604 Comm: nrpe Not tainted 3.10.44-xenU-corp-13-930d568-x86_64 #6
ffffffff813e78ee 0000000000000000 ffffffff8104876a ffff880189c65000
0000000004604210 ffff8801c012e000 ffff880126e14b40 000077ff80000000
ffffffff81006ea0 ffff880126e14bac ffff880126e14b40 ffff8801f2d447c0
Call Trace:
[<ffffffff813e78ee>] ? dump_stack+0xd/0x17
[<ffffffff8104876a>] ? warn_slowpath_common+0x6a/0xa0
[<ffffffff81006ea0>] ? __xen_pgd_pin+0x100/0x1b0
[<ffffffff81007169>] ? xen_dup_mmap+0x29/0x50
[<ffffffff81046453>] ? dup_mm+0x4d3/0x590
[<ffffffff810477fe>] ? copy_process.part.42+0x12be/0x13e0
[<ffffffff8102fb50>] ? __do_page_fault+0x2c0/0x4b0
[<ffffffff81132f03>] ? __d_instantiate+0xd3/0x110
[<ffffffff81047a16>] ? do_fork+0xb6/0x330
[<ffffffff813ebd89>] ? stub_clone+0x69/0x90
[<ffffffff813eba2d>] ? system_call_fastpath+0x1a/0x1f
---[ end trace 1caf04ec12866833 ]---
------------[ cut here ]------------
WARNING: at /home/package/git/hosting-paravirt/src/linux-3.10/arch/x86/xen/multicalls.c:129 xen_end_context_switch+0x9/0x20()
Modules linked in: nfnetlink_log nfnetlink crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel aes_x86_64
CPU: 1 PID: 8973 Comm: mysqld Tainted: G W 3.10.44-xenU-corp-13-930d568-x86_64 #6
ffffffff813e78ee 0000000000000000 ffffffff8104876a ffff88003bc29060
ffff88010fdcca90 0000000000000000 0000000000000001 ffff8801fec2f600
ffffffff81003259 ffff88003bc29060 ffffffff8100a57f ffff88004b4ebef8
Call Trace:
---[ end trace 1caf04ec12866834 ]---
------------[ cut here ]------------
WARNING: at /home/package/git/hosting-paravirt/src/linux-3.10/arch/x86/xen/multicalls.c:129 xen_end_context_switch+0x9/0x20()
Modules linked in: nfnetlink_log nfnetlink crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel aes_x86_64
CPU: 1 PID: 27353 Comm: rs:main Q:Reg Tainted: G W 3.10.44-xenU-corp-13-930d568-x86_64 #6
ffffffff813e78ee 0000000000000000 ffffffff8104876a ffff88003bc29060
ffff880003cf74b0 0000000000000000 0000000000000001 ffff8801fec2f600
ffffffff81003259 ffff88003bc29060 ffffffff8100a57f ffff8801fec321b0
Call Trace:
---[ end trace 1caf04ec12866835 ]---
------------[ cut here ]------------
WARNING: at /home/package/git/hosting-paravirt/src/linux-3.10/arch/x86/xen/multicalls.c:129 xen_end_context_switch+0x9/0x20()
Modules linked in: nfnetlink_log nfnetlink crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel aes_x86_64
CPU: 1 PID: 27353 Comm: rs:main Q:Reg Tainted: G W 3.10.44-xenU-corp-13-930d568-x86_64 #6
ffffffff813e78ee 0000000000000000 ffffffff8104876a ffff88003bc29060
ffff880003cf74b0 0000000000000000 0000000000000001 ffff8801fec2f600
ffffffff81003259 ffff88003bc29060 ffffffff8100a57f ffff8801fec321b0
Call Trace:
---[ end trace 1caf04ec12866836 ]---
------------[ cut here ]------------
WARNING: at /home/package/git/hosting-paravirt/src/linux-3.10/arch/x86/xen/multicalls.c:129 xen_end_context_switch+0x9/0x20()
Modules linked in: nfnetlink_log nfnetlink crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel aes_x86_64
CPU: 1 PID: 14164 Comm: mysqld Tainted: G W 3.10.44-xenU-corp-13-930d568-x86_64 #6
ffffffff813e78ee 0000000000000000 ffffffff8104876a ffff88003bc29060
ffff880019266400 0000000000000000 0000000000000001 ffff8801fec2f600
ffffffff81003259 ffff88003bc29060 ffffffff8100a57f ffff8801fec321b0
Call Trace:
---[ end trace 1caf04ec12866837 ]---
------------[ cut here ]------------
WARNING: at /home/package/git/hosting-paravirt/src/linux-3.10/arch/x86/xen/multicalls.c:129 xen_end_context_switch+0x9/0x20()
Modules linked in: nfnetlink_log nfnetlink crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel aes_x86_64
CPU: 1 PID: 14172 Comm: mysqld Tainted: G W 3.10.44-xenU-corp-13-930d568-x86_64 #6
ffffffff813e78ee 0000000000000000 ffffffff8104876a ffff88003bc29060
ffff8801009603c0 0000000000000000 0000000000000001 ffff8801fec2f600
ffffffff81003259 ffff88003bc29060 ffffffff8100a57f ffff8801fec321b0
Call Trace:
---[ end trace 1caf04ec12866838 ]---
------------[ cut here ]------------
WARNING: at /home/package/git/hosting-paravirt/src/linux-3.10/arch/x86/xen/multicalls.c:129 xen_end_context_switch+0x9/0x20()
Modules linked in: nfnetlink_log nfnetlink crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel aes_x86_64
CPU: 1 PID: 9 Comm: rcu_sched Tainted: G W 3.10.44-xenU-corp-13-930d568-x86_64 #6
ffffffff813e78ee 0000000000000000 ffffffff8104876a ffff88003bc29060
ffff8801f3127530 0000000000000000 0000000000000001 ffff8801fec2f600
ffffffff81003259 ffff88003bc29060 ffffffff8100a57f ffff8801fec321b0
Call Trace:
---[ end trace 1caf04ec12866839 ]---
------------[ cut here ]------------
WARNING: at /home/package/git/hosting-paravirt/src/linux-3.10/arch/x86/xen/multicalls.c:129 xen_end_context_switch+0x9/0x20()
Modules linked in: nfnetlink_log nfnetlink crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel aes_x86_64
CPU: 1 PID: 27353 Comm: rs:main Q:Reg Tainted: G W 3.10.44-xenU-corp-13-930d568-x86_64 #6
ffffffff813e78ee 0000000000000000 ffffffff8104876a ffff88003bc29060
ffff880003cf74b0 0000000000000000 0000000000000001 ffff8801fec2f600
ffffffff81003259 ffff88003bc29060 ffffffff8100a57f ffff8801fec321b0
Call Trace:
---[ end trace 1caf04ec1286683a ]---
------------[ cut here ]------------
WARNING: at /home/package/git/hosting-paravirt/src/linux-3.10/arch/x86/xen/multicalls.c:129 xen_end_context_switch+0x9/0x20()
Modules linked in: nfnetlink_log nfnetlink crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel aes_x86_64
CPU: 1 PID: 14240 Comm: mysqld Tainted: G W 3.10.44-xenU-corp-13-930d568-x86_64 #6
ffffffff813e78ee 0000000000000000 ffffffff8104876a ffff88003bc29060
ffff8801f29c4100 0000000000000000 0000000000000001 ffff8801fec2f600
ffffffff81003259 ffff88003bc29060 ffffffff8100a57f ffff8801fec321b0
Call Trace:
---[ end trace 1caf04ec1286683b ]---
------------[ cut here ]------------
WARNING: at /home/package/git/hosting-paravirt/src/linux-3.10/arch/x86/xen/multicalls.c:129 xen_end_context_switch+0x9/0x20()
Modules linked in: nfnetlink_log nfnetlink crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel aes_x86_64
CPU: 1 PID: 27969 Comm: nrpe Tainted: G W 3.10.44-xenU-corp-13-930d568-x86_64 #6
ffffffff813e78ee 0000000000000000 ffffffff8104876a ffff88003bc29060
ffff88000827a710 0000000000000000 0000000000000001 ffff8801fec2f600
ffffffff81003259 ffff88003bc29060 ffffffff8100a57f ffff8801fec321b0
Call Trace:
---[ end trace 1caf04ec1286683c ]---
------------[ cut here ]------------
WARNING: at /home/package/git/hosting-paravirt/src/linux-3.10/arch/x86/xen/multicalls.c:129 xen_end_context_switch+0x9/0x20()
Modules linked in: nfnetlink_log nfnetlink crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel aes_x86_64
CPU: 1 PID: 27358 Comm: rsyslogd Tainted: G W 3.10.44-xenU-corp-13-930d568-x86_64 #6
ffffffff813e78ee 0000000000000000 ffffffff8104876a ffff88003bc29060
ffff880041fda280 0000000000000000 0000000000000001 ffff8801fec2f600
ffffffff81003259 ffff88003bc29060 ffffffff8100a57f ffff8801fec321b0
Call Trace:
---[ end trace 1caf04ec1286683d ]---
------------[ cut here ]------------
WARNING: at /home/package/git/hosting-paravirt/src/linux-3.10/arch/x86/xen/multicalls.c:129 xen_end_context_switch+0x9/0x20()
Modules linked in: nfnetlink_log nfnetlink crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel aes_x86_64
CPU: 1 PID: 27358 Comm: rsyslogd Tainted: G W 3.10.44-xenU-corp-13-930d568-x86_64 #6
ffffffff813e78ee 0000000000000000 ffffffff8104876a ffff88003bc29060
ffff880041fda280 0000000000000000 0000000000000001 ffff8801fec2f600
ffffffff81003259 ffff88003bc29060 ffffffff8100a57f ffff8801fec321b0
Call Trace:
---[ end trace 1caf04ec1286683e ]---
------------[ cut here ]------------
WARNING: at /home/package/git/hosting-paravirt/src/linux-3.10/arch/x86/xen/multicalls.c:129 xen_end_context_switch+0x9/0x20()
Modules linked in: nfnetlink_log nfnetlink crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel aes_x86_64
CPU: 1 PID: 22609 Comm: nrpe Tainted: G W 3.10.44-xenU-corp-13-930d568-x86_64 #6
ffffffff813e78ee 0000000000000000 ffffffff8104876a ffff88003bc29060
ffff88000ac77730 0000000000000000 0000000000000001 ffff8801fec2f600
ffffffff81003259 ffff88003bc29060 ffffffff8100a57f ffff8801fec321b0
Call Trace:
---[ end trace 1caf04ec1286683f ]---
show_signal_msg: 9 callbacks suppressed
nrpe[22606]: segfault at 817 ip 00007f29a2a02dcb sp 00007fffafebccd0 error 4------------[ cut here ]------------
WARNING: at /home/package/git/hosting-paravirt/src/linux-3.10/arch/x86/xen/multicalls.c:129 xen_end_context_switch+0x9/0x20()
Modules linked in: nfnetlink_log nfnetlink crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel aes_x86_64
CPU: 1 PID: 22610 Comm: sh Tainted: G W 3.10.44-xenU-corp-13-930d568-x86_64 #6
ffffffff813e78ee 0000000000000000 ffffffff8104876a ffff88003bc29060
ffff8800b6f7a240 0000000000000000 0000000000000001 ffff8801fec2f600
ffffffff81003259 ffff88003bc29060 ffffffff8100a57f ffff8801fec321b0
Call Trace:
---[ end trace 1caf04ec12866840 ]---
in libc-2.13.so[7f29a2957000+180000]
------------[ cut here ]------------
WARNING: at /home/package/git/hosting-paravirt/src/linux-3.10/arch/x86/xen/multicalls.c:129 __xen_mc_entry+0x7b/0xa0()
Modules linked in: nfnetlink_log nfnetlink crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel aes_x86_64
CPU: 1 PID: 22606 Comm: nrpe Tainted: G W 3.10.44-xenU-corp-13-930d568-x86_64 #6
ffffffff813e78ee 0000000000000000 ffffffff8104876a ffff8801fec2b120
0000000000000003 ffff88019dd00000 000000019dd00000 ffffea0005a85830
ffffffff810045ab ffffffff81749f44 0000000000000018 ffffffff8174b0e4
Call Trace:
[<ffffffff813e78ee>] ? dump_stack+0xd/0x17
[<ffffffff8104876a>] ? warn_slowpath_common+0x6a/0xa0
[<ffffffff810045ab>] ? __xen_mc_entry+0x7b/0xa0
[<ffffffff813ead65>] ? _raw_spin_unlock_irqrestore+0x15/0x20
[<ffffffff8104a333>] ? console_unlock+0x1e3/0x3f0
[<ffffffff81005c1a>] ? xen_extend_mmuext_op+0x5a/0x80
[<ffffffff81005c60>] ? xen_do_pin+0x20/0x50
[<ffffffff813eacf5>] ? _raw_spin_lock+0x5/0x10
[<ffffffff810072ae>] ? xen_unpin_page+0x11e/0x150
[<ffffffff81004739>] ? __raw_callee_save_xen_pmd_val+0x11/0x1e
[<ffffffff81007190>] ? xen_dup_mmap+0x50/0x50
[<ffffffff81004c98>] ? __xen_pgd_walk+0x248/0x310
[<ffffffff810073c8>] ? __xen_pgd_unpin+0xe8/0x150
[<ffffffff81007112>] ? xen_write_cr3+0x92/0xc0
[<ffffffff8100754e>] ? xen_exit_mmap+0x11e/0x190
[<ffffffff810f8cc0>] ? exit_mmap+0x40/0x150
[<ffffffff812a4555>] ? notify_update+0x25/0x30
[<ffffffff81171bc3>] ? do_coredump+0xb3/0xe90
[<ffffffff813ead26>] ? _raw_spin_lock_irqsave+0x26/0x40
[<ffffffff8106c509>] ? lock_hrtimer_base.isra.32+0x29/0x60
[<ffffffff813ead65>] ? _raw_spin_unlock_irqrestore+0x15/0x20
[<ffffffff81045c64>] ? mmput+0x54/0x110
[<ffffffff8104dbbd>] ? do_exit+0x28d/0xa10
[<ffffffff8110afba>] ? kmem_cache_free+0x8a/0x1a0
[<ffffffff8104e3b8>] ? do_group_exit+0x38/0xb0
[<ffffffff81059d5a>] ? get_signal_to_deliver+0x17a/0x540
[<ffffffff8100b383>] ? do_signal+0x63/0x930
[<ffffffff8102fcb9>] ? __do_page_fault+0x429/0x4b0
[<ffffffff813e96bb>] ? __schedule+0x28b/0x770
[<ffffffff813eb6da>] ? error_exit+0x2a/0x60
[<ffffffff813eb23b>] ? retint_restore_args+0x5/0x6
[<ffffffff8100bc99>] ? do_notify_resume+0x49/0x60
[<ffffffff813eb2fc>] ? retint_signal+0x48/0x8c
---[ end trace 1caf04ec12866841 ]---
------------[ cut here ]------------
WARNING: at /home/package/git/hosting-paravirt/src/linux-3.10/arch/x86/xen/multicalls.c:129 __xen_pgd_pin+0x100/0x1b0()
Modules linked in: nfnetlink_log nfnetlink crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel aes_x86_64
CPU: 0 PID: 23817 Comm: nrpe Tainted: G W 3.10.44-xenU-corp-13-930d568-x86_64 #6
ffffffff813e78ee 0000000000000000 ffffffff8104876a ffff88018a641000
000000000438f8d0 ffff8801b4d96000 ffff8801f2d44440 000077ff80000000
ffffffff81006ea0 ffff8801f2d444ac ffff8801f2d44440 ffff8801f2d447c0
Call Trace:
[<ffffffff813e78ee>] ? dump_stack+0xd/0x17
[<ffffffff8104876a>] ? warn_slowpath_common+0x6a/0xa0
[<ffffffff81006ea0>] ? __xen_pgd_pin+0x100/0x1b0
[<ffffffff81007169>] ? xen_dup_mmap+0x29/0x50
[<ffffffff81046453>] ? dup_mm+0x4d3/0x590
[<ffffffff810477fe>] ? copy_process.part.42+0x12be/0x13e0
[<ffffffff8102fb50>] ? __do_page_fault+0x2c0/0x4b0
[<ffffffff81132f03>] ? __d_instantiate+0xd3/0x110
[<ffffffff81047a16>] ? do_fork+0xb6/0x330
[<ffffffff813ebd89>] ? stub_clone+0x69/0x90
[<ffffffff813eba2d>] ? system_call_fastpath+0x1a/0x1f
---[ end trace 1caf04ec12866842 ]---
------------[ cut here ]------------
WARNING: at /home/package/git/hosting-paravirt/src/linux-3.10/arch/x86/xen/multicalls.c:129 xen_end_context_switch+0x9/0x20()
Modules linked in: nfnetlink_log nfnetlink crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel aes_x86_64
CPU: 1 PID: 0 Comm: swapper/1 Tainted: G W 3.10.44-xenU-corp-13-930d568-x86_64 #6
ffffffff813e78ee 0000000000000000 ffffffff8104876a ffff8800129d58b0
ffff8801f3126e60 0000000000000000 0000000000000001 ffff8801fec2f600
ffffffff81003259 ffff8800129d58b0 ffffffff8100a57f ffff8800457cbef8
Call Trace:
---[ end trace 1caf04ec12866843 ]---
------------[ cut here ]------------
WARNING: at /home/package/git/hosting-paravirt/src/linux-3.10/arch/x86/xen/multicalls.c:129 xen_end_context_switch+0x9/0x20()
Modules linked in: nfnetlink_log nfnetlink crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel aes_x86_64
CPU: 1 PID: 21388 Comm: mysqld Tainted: G W 3.10.44-xenU-corp-13-930d568-x86_64 #6
ffffffff813e78ee 0000000000000000 ffffffff8104876a ffff8800129d58b0
ffff8800bad20140 0000000000000000 0000000000000001 ffff8801fec2f600
ffffffff81003259 ffff8800129d58b0 ffffffff8100a57f ffff8801fec321b0
Call Trace:
---[ end trace 1caf04ec12866844 ]---
------------[ cut here ]------------
WARNING: at /home/package/git/hosting-paravirt/src/linux-3.10/arch/x86/xen/multicalls.c:129 xen_end_context_switch+0x9/0x20()
Modules linked in: nfnetlink_log nfnetlink crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel aes_x86_64
CPU: 1 PID: 27358 Comm: rsyslogd Tainted: G W 3.10.44-xenU-corp-13-930d568-x86_64 #6
ffffffff813e78ee 0000000000000000 ffffffff8104876a ffff8800129d58b0
ffff880041fda280 0000000000000000 0000000000000001 ffff8801fec2f600
ffffffff81003259 ffff8800129d58b0 ffffffff8100a57f ffff8801fec321b0
Call Trace:
---[ end trace 1caf04ec12866845 ]---
------------[ cut here ]------------
WARNING: at /home/package/git/hosting-paravirt/src/linux-3.10/arch/x86/xen/multicalls.c:129 xen_end_context_switch+0x9/0x20()
Modules linked in: nfnetlink_log nfnetlink crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel aes_x86_64
CPU: 1 PID: 27358 Comm: rsyslogd Tainted: G W 3.10.44-xenU-corp-13-930d568-x86_64 #6
ffffffff813e78ee 0000000000000000 ffffffff8104876a ffff8800129d58b0
ffff880041fda280 0000000000000000 0000000000000001 ffff8801fec2f600
ffffffff81003259 ffff8800129d58b0 ffffffff8100a57f ffff8801fec321b0
Call Trace:
---[ end trace 1caf04ec12866846 ]---
------------[ cut here ]------------
WARNING: at /home/package/git/hosting-paravirt/src/linux-3.10/arch/x86/xen/multicalls.c:129 xen_end_context_switch+0x9/0x20()
Modules linked in: nfnetlink_log nfnetlink crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel aes_x86_64
CPU: 1 PID: 14162 Comm: mysqld Tainted: G W 3.10.44-xenU-corp-13-930d568-x86_64 #6
ffffffff813e78ee 0000000000000000 ffffffff8104876a ffff8800129d58b0
ffff8800202ae200 0000000000000000 0000000000000001 ffff8801fec2f600
ffffffff81003259 ffff8800129d58b0 ffffffff8100a57f ffff8801fec321b0
Call Trace:
---[ end trace 1caf04ec12866847 ]---
------------[ cut here ]------------
WARNING: at /home/package/git/hosting-paravirt/src/linux-3.10/arch/x86/xen/multicalls.c:129 xen_end_context_switch+0x9/0x20()
Modules linked in: nfnetlink_log nfnetlink crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel aes_x86_64
CPU: 1 PID: 23270 Comm: mysqld Tainted: G W 3.10.44-xenU-corp-13-930d568-x86_64 #6
ffffffff813e78ee 0000000000000000 ffffffff8104876a ffff8800129d58b0
ffff88014142e400 0000000000000000 0000000000000001 ffff8801fec2f600
ffffffff81003259 ffff8800129d58b0 ffffffff8100a57f ffff8801fec321b0
Call Trace:
---[ end trace 1caf04ec12866848 ]---
------------[ cut here ]------------
WARNING: at /home/package/git/hosting-paravirt/src/linux-3.10/arch/x86/xen/multicalls.c:129 xen_end_context_switch+0x9/0x20()
Modules linked in: nfnetlink_log nfnetlink crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel aes_x86_64
CPU: 1 PID: 14168 Comm: mysqld Tainted: G W 3.10.44-xenU-corp-13-930d568-x86_64 #6
ffffffff813e78ee 0000000000000000 ffffffff8104876a ffff8800129d58b0
ffff880012b0e400 0000000000000000 0000000000000001 ffff8801fec2f600
ffffffff81003259 ffff8800129d58b0 ffffffff8100a57f ffff8801fec321b0
Call Trace:
---[ end trace 1caf04ec12866849 ]---
------------[ cut here ]------------
WARNING: at /home/package/git/hosting-paravirt/src/linux-3.10/arch/x86/xen/multicalls.c:129 xen_end_context_switch+0x9/0x20()
Modules linked in: nfnetlink_log nfnetlink crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel aes_x86_64
CPU: 1 PID: 14173 Comm: mysqld Tainted: G W 3.10.44-xenU-corp-13-930d568-x86_64 #6
ffffffff813e78ee 0000000000000000 ffffffff8104876a ffff8800129d58b0
ffff880005402a90 0000000000000000 0000000000000001 ffff8801fec2f600
ffffffff81003259 ffff8800129d58b0 ffffffff8100a57f ffff8801fec321b0
Call Trace:
---[ end trace 1caf04ec1286684a ]---
------------[ cut here ]------------
WARNING: at /home/package/git/hosting-paravirt/src/linux-3.10/arch/x86/xen/multicalls.c:129 xen_end_context_switch+0x9/0x20()
Modules linked in: nfnetlink_log nfnetlink crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel aes_x86_64
CPU: 1 PID: 14163 Comm: mysqld Tainted: G W 3.10.44-xenU-corp-13-930d568-x86_64 #6
ffffffff813e78ee 0000000000000000 ffffffff8104876a ffff8800129d58b0
ffff880008510140 0000000000000000 0000000000000001 ffff8801fec2f600
ffffffff81003259 ffff8800129d58b0 ffffffff8100a57f ffff8801fec321b0
Call Trace:
---[ end trace 1caf04ec1286684b ]---
------------[ cut here ]------------
WARNING: at /home/package/git/hosting-paravirt/src/linux-3.10/arch/x86/xen/multicalls.c:129 xen_end_context_switch+0x9/0x20()
Modules linked in: nfnetlink_log nfnetlink crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel aes_x86_64
CPU: 1 PID: 14163 Comm: mysqld Tainted: G W 3.10.44-xenU-corp-13-930d568-x86_64 #6
ffffffff813e78ee 0000000000000000 ffffffff8104876a ffff8800129d58b0
ffff880008510140 0000000000000000 0000000000000001 ffff8801fec2f600
ffffffff81003259 ffff8800129d58b0 ffffffff8100a57f ffff8801fec321b0
Call Trace:
---[ end trace 1caf04ec1286684c ]---
------------[ cut here ]------------
WARNING: at /home/package/git/hosting-paravirt/src/linux-3.10/arch/x86/xen/multicalls.c:129 xen_end_context_switch+0x9/0x20()
Modules linked in: nfnetlink_log nfnetlink crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel aes_x86_64
CPU: 1 PID: 23819 Comm: nrpe Tainted: G W 3.10.44-xenU-corp-13-930d568-x86_64 #6
ffffffff813e78ee 0000000000000000 ffffffff8104876a ffff8800129d58b0
ffff88000cc18750 0000000000000000 0000000000000001 ffff8801fec2f600
ffffffff81003259 ffff8800129d58b0 ffffffff8100a57f ffff8801fec321b0
Call Trace:
---[ end trace 1caf04ec1286684d ]---
------------[ cut here ]------------
WARNING: at /home/package/git/hosting-paravirt/src/linux-3.10/arch/x86/xen/multicalls.c:129 xen_end_context_switch+0x9/0x20()
Modules linked in: nfnetlink_log nfnetlink crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel aes_x86_64
CPU: 1 PID: 27353 Comm: rs:main Q:Reg Tainted: G W 3.10.44-xenU-corp-13-930d568-x86_64 #6
ffffffff813e78ee 0000000000000000 ffffffff8104876a ffff8800129d58b0
ffff880003cf74b0 0000000000000000 0000000000000001 ffff8801fec2f600
ffffffff81003259 ffff8800129d58b0 ffffffff8100a57f ffff8801fec321b0
Call Trace:
---[ end trace 1caf04ec1286684e ]---
------------[ cut here ]------------
WARNING: at /home/package/git/hosting-paravirt/src/linux-3.10/arch/x86/xen/multicalls.c:129 xen_end_context_switch+0x9/0x20()
Modules linked in: nfnetlink_log nfnetlink crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel aes_x86_64
CPU: 1 PID: 27353 Comm: rs:main Q:Reg Tainted: G W 3.10.44-xenU-corp-13-930d568-x86_64 #6
ffffffff813e78ee 0000000000000000 ffffffff8104876a ffff8800129d58b0
ffff880003cf74b0 0000000000000000 0000000000000001 ffff8801fec2f600
ffffffff81003259 ffff8800129d58b0 ffffffff8100a57f ffff8801fec321b0
Call Trace:
---[ end trace 1caf04ec1286684f ]---
------------[ cut here ]------------
WARNING: at /home/package/git/hosting-paravirt/src/linux-3.10/arch/x86/xen/multicalls.c:129 xen_end_context_switch+0x9/0x20()
Modules linked in: nfnetlink_log nfnetlink crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel aes_x86_64
CPU: 1 PID: 27353 Comm: rs:main Q:Reg Tainted: G W 3.10.44-xenU-corp-13-930d568-x86_64 #6
ffffffff813e78ee 0000000000000000 ffffffff8104876a ffff8800129d58b0
ffff880003cf74b0 0000000000000000 0000000000000001 ffff8801fec2f600
ffffffff81003259 ffff8800129d58b0 ffffffff8100a57f ffff8801fec321b0
Call Trace:
---[ end trace 1caf04ec12866850 ]---
------------[ cut here ]------------
WARNING: at /home/package/git/hosting-paravirt/src/linux-3.10/arch/x86/xen/multicalls.c:129 xen_end_context_switch+0x9/0x20()
Modules linked in: nfnetlink_log nfnetlink crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel aes_x86_64
CPU: 1 PID: 14015 Comm: kworker/u32:2 Tainted: G W 3.10.44-xenU-corp-13-930d568-x86_64 #6
ffffffff813e78ee 0000000000000000 ffffffff8104876a ffff8800129d58b0
ffff88002e12cb10 0000000000000000 0000000000000001 ffff8801fec2f600
ffffffff81003259 ffff8800129d58b0 ffffffff8100a57f ffff8801fec321b0
Call Trace:
---[ end trace 1caf04ec12866851 ]---
------------[ cut here ]------------
WARNING: at /home/package/git/hosting-paravirt/src/linux-3.10/arch/x86/xen/multicalls.c:129 xen_end_context_switch+0x9/0x20()
Modules linked in: nfnetlink_log nfnetlink crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel aes_x86_64
CPU: 1 PID: 23821 Comm: nrpe Tainted: G W 3.10.44-xenU-corp-13-930d568-x86_64 #6
ffffffff813e78ee 0000000000000000 ffffffff8104876a ffff8800129d58b0
ffff8801f2d52910 0000000000000000 0000000000000001 ffff8801fec2f600
ffffffff81003259 ffff8800129d58b0 ffffffff8100a57f ffff8801fec321b0
Call Trace:
---[ end trace 1caf04ec12866852 ]---
nrpe[23818]: segfault at 817 ip 00007f29a2a02dcb sp 00007fffafebccd0 error 4 in libc-2.13.so[7f29a2957000+180000]
------------[ cut here ]------------
WARNING: at /home/package/git/hosting-paravirt/src/linux-3.10/arch/x86/xen/multicalls.c:129 __xen_mc_entry+0x7b/0xa0()
Modules linked in: nfnetlink_log nfnetlink crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel aes_x86_64
CPU: 1 PID: 23818 Comm: nrpe Tainted: G W 3.10.44-xenU-corp-13-930d568-x86_64 #6
ffffffff813e78ee 0000000000000000 ffffffff8104876a ffff8801fec2b120
0000000000000001 ffff880145176000 0000000145176000 0000000000000000
ffffffff810045ab 0000000000007ff0 0000000000000000 0000000000000150
Call Trace:
[<ffffffff813e78ee>] ? dump_stack+0xd/0x17
[<ffffffff8104876a>] ? warn_slowpath_common+0x6a/0xa0
[<ffffffff810045ab>] ? __xen_mc_entry+0x7b/0xa0
[<ffffffff8100689f>] ? pte_mfn_to_pfn+0x5f/0xf0
[<ffffffff81007204>] ? xen_unpin_page+0x74/0x150
[<ffffffff810046df>] ? __raw_callee_save_xen_pgd_val+0x11/0x1e
[<ffffffff81007190>] ? xen_dup_mmap+0x50/0x50
[<ffffffff81004b6a>] ? __xen_pgd_walk+0x11a/0x310
[<ffffffff810073c8>] ? __xen_pgd_unpin+0xe8/0x150
[<ffffffff81007112>] ? xen_write_cr3+0x92/0xc0
[<ffffffff8100754e>] ? xen_exit_mmap+0x11e/0x190
[<ffffffff810f8cc0>] ? exit_mmap+0x40/0x150
[<ffffffff812a4555>] ? notify_update+0x25/0x30
[<ffffffff81171bc3>] ? do_coredump+0xb3/0xe90
[<ffffffff813ead26>] ? _raw_spin_lock_irqsave+0x26/0x40
[<ffffffff8106c509>] ? lock_hrtimer_base.isra.32+0x29/0x60
[<ffffffff813ead65>] ? _raw_spin_unlock_irqrestore+0x15/0x20
[<ffffffff81045c64>] ? mmput+0x54/0x110
[<ffffffff8104dbbd>] ? do_exit+0x28d/0xa10
[<ffffffff8110afba>] ? kmem_cache_free+0x8a/0x1a0
[<ffffffff8104e3b8>] ? do_group_exit+0x38/0xb0
[<ffffffff81059d5a>] ? get_signal_to_deliver+0x17a/0x540
[<ffffffff8100b383>] ? do_signal+0x63/0x930
[<ffffffff8102fcb9>] ? __do_page_fault+0x429/0x4b0
[<ffffffff813eb6da>] ? error_exit+0x2a/0x60
[<ffffffff813eb23b>] ? retint_restore_args+0x5/0x6
[<ffffffff8100bc99>] ? do_notify_resume+0x49/0x60
[<ffffffff813eb2fc>] ? retint_signal+0x48/0x8c
---[ end trace 1caf04ec12866853 ]---
------------[ cut here ]------------
WARNING: at /home/package/git/hosting-paravirt/src/linux-3.10/arch/x86/xen/multicalls.c:129 xen_alloc_pte+0x1ad/0x1b0()
Modules linked in: nfnetlink_log nfnetlink crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel aes_x86_64
CPU: 1 PID: 23831 Comm: sh Tainted: G W 3.10.44-xenU-corp-13-930d568-x86_64 #6
ffffffff813e78ee 0000000000000000 ffffffff8104876a 0000000000503a77
ffff880192361000 ffff8801f2d4412c ffff8801a4d44cc0 ffff8801f2d440c0
ffffffff810064ed ffff8801154cd000 ffff8801fec2b930 ffff8801fec2b170
Call Trace:
[<ffffffff813e78ee>] ? dump_stack+0xd/0x17
[<ffffffff8104876a>] ? warn_slowpath_common+0x6a/0xa0
[<ffffffff810064ed>] ? xen_alloc_pte+0x1ad/0x1b0
[<ffffffff810eeaaa>] ? __pte_alloc+0x8a/0x180
[<ffffffff810f3035>] ? handle_mm_fault+0x375/0x390
[<ffffffff8102f9fa>] ? __do_page_fault+0x16a/0x4b0
[<ffffffff810f6456>] ? vma_link+0x86/0xe0
[<ffffffff810f83e7>] ? mmap_region+0x167/0x640
[<ffffffff810f56e6>] ? vma_gap_update+0x26/0x30
[<ffffffff810f6382>] ? __vma_link_rb+0x62/0xb0
[<ffffffff813eb4d8>] ? page_fault+0x28/0x30
[<ffffffff8126fd08>] ? __clear_user+0x38/0x70
[<ffffffff8126fce9>] ? __clear_user+0x19/0x70
[<ffffffff81169a81>] ? padzero+0x21/0x40
[<ffffffff8116ba6f>] ? load_elf_binary+0x97f/0x19d0
[<ffffffff81124397>] ? copy_strings.isra.21+0x297/0x2e0
[<ffffffff8116b0f0>] ? elf_core_dump+0x1420/0x1420
[<ffffffff8112357e>] ? search_binary_handler+0x10e/0x2b0
[<ffffffff811256a9>] ? do_execve_common.isra.25+0x519/0x5e0
[<ffffffff811259d6>] ? SyS_execve+0x36/0x60
[<ffffffff813ebfd9>] ? stub_execve+0x69/0xa0
---[ end trace 1caf04ec12866854 ]---
------------[ cut here ]------------
WARNING: at /home/package/git/hosting-paravirt/src/linux-3.10/arch/x86/xen/multicalls.c:129 xen_set_pmd_hyper+0x85/0x90()
Modules linked in: nfnetlink_log nfnetlink crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel aes_x86_64
CPU: 1 PID: 23831 Comm: sh Tainted: G W 3.10.44-xenU-corp-13-930d568-x86_64 #6
ffffffff813e78ee 0000000000000000 ffffffff8104876a 8000000000000200
ffff880126b8c018 0000000503a77067 ffff8801a4d44cc0 ffff8801f2d440c0
ffffffff810062d5 00000004faab1018 0000000503a77067 0000000000192361
Call Trace:
[<ffffffff813e78ee>] ? dump_stack+0xd/0x17
[<ffffffff8104876a>] ? warn_slowpath_common+0x6a/0xa0
[<ffffffff810062d5>] ? xen_set_pmd_hyper+0x85/0x90
[<ffffffff810eeac9>] ? __pte_alloc+0xa9/0x180
[<ffffffff810f3035>] ? handle_mm_fault+0x375/0x390
[<ffffffff8102f9fa>] ? __do_page_fault+0x16a/0x4b0
[<ffffffff810f6456>] ? vma_link+0x86/0xe0
[<ffffffff810f83e7>] ? mmap_region+0x167/0x640
[<ffffffff810f56e6>] ? vma_gap_update+0x26/0x30
[<ffffffff810f6382>] ? __vma_link_rb+0x62/0xb0
[<ffffffff813eb4d8>] ? page_fault+0x28/0x30
[<ffffffff8126fd08>] ? __clear_user+0x38/0x70
[<ffffffff8126fce9>] ? __clear_user+0x19/0x70
[<ffffffff81169a81>] ? padzero+0x21/0x40
[<ffffffff8116ba6f>] ? load_elf_binary+0x97f/0x19d0
[<ffffffff81124397>] ? copy_strings.isra.21+0x297/0x2e0
[<ffffffff8116b0f0>] ? elf_core_dump+0x1420/0x1420
[<ffffffff8112357e>] ? search_binary_handler+0x10e/0x2b0
[<ffffffff811256a9>] ? do_execve_common.isra.25+0x519/0x5e0
[<ffffffff811259d6>] ? SyS_execve+0x36/0x60
[<ffffffff813ebfd9>] ? stub_execve+0x69/0xa0
---[ end trace 1caf04ec12866855 ]---
------------[ cut here ]------------
WARNING: at /home/package/git/hosting-paravirt/src/linux-3.10/mm/mmap.c:2757 exit_mmap+0x14c/0x150()
Modules linked in: nfnetlink_log nfnetlink crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel aes_x86_64
CPU: 0 PID: 23831 Comm: sh Tainted: G W 3.10.44-xenU-corp-13-930d568-x86_64 #6
ffffffff813e78ee 0000000000000000 ffffffff8104876a 0000000000000059
ffff8801f2d440c0 ffff8801f2d440c0 ffff8800129d5010 ffff8801f2d44130
ffffffff810f8dcc 000000000000006a ffff880005b17b50 ffff8801f2d440c0
Call Trace:
[<ffffffff813e78ee>] ? dump_stack+0xd/0x17
[<ffffffff8104876a>] ? warn_slowpath_common+0x6a/0xa0
[<ffffffff810f8dcc>] ? exit_mmap+0x14c/0x150
[<ffffffff81003422>] ? xen_clts+0x62/0x80
[<ffffffff81045c64>] ? mmput+0x54/0x110
[<ffffffff8104dbbd>] ? do_exit+0x28d/0xa10
[<ffffffff8104e3b8>] ? do_group_exit+0x38/0xb0
[<ffffffff8104e442>] ? SyS_exit_group+0x12/0x20
[<ffffffff813ebc4b>] ? int_ret_from_sys_call+0x7/0xd
[<ffffffff813eba2d>] ? system_call_fastpath+0x1a/0x1f
---[ end trace 1caf04ec12866856 ]---
BUG: Bad rss-counter state mm:ffff8801f2d440c0 idx:1 val:1
BUG: unable to handle kernel paging request at ffff880192361000
IP: [<ffffffff8126f62d>] memcpy+0xd/0x110
PGD 160e067 PUD 1fffe4067 PMD 1fff52067 PTE 8010000192361065
Oops: 0003 [#1] SMP
Modules linked in: nfnetlink_log nfnetlink crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel aes_x86_64
CPU: 1 PID: 25498 Comm: kworker/u32:2 Tainted: G W 3.10.44-xenU-corp-13-930d568-x86_64 #6
Workqueue: writeback bdi_writeback_workfn (flush-202:1)
task: ffff880108db43c0 ti: ffff88003218e000 task.ti: ffff88003218e000
RIP: e030:[<ffffffff8126f62d>] [<ffffffff8126f62d>] memcpy+0xd/0x110
RSP: e02b:ffff88003218fa30 EFLAGS: 00010046
RAX: ffff880192361000 RBX: 0000000000000000 RCX: 0000000000000200
RDX: 0000000000000000 RSI: ffff88009beda000 RDI: ffff880192361000
RBP: ffff8801f2a98900 R08: ffff88003218e000 R09: 6db6db6db6db6db7
R10: ffff8801f2d042d0 R11: ffff880000000000 R12: ffff8801f2d042d0
R13: ffff8801f2d04328 R14: ffff8801f2d00000 R15: 0000000000000000
FS: 0000000000000000(0000) GS:ffff8801fec20000(0000) knlGS:ffff8801fec20000
CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: ffff880192361000 CR3: 000000015184d000 CR4: 0000000000002660
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
ffffffff812bfe3a ffff880107e87830 ffff88003218e000 0000000000000021
0000000700000000 ffff8801f2a98900 ffff8801f2d046b8 ffff8801421f9058
ffff8801f2d00000 ffff88003218ffd8 000000000000001e 0000000000000014
Call Trace:
[<ffffffff812bfe3a>] ? do_blkif_request+0x35a/0x6a0
[<ffffffff81242d8a>] ? __blk_run_queue+0x2a/0x40
[<ffffffff812456fd>] ? queue_unplugged.isra.53+0xd/0x30
[<ffffffff81246159>] ? blk_flush_plug_list+0x1b9/0x210
[<ffffffff812461bb>] ? blk_finish_plug+0xb/0x30
[<ffffffff810d8de0>] ? generic_writepages+0x50/0x70
[<ffffffff8114349e>] ? __writeback_single_inode+0x3e/0x140
[<ffffffff8114467b>] ? writeback_sb_inodes+0x1ab/0x380
[<ffffffff811448de>] ? __writeback_inodes_wb+0x8e/0xc0
[<ffffffff81144ae3>] ? wb_writeback+0x1d3/0x1e0
[<ffffffff81137142>] ? get_nr_inodes+0x42/0x60
[<ffffffff81144e2b>] ? wb_do_writeback+0x19b/0x1a0
[<ffffffff81144e9a>] ? bdi_writeback_workfn+0x6a/0x140
[<ffffffff810629ae>] ? process_one_work+0x13e/0x380
[<ffffffff81063962>] ? worker_thread+0x112/0x370
[<ffffffff81063850>] ? manage_workers.isra.28+0x2a0/0x2a0
[<ffffffff81069203>] ? kthread+0xb3/0xc0
[<ffffffff81070000>] ? set_current_groups+0x10/0x70
[<ffffffff81069150>] ? kthread_freezable_should_stop+0x60/0x60
[<ffffffff813eb97c>] ? ret_from_fork+0x7c/0xb0
[<ffffffff81069150>] ? kthread_freezable_should_stop+0x60/0x60
Code: fc ff ff 48 8b 43 58 48 2b 43 50 88 43 4e eb e9 90 90 90 90 90 90 90 90 90 90 90 90 90 90 48 89 f8 48 89 d1 48 c1 e9 03 83 e2 07 <f3> 48 a5 89 d1 f3 a4 c3 20 4c 8b 06 4c 8b 4e 08 4c 8b 56 10 4c
RIP [<ffffffff8126f62d>] memcpy+0xd/0x110
RSP <ffff88003218fa30>
CR2: ffff880192361000
---[ end trace 1caf04ec12866857 ]---
note: kworker/u32:2[25498] exited with preempt_count 2
BUG: unable to handle kernel paging request at ffffffffffffffd8
IP: [<ffffffff81069607>] kthread_data+0x7/0x10
PGD 1610067 PUD 1612067 PMD 0
Oops: 0000 [#2] SMP
Modules linked in: nfnetlink_log nfnetlink crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel aes_x86_64
CPU: 1 PID: 25498 Comm: kworker/u32:2 Tainted: G D W 3.10.44-xenU-corp-13-930d568-x86_64 #6
task: ffff880108db43c0 ti: ffff88003218e000 task.ti: ffff88003218e000
RIP: e030:[<ffffffff81069607>] [<ffffffff81069607>] kthread_data+0x7/0x10
RSP: e02b:ffff88003218f650 EFLAGS: 00010002
RAX: 0000000000000000 RBX: 0000000000000001 RCX: 0000000000000001
RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff880108db43c0
RBP: ffff880108db43c0 R08: 00000000000003ed R09: ffff8801f29d8ab0
R10: 0000000000000000 R11: dead000000200200 R12: ffff880108db46b8
R13: 0000000000000001 R14: ffff880192361000 R15: ffff880108db43b0
FS: 0000000000000000(0000) GS:ffff8801fec20000(0000) knlGS:ffff8801fec20000
CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000028 CR3: 000000015184d000 CR4: 0000000000002660
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
ffffffff81063e28 ffff8801fec32140 ffffffff813e97fb ffff8801f5002980
0000000019c845c8 ffff88003218ffd8 ffff88003218ffd8 ffff88003218ffd8
ffff880108db43c0 ffffffff81641280 0000000000000000 ffff88003218f700
Call Trace:
[<ffffffff81063e28>] ? wq_worker_sleeping+0x8/0x80
[<ffffffff813e97fb>] ? __schedule+0x3cb/0x770
[<ffffffff812ad3d4>] ? mix_pool_bytes.constprop.15+0x44/0x70
[<ffffffff813ead65>] ? _raw_spin_unlock_irqrestore+0x15/0x20
[<ffffffff810be292>] ? __call_rcu.constprop.56+0xe2/0x260
[<ffffffff813ead26>] ? _raw_spin_lock_irqsave+0x26/0x40
[<ffffffff8110afba>] ? kmem_cache_free+0x8a/0x1a0
[<ffffffff813ead26>] ? _raw_spin_lock_irqsave+0x26/0x40
[<ffffffff8104dff5>] ? do_exit+0x6c5/0xa10
[<ffffffff8100f182>] ? oops_end+0xa2/0xf0
[<ffffffff813e4ccb>] ? no_context+0x1ee/0x1fd
[<ffffffff8102fb50>] ? __do_page_fault+0x2c0/0x4b0
[<ffffffff81244721>] ? bio_attempt_back_merge+0x71/0x90
[<ffffffff812464b6>] ? blk_queue_bio+0x2d6/0x360
[<ffffffff81244532>] ? generic_make_request+0xb2/0x100
[<ffffffff813eb4d8>] ? page_fault+0x28/0x30
[<ffffffff8126f62d>] ? memcpy+0xd/0x110
[<ffffffff812bfe3a>] ? do_blkif_request+0x35a/0x6a0
[<ffffffff81242d8a>] ? __blk_run_queue+0x2a/0x40
[<ffffffff812456fd>] ? queue_unplugged.isra.53+0xd/0x30
[<ffffffff81246159>] ? blk_flush_plug_list+0x1b9/0x210
[<ffffffff812461bb>] ? blk_finish_plug+0xb/0x30
[<ffffffff810d8de0>] ? generic_writepages+0x50/0x70
[<ffffffff8114349e>] ? __writeback_single_inode+0x3e/0x140
[<ffffffff8114467b>] ? writeback_sb_inodes+0x1ab/0x380
[<ffffffff811448de>] ? __writeback_inodes_wb+0x8e/0xc0
[<ffffffff81144ae3>] ? wb_writeback+0x1d3/0x1e0
[<ffffffff81137142>] ? get_nr_inodes+0x42/0x60
[<ffffffff81144e2b>] ? wb_do_writeback+0x19b/0x1a0
[<ffffffff81144e9a>] ? bdi_writeback_workfn+0x6a/0x140
[<ffffffff810629ae>] ? process_one_work+0x13e/0x380
[<ffffffff81063962>] ? worker_thread+0x112/0x370
[<ffffffff81063850>] ? manage_workers.isra.28+0x2a0/0x2a0
[<ffffffff81069203>] ? kthread+0xb3/0xc0
[<ffffffff81070000>] ? set_current_groups+0x10/0x70
[<ffffffff81069150>] ? kthread_freezable_should_stop+0x60/0x60
[<ffffffff813eb97c>] ? ret_from_fork+0x7c/0xb0
[<ffffffff81069150>] ? kthread_freezable_should_stop+0x60/0x60
Code: 00 00 00 00 65 48 8b 04 25 80 c6 00 00 48 8b 80 a0 02 00 00 48 8b 40 c8 48 c1 e8 02 83 e0 01 c3 0f 1f 40 00 48 8b 87 a0 02 00 00 <48> 8b 40 d8 c3 0f 1f 40 00 48 83 ec 18 48 8b b7 a0 02 00 00 ba
RIP [<ffffffff81069607>] kthread_data+0x7/0x10
RSP <ffff88003218f650>
CR2: ffffffffffffffd8
---[ end trace 1caf04ec12866858 ]---
Fixing recursive fault but reboot is needed!
[-- Attachment #6: domu5 --]
[-- Type: text/plain, Size: 5577 bytes --]
xen:grant_table: Grant tables using version 2 layout
PM: noirq restore of devices complete after 0.038 msecs
PM: early restore of devices complete after 0.014 msecs
PM: restore of devices complete after 36.616 msecs
------------[ cut here ]------------
WARNING: at /home/package/gitlab/hosting-paravirt/src/linux-3.10/arch/x86/xen/multicalls.c:129 xen_alloc_pte+0x195/0x1a0()
Modules linked in:
CPU: 0 PID: 29313 Comm: sshd Not tainted 3.10.99-xenU-corp-26-7336ef7-x86_64 #18
ffffffff813d2abf ffffffff81048136 3fffffffffffffff ffff8800e3925dc0
ffff88007ee0a1d0 ffff8800db4d3d10 ffff8800450f5a28 ffffffff81006205
ffff88007ee0a990 00000000000abefe ffff880074eb3cac ffff880074eb3c40
Call Trace:
[<ffffffff813d2abf>] ? dump_stack+0xc/0x15
[<ffffffff81048136>] ? warn_slowpath_common+0x56/0x70
[<ffffffff81006205>] ? xen_alloc_pte+0x195/0x1a0
[<ffffffff810eb2b2>] ? __pte_alloc+0x72/0x160
[<ffffffff810ef7fc>] ? handle_mm_fault+0x3ec/0x430
[<ffffffff8102f7f2>] ? __do_page_fault+0x172/0x4a0
[<ffffffff810f51e2>] ? do_mmap_pgoff+0x2e2/0x3d0
[<ffffffff810e240b>] ? vm_mmap_pgoff+0x9b/0xc0
[<ffffffff813d6458>] ? page_fault+0x28/0x30
---[ end trace 4c854b554637324b ]---
------------[ cut here ]------------
WARNING: at /home/package/gitlab/hosting-paravirt/src/linux-3.10/arch/x86/xen/multicalls.c:129 xen_set_pmd_hyper+0x75/0x80()
Modules linked in:
CPU: 0 PID: 29313 Comm: sshd Tainted: G W 3.10.99-xenU-corp-26-7336ef7-x86_64 #18
ffffffff813d2abf ffffffff81048136 0000000bd1815067 ffff8800db4d3d10
4000000000000200 ffff8800db4d3d10 ffff8800450f5a28 ffffffff810054d5
0000000bf7d92d10 0000000bd1815067 00000000abefe000 ffff880074eb3c40
Call Trace:
[<ffffffff813d2abf>] ? dump_stack+0xc/0x15
[<ffffffff81048136>] ? warn_slowpath_common+0x56/0x70
[<ffffffff810054d5>] ? xen_set_pmd_hyper+0x75/0x80
[<ffffffff810eb2d1>] ? __pte_alloc+0x91/0x160
[<ffffffff810ef7fc>] ? handle_mm_fault+0x3ec/0x430
[<ffffffff8102f7f2>] ? __do_page_fault+0x172/0x4a0
[<ffffffff810f51e2>] ? do_mmap_pgoff+0x2e2/0x3d0
[<ffffffff810e240b>] ? vm_mmap_pgoff+0x9b/0xc0
[<ffffffff813d6458>] ? page_fault+0x28/0x30
---[ end trace 4c854b554637324c ]---
------------[ cut here ]------------
WARNING: at /home/package/gitlab/hosting-paravirt/src/linux-3.10/mm/mmap.c:2760 exit_mmap+0x153/0x160()
Modules linked in:
CPU: 0 PID: 29313 Comm: sshd Tainted: G W 3.10.99-xenU-corp-26-7336ef7-x86_64 #18
ffffffff813d2abf ffffffff81048136 0000000000000101 ffff880074eb3c40
ffff880074eb3c40 ffff880074eb3cb0 ffff8800d26d2650 ffffffff810f5433
ffff880074914b88 ffff880074eb3c40 0000000000000000 ffffffffffffffff
Call Trace:
[<ffffffff813d2abf>] ? dump_stack+0xc/0x15
[<ffffffff81048136>] ? warn_slowpath_common+0x56/0x70
[<ffffffff810f5433>] ? exit_mmap+0x153/0x160
[<ffffffff810455b4>] ? mmput+0x54/0x110
[<ffffffff8104d6be>] ? do_exit+0x2be/0xa70
[<ffffffff8104ded4>] ? do_group_exit+0x34/0xb0
[<ffffffff8104df5b>] ? SyS_exit_group+0xb/0x10
[<ffffffff813d698d>] ? system_call_fastpath+0x1a/0x1f
---[ end trace 4c854b554637324d ]---
BUG: Bad rss-counter state mm:ffff880074eb3c40 idx:0 val:1
BUG: unable to handle kernel paging request at ffff8800abefe000
IP: [<ffffffff8126303d>] memcpy+0xd/0x110
PGD 160e067 PUD 7fc16067 PMD 7fab6067 PTE 80100000abefe065
Oops: 0003 [#1] SMP
Modules linked in:
CPU: 0 PID: 401 Comm: jbd2/xvda1-8 Tainted: G W 3.10.99-xenU-corp-26-7336ef7-x86_64 #18
task: ffff8800756d5880 ti: ffff880075168000 task.ti: ffff880075168000
RIP: e030:[<ffffffff8126303d>] [<ffffffff8126303d>] memcpy+0xd/0x110
RSP: e02b:ffff880075169bc8 EFLAGS: 00010046
RAX: ffff8800abefe000 RBX: ffff880000000000 RCX: 0000000000000200
RDX: 0000000000000000 RSI: ffff88003d3b1000 RDI: ffff8800abefe000
RBP: ffff8800753cd6f0 R08: 0000000000000000 R09: 0000000000000af0
R10: ffff88003d3b1000 R11: 0000000000000000 R12: ffff8800750b8c98
R13: ffff8800750b8000 R14: 0000000000000000 R15: ffff8800750b8c28
FS: 00007fe94d318700(0000) GS:ffff88007ee00000(0000) knlGS:0000000000000000
CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: ffff8800abefe000 CR3: 0000000074ba4000 CR4: 0000000000002660
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
ffffffff812b1c37 0000000000000001 ffff8800753cd000 0000000700000000
ffff8800750ea7e0 ffff88006e9d2348 00000000000006d0 ffff8800750bc6b8
0000000000000005 0000000000000032 0000000000000000 000000000000000f
Call Trace:
[<ffffffff812b1c37>] ? do_blkif_request+0x387/0x760
[<ffffffff81237afa>] ? __blk_run_queue+0x2a/0x40
[<ffffffff81237c3d>] ? queue_unplugged+0xd/0x30
[<ffffffff8123ab64>] ? blk_flush_plug_list+0x1d4/0x230
[<ffffffff8123af1b>] ? blk_finish_plug+0xb/0x40
[<ffffffff811c67fd>] ? jbd2_journal_commit_transaction+0xcbd/0x1530
[<ffffffff813d5cea>] ? _raw_spin_lock_irqsave+0x1a/0x30
[<ffffffff811c9fea>] ? kjournald2+0xca/0x270
[<ffffffff81068b60>] ? wake_up_bit+0x20/0x20
[<ffffffff811c9f20>] ? commit_timeout+0x10/0x10
[<ffffffff81067f62>] ? kthread+0xc2/0xd0
[<ffffffff81067ea0>] ? kthread_create_on_node+0x120/0x120
[<ffffffff813d68d8>] ? ret_from_fork+0x58/0x90
[<ffffffff81067ea0>] ? kthread_create_on_node+0x120/0x120
Code: fb ff ff 48 8b 43 58 48 2b 43 50 88 43 4e 5b f3 c3 90 90 90 90 90 90 90 90 90 90 90 90 90 48 89 f8 48 89 d1 48 c1 e9 03 83 e2 07 <f3> 48 a5 89 d1 f3 a4 c3 20 4c 8b 06 4c 8b 4e 08 4c 8b 56 10 4c
RIP [<ffffffff8126303d>] memcpy+0xd/0x110
RSP <ffff880075169bc8>
CR2: ffff8800abefe000
---[ end trace 4c854b554637324e ]---
note: jbd2/xvda1-8[401] exited with preempt_count 2
[-- Attachment #7: Type: text/plain, Size: 127 bytes --]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: several domU crashes after 4.1->4.8 live migration
2017-02-02 7:58 several domU crashes after 4.1->4.8 live migration Vincent Legout
@ 2017-02-02 11:22 ` George Dunlap
2017-02-02 12:57 ` Vincent Legout
2017-02-02 12:05 ` Andrew Cooper
1 sibling, 1 reply; 8+ messages in thread
From: George Dunlap @ 2017-02-02 11:22 UTC (permalink / raw)
To: Vincent Legout; +Cc: xen-devel
On Thu, Feb 2, 2017 at 7:58 AM, Vincent Legout <vincent.legout@gandi.net> wrote:
> Hello,
>
> We had some issues after live migrating several domU from xen 4.1 to xen
> 4.8. We migrated around 200 domU and 5 crashed, from a few hours up to
> several days after the migration. All the domU had more than 1 year of
> uptime, and for example one crashed several days after the migration
> during a high load period.
>
> All 5 domU are running a 3.10 kernel (from 3.10.44 to 3.10.103). They
> have between 2GB and 16GB of RAM, and between 1 and 4 vCPUS.
>
> We use 3 types of machines (several PowerEdge C6100 (Intel L5640) and
> R710 (Intel L5520), and one C8220 (Intel E5-2650)). The C6100 and R710
> have 24 logical cores with HT enabled. The PowerEdge C8220 is only used
> for Xen 4.8, and has 32 logical cores with HT enabled. The C6100 and the
> R710 have 50GB of RAM, and the C8220 128GB. The xen 4.1 dom0 is running
> a 3.4.69 kernel, and the xen 4.8 one a 4.1.37 kernel.
>
> I've attached the most relevant parts of the domU kernel logs we could
> get. It seems the crashes came from different components of the kernel,
> though most of them seem to be related to memory.
>
> Would anyone have any idea if that's something that could be fixed? Or
> is it just that migrating from 4.1 to 4.8 is not supported?
Officially the open-source project only supports migrations between
single releases -- i.e., 4.1 -> 4.2 or 4.7 -> 4.8. This is mainly
because we haven't had the bandwidth to test more than that. I know
XenServer tests and supports a broader window for migrations between
their release versions; other downstreams (Oracle, SLES) may as well;
but that probably won't help you in your current situation.
-George
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: several domU crashes after 4.1->4.8 live migration
2017-02-02 7:58 several domU crashes after 4.1->4.8 live migration Vincent Legout
2017-02-02 11:22 ` George Dunlap
@ 2017-02-02 12:05 ` Andrew Cooper
2017-02-02 13:30 ` Vincent Legout
1 sibling, 1 reply; 8+ messages in thread
From: Andrew Cooper @ 2017-02-02 12:05 UTC (permalink / raw)
To: Vincent Legout, xen-devel
On 02/02/17 07:58, Vincent Legout wrote:
> Hello,
>
> We had some issues after live migrating several domU from xen 4.1 to xen
> 4.8. We migrated around 200 domU and 5 crashed, from a few hours up to
> several days after the migration. All the domU had more than 1 year of
> uptime, and for example one crashed several days after the migration
> during a high load period.
>
> All 5 domU are running a 3.10 kernel (from 3.10.44 to 3.10.103). They
> have between 2GB and 16GB of RAM, and between 1 and 4 vCPUS.
>
> We use 3 types of machines (several PowerEdge C6100 (Intel L5640) and
> R710 (Intel L5520), and one C8220 (Intel E5-2650)). The C6100 and R710
> have 24 logical cores with HT enabled. The PowerEdge C8220 is only used
> for Xen 4.8, and has 32 logical cores with HT enabled. The C6100 and the
> R710 have 50GB of RAM, and the C8220 128GB. The xen 4.1 dom0 is running
> a 3.4.69 kernel, and the xen 4.8 one a 4.1.37 kernel.
>
> I've attached the most relevant parts of the domU kernel logs we could
> get. It seems the crashes came from different components of the kernel,
> though most of them seem to be related to memory.
>
> Would anyone have any idea if that's something that could be fixed? Or
> is it just that migrating from 4.1 to 4.8 is not supported?
Do the VMs migrate normally in production?
This looks like a Linux kernel bug in the Xen suspend/resume paths, and
unlikely to be related to the version of the hypervisor in use.
~Andrew
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: several domU crashes after 4.1->4.8 live migration
2017-02-02 11:22 ` George Dunlap
@ 2017-02-02 12:57 ` Vincent Legout
0 siblings, 0 replies; 8+ messages in thread
From: Vincent Legout @ 2017-02-02 12:57 UTC (permalink / raw)
To: George Dunlap; +Cc: xen-devel
On Thu, Feb 02, 2017 at 11:22:02AM +0000, George Dunlap wrote :
> On Thu, Feb 2, 2017 at 7:58 AM, Vincent Legout <vincent.legout@gandi.net> wrote:
> > Hello,
> >
> > We had some issues after live migrating several domU from xen 4.1 to xen
> > 4.8. We migrated around 200 domU and 5 crashed, from a few hours up to
> > several days after the migration. All the domU had more than 1 year of
> > uptime, and for example one crashed several days after the migration
> > during a high load period.
> >
> > All 5 domU are running a 3.10 kernel (from 3.10.44 to 3.10.103). They
> > have between 2GB and 16GB of RAM, and between 1 and 4 vCPUS.
> >
> > We use 3 types of machines (several PowerEdge C6100 (Intel L5640) and
> > R710 (Intel L5520), and one C8220 (Intel E5-2650)). The C6100 and R710
> > have 24 logical cores with HT enabled. The PowerEdge C8220 is only used
> > for Xen 4.8, and has 32 logical cores with HT enabled. The C6100 and the
> > R710 have 50GB of RAM, and the C8220 128GB. The xen 4.1 dom0 is running
> > a 3.4.69 kernel, and the xen 4.8 one a 4.1.37 kernel.
> >
> > I've attached the most relevant parts of the domU kernel logs we could
> > get. It seems the crashes came from different components of the kernel,
> > though most of them seem to be related to memory.
> >
> > Would anyone have any idea if that's something that could be fixed? Or
> > is it just that migrating from 4.1 to 4.8 is not supported?
>
> Officially the open-source project only supports migrations between
> single releases -- i.e., 4.1 -> 4.2 or 4.7 -> 4.8. This is mainly
> because we haven't had the bandwidth to test more than that. I know
> XenServer tests and supports a broader window for migrations between
> their release versions; other downstreams (Oracle, SLES) may as well;
> but that probably won't help you in your current situation.
Thanks. Yes, we know it's not officially supported, but we wanted to see
if that was something we could consider.
Vincent
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: several domU crashes after 4.1->4.8 live migration
2017-02-02 12:05 ` Andrew Cooper
@ 2017-02-02 13:30 ` Vincent Legout
2017-02-02 14:07 ` Andrew Cooper
2017-02-02 15:40 ` George Dunlap
0 siblings, 2 replies; 8+ messages in thread
From: Vincent Legout @ 2017-02-02 13:30 UTC (permalink / raw)
To: Andrew Cooper; +Cc: xen-devel
On Thu, Feb 02, 2017 at 12:05:09PM +0000, Andrew Cooper wrote :
> On 02/02/17 07:58, Vincent Legout wrote:
> > Hello,
> >
> > We had some issues after live migrating several domU from xen 4.1 to xen
> > 4.8. We migrated around 200 domU and 5 crashed, from a few hours up to
> > several days after the migration. All the domU had more than 1 year of
> > uptime, and for example one crashed several days after the migration
> > during a high load period.
> >
> > All 5 domU are running a 3.10 kernel (from 3.10.44 to 3.10.103). They
> > have between 2GB and 16GB of RAM, and between 1 and 4 vCPUS.
> >
> > We use 3 types of machines (several PowerEdge C6100 (Intel L5640) and
> > R710 (Intel L5520), and one C8220 (Intel E5-2650)). The C6100 and R710
> > have 24 logical cores with HT enabled. The PowerEdge C8220 is only used
> > for Xen 4.8, and has 32 logical cores with HT enabled. The C6100 and the
> > R710 have 50GB of RAM, and the C8220 128GB. The xen 4.1 dom0 is running
> > a 3.4.69 kernel, and the xen 4.8 one a 4.1.37 kernel.
> >
> > I've attached the most relevant parts of the domU kernel logs we could
> > get. It seems the crashes came from different components of the kernel,
> > though most of them seem to be related to memory.
> >
> > Would anyone have any idea if that's something that could be fixed? Or
> > is it just that migrating from 4.1 to 4.8 is not supported?
>
> Do the VMs migrate normally in production?
Thanks for the comment. Yes, the migrations took place normally, at
least we didn't see anything wrong then. These crashes happened randomly
on a few VMs only, and at least a few hours after the migration.
> This looks like a Linux kernel bug in the Xen suspend/resume paths, and
> unlikely to be related to the version of the hypervisor in use.
I agree about the Linux kernel bug. But I still think it should also be
related to the migration because we never had anything like that without
migration, on either xen 4.1 or 4.8.
Vincent
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: several domU crashes after 4.1->4.8 live migration
2017-02-02 13:30 ` Vincent Legout
@ 2017-02-02 14:07 ` Andrew Cooper
2017-02-02 15:40 ` George Dunlap
1 sibling, 0 replies; 8+ messages in thread
From: Andrew Cooper @ 2017-02-02 14:07 UTC (permalink / raw)
To: Vincent Legout; +Cc: xen-devel
On 02/02/17 13:30, Vincent Legout wrote:
> On Thu, Feb 02, 2017 at 12:05:09PM +0000, Andrew Cooper wrote :
>> On 02/02/17 07:58, Vincent Legout wrote:
>>> Hello,
>>>
>>> We had some issues after live migrating several domU from xen 4.1 to xen
>>> 4.8. We migrated around 200 domU and 5 crashed, from a few hours up to
>>> several days after the migration. All the domU had more than 1 year of
>>> uptime, and for example one crashed several days after the migration
>>> during a high load period.
>>>
>>> All 5 domU are running a 3.10 kernel (from 3.10.44 to 3.10.103). They
>>> have between 2GB and 16GB of RAM, and between 1 and 4 vCPUS.
>>>
>>> We use 3 types of machines (several PowerEdge C6100 (Intel L5640) and
>>> R710 (Intel L5520), and one C8220 (Intel E5-2650)). The C6100 and R710
>>> have 24 logical cores with HT enabled. The PowerEdge C8220 is only used
>>> for Xen 4.8, and has 32 logical cores with HT enabled. The C6100 and the
>>> R710 have 50GB of RAM, and the C8220 128GB. The xen 4.1 dom0 is running
>>> a 3.4.69 kernel, and the xen 4.8 one a 4.1.37 kernel.
>>>
>>> I've attached the most relevant parts of the domU kernel logs we could
>>> get. It seems the crashes came from different components of the kernel,
>>> though most of them seem to be related to memory.
>>>
>>> Would anyone have any idea if that's something that could be fixed? Or
>>> is it just that migrating from 4.1 to 4.8 is not supported?
>> Do the VMs migrate normally in production?
> Thanks for the comment. Yes, the migrations took place normally, at
> least we didn't see anything wrong then. These crashes happened randomly
> on a few VMs only, and at least a few hours after the migration.
>
>> This looks like a Linux kernel bug in the Xen suspend/resume paths, and
>> unlikely to be related to the version of the hypervisor in use.
> I agree about the Linux kernel bug. But I still think it should also be
> related to the migration because we never had anything like that without
> migration, on either xen 4.1 or 4.8.
Ok, so it does look like a change in behaviour between 4.1 and 4.8.
Have you observed any further crashes from migrations on 4.8 after the
upgrade?
I am not aware of any alterations to the hypervisor side of things which
would be relevant.
The toolstack however has changed quite a lot.
The first change is with the migration stream itself. A migration like
that will be piped through tools/python/scripts/convert-legacy-stream to
convert from the old format to the new. It is certainly possible that
there is a bug in that process, although we test it extensively in
XenServer and have never encountered a crash looking like this in any
vintage of PV guest (all the way back to the RHEL 4 days). Also, the
fact that it didn't abort midwaythrough is a good sign that nothing
unexpected was encountered during the conversion process.
Another area which has changed is the semantics of how the toolstack
returns from the suspend call. There have been various changes to
support fast resume, all of which revolve around modifying the return
value from the hypercall as observed by the guest.
~Andrew
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: several domU crashes after 4.1->4.8 live migration
2017-02-02 13:30 ` Vincent Legout
2017-02-02 14:07 ` Andrew Cooper
@ 2017-02-02 15:40 ` George Dunlap
2017-02-03 9:43 ` Fatih Acar
1 sibling, 1 reply; 8+ messages in thread
From: George Dunlap @ 2017-02-02 15:40 UTC (permalink / raw)
To: Vincent Legout; +Cc: Andrew Cooper, xen-devel
On Thu, Feb 2, 2017 at 1:30 PM, Vincent Legout <vincent.legout@gandi.net> wrote:
> On Thu, Feb 02, 2017 at 12:05:09PM +0000, Andrew Cooper wrote :
>> On 02/02/17 07:58, Vincent Legout wrote:
>> > Hello,
>> >
>> > We had some issues after live migrating several domU from xen 4.1 to xen
>> > 4.8. We migrated around 200 domU and 5 crashed, from a few hours up to
>> > several days after the migration. All the domU had more than 1 year of
>> > uptime, and for example one crashed several days after the migration
>> > during a high load period.
>> >
>> > All 5 domU are running a 3.10 kernel (from 3.10.44 to 3.10.103). They
>> > have between 2GB and 16GB of RAM, and between 1 and 4 vCPUS.
>> >
>> > We use 3 types of machines (several PowerEdge C6100 (Intel L5640) and
>> > R710 (Intel L5520), and one C8220 (Intel E5-2650)). The C6100 and R710
>> > have 24 logical cores with HT enabled. The PowerEdge C8220 is only used
>> > for Xen 4.8, and has 32 logical cores with HT enabled. The C6100 and the
>> > R710 have 50GB of RAM, and the C8220 128GB. The xen 4.1 dom0 is running
>> > a 3.4.69 kernel, and the xen 4.8 one a 4.1.37 kernel.
>> >
>> > I've attached the most relevant parts of the domU kernel logs we could
>> > get. It seems the crashes came from different components of the kernel,
>> > though most of them seem to be related to memory.
>> >
>> > Would anyone have any idea if that's something that could be fixed? Or
>> > is it just that migrating from 4.1 to 4.8 is not supported?
>>
>> Do the VMs migrate normally in production?
>
> Thanks for the comment. Yes, the migrations took place normally, at
> least we didn't see anything wrong then. These crashes happened randomly
> on a few VMs only, and at least a few hours after the migration.
Sorry, I just want to clarify: Is it the case that before you tried to
upgrade, you often migrated VMs between your Xen 4.1 hosts without
problems?
In other words, is it *4.8* that's the problem, or is it *migration*
that's the problem?
-George
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: several domU crashes after 4.1->4.8 live migration
2017-02-02 15:40 ` George Dunlap
@ 2017-02-03 9:43 ` Fatih Acar
0 siblings, 0 replies; 8+ messages in thread
From: Fatih Acar @ 2017-02-03 9:43 UTC (permalink / raw)
To: George Dunlap; +Cc: Andrew Cooper, Vincent Legout, xen-devel
On 02/02/2017 16:40, George Dunlap wrote:
>
> Sorry, I just want to clarify: Is it the case that before you tried to
> upgrade, you often migrated VMs between your Xen 4.1 hosts without
> problems?
>
> In other words, is it *4.8* that's the problem, or is it *migration*
> that's the problem?
>
> -George
>
We have been using live migration on Xen 4.1 for several years now
without encountering any problem like this.
Domains work fine after a complete restart on Xen 4.8 although we
couldn't see how 4.8 to 4.8 live migration goes on long term yet, so the
problem is much likely to be the 4.1 to 4.8 migration.
--
Fatih ACAR
Gandi
fatih.acar@gandi.net
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2017-02-03 9:43 UTC | newest]
Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2017-02-02 7:58 several domU crashes after 4.1->4.8 live migration Vincent Legout
2017-02-02 11:22 ` George Dunlap
2017-02-02 12:57 ` Vincent Legout
2017-02-02 12:05 ` Andrew Cooper
2017-02-02 13:30 ` Vincent Legout
2017-02-02 14:07 ` Andrew Cooper
2017-02-02 15:40 ` George Dunlap
2017-02-03 9:43 ` Fatih Acar
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.