All of lore.kernel.org
 help / color / mirror / Atom feed
* KVM induced panic on 2.6.38[2367] & 2.6.39
@ 2011-05-31  1:24 Brad Campbell
  2011-05-31  5:47   ` Borislav Petkov
  0 siblings, 1 reply; 111+ messages in thread
From: Brad Campbell @ 2011-05-31  1:24 UTC (permalink / raw)
  To: linux-kernel; +Cc: kvm

G'day all,

I'm running a pretty standard home server
x86_64 Phenom-II 6 Core
16GB DDR 3

I run some virtual machines under that. 3 x Debian 64 Bit, 1 x XP 32 
Bit. These run at boot.

When I fire up another XP 32 bit instance and play with it for more than 
about 2 minutes, I get the panics included in this mail.

I've included three of them here. The first and third are as booted. The 
second was with ksmd disabled just as a data point.

The machine passes every load test and memory test I can throw at it, 
but I still can't rule out this being a hardware issue.

Provided I don't start this XP VM the machine is quite stable, but 
running this VM will kill it within minutes.

This was tested with qemu-kvm.
The last commit in the git tree was
commit c007db193eb6b2557acb5caf2dc4d7023639e6f3
Author: Avi Kivity <avi@redhat.com>
Date:   Sun May 29 09:00:42 2011 -0400
(I pulled it yesterday)

These panics were captured with netconsole to a remote syslog daemon, 
and the formatting was ruined, so I've reformatted them by hand prior to 
posting.

I've tested and reproduced this on 2.6.38.[2,3,6 & 7] and 2.6.39 obviously.

Can anyone help shed some light on this?

Regards,
Brad

[ 438.632061] general protection fault: 0000 [#1] SMP
[ 438.632196] last sysfs file: /sys/module/x_tables/initstate
[ 438.632242] CPU 4
[ 438.632282] Modules linked in: xt_iprange xt_DSCP xt_length 
xt_CLASSIFY sch_sfq xt_CHECKSUM ipt_REJECT ipt_MASQUERADE ipt_REDIRECT 
xt_recent xt_state iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack 
nf_defrag_ipv4 iptable_filter xt_TCPMSS xt_tcpmss xt_tcpudp 
iptable_mangle ip_tables x_tables pppoe pppox ppp_generic slhc cls_u32 
sch_htb deflate zlib_deflate des_generic cbc ecb crypto_blkcipher 
sha1_generic md5 hmac crypto_hash cryptomgr aead crypto_algapi af_key 
fuse netconsole configfs= vhost_net powernow_k8 mperf i2c_nforce2 
kvm_amd kvm pl2303 usbserial xhci_hcd k10temp i2c_piix4 usb_storage 
usb_libusual ohci_hcd ehci_hcd usbcore ahci libahci r8169 mii sata_mv 
megaraid_sas [last unloaded: scsi_wait_scan]
[ 438.634960]
[ 438.635006] Pid: 551, comm: ksmd Not tainted 2.6.39 #3 To Be Filled By 
O.E.M. To Be Filled By O.E.M. /880G Extreme3
[ 438.635170] RIP: 0010:[<ffffffff810b4596>] [<ffffffff810b4596>] 
remove_rmap_item_from_tree+0x96/0x150
[ 438.635268] RSP: 0018:ffff88041c065e20  EFLAGS: 00010282
[ 438.635314] RAX: ffff8804153bd8b0 RBX: ffff8804176c3fc0 RCX: 
0000000000057754
[ 438.635362] RDX: 0000880415418030 RSI: ffff880414b65003 RDI: 
ffffea000dde6030
[ 438.635410] RBP: ffff880414b65000 R08: 0000000000057755 R09: 
000000001bbde1bd
[ 438.635458] R10: 000000001c28e0c3 R11: 0000000000000002 R12: 
ffffea000dde6030
[ 438.635506] R13: ffff8804176c3f80 R14: ffff8804151ed7b0 R15: 
ffff88041bf23be0
[ 438.635555] FS:  00007f617772e700(0000) GS:ffff88041fd00000(0000) 
knlGS:0000000000000000
[ 438.635607] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 438.635654] CR2: 0000000000e70000 CR3: 0000000001583000 CR4: 
00000000000006e0
[ 438.635703] DR0: 0000000000000045 DR1: 0000000000000000 DR2: 
0000000000000000
[ 438.635750] DR3: 0000000000000005 DR6: 00000000ffff0ff0 DR7: 
0000000000000400
[ 438.635799] Process ksmd (pid: 551, threadinfo ffff88041c064000, task 
ffff88041d8caa70)
[ 438.635851] Stack:
[ 438.635893]  ffffea000c43c408 ffff8804176c3fc0 000000000000036c 
ffffffff810b58f2
[ 438.636088]  ffff88041c782a00 00007fc2d81b5000 ffff88041c064000 
00000000003808ff
[ 438.636281]  ffff88041c064000 ffff88041c065e98 ffff88041d8caa70 
ffff8804165d4480
[ 438.636479] Call Trace:
[ 438.636528]  [<ffffffff810b58f2>] ? ksm_scan_thread+0x4e2/0xc20
[ 438.636580]  [<ffffffff81052a20>] ? wake_up_bit+0x40/0x40
[ 438.636628]  [<ffffffff810b5410>] ? try_to_merge_with_ksm_page+0x570/0x570
[ 438.636679]  [<ffffffff810b5410>] ? try_to_merge_with_ksm_page+0x570/0x570
[ 438.636730]  [<ffffffff810525b6>] ? kthread+0x96/0xa0
[ 438.636781]  [<ffffffff813d1794>] ? kernel_thread_helper+0x4/0x10
[ 438.636832]  [<ffffffff81052520>] ? kthread_worker_fn+0x120/0x120
[ 438.636882]  [<ffffffff813d1790>] ? gs_change+0xb/0xb
[ 438.636926] Code: 28 48 89 ef e8 6c fe ff ff 48 85 c0 49 89 c4 74 d2 
f0 0f ba 28 00 19 c0 85 c0 0f 85 ae 00 00 00 00 48 8b 43 30 48 8b 53 38 
48 85 c0
[ 438.638504] 89 02 74 04 48 89 50 08 48 ba 00 01 10 00 00 00 00 ad de 48 b8
[ 438.639329] RIP [<ffffffff810b4596>] remove_rmap_item_from_tree+0x96/0x150
[ 438.639414]  RSP <ffff88041c065e20>
[ 438.639460] ---[ end trace c29fb871f6b874e3 ]---
[ 438.639506] Kernel panic - not syncing: Fatal exception
[ 438.639553] Pid: 551, comm: ksmd Tainted: G      D     2.6.39 #3
[ 438.639598] Call Trace:
[ 438.639644]  [<ffffffff813cd6f5>] ? panic+0x92/0x18a
[ 438.639693]  [<ffffffff81038b61>] ? kmsg_dump+0x41/0xf0
[ 438.639745]  [<ffffffff810050ad>] ? oops_end+0x8d/0xa0
[ 438.639794]  [<ffffffff813d05ef>] ? general_protection+0x1f/0x30
[ 438.639842]  [<ffffffff810b4596>] ? remove_rmap_item_from_tree+0x96/0x150
[ 438.639891]  [<ffffffff810b58f2>] ? ksm_scan_thread+0x4e2/0xc20
[ 438.639941]  [<ffffffff81052a20>] ? wake_up_bit+0x40/0x40
[ 438.640045]  [<ffffffff810b5410>] ? try_to_merge_with_ksm_page+0x570/0x570
[ 438.640096]  [<ffffffff810b5410>] ? try_to_merge_with_ksm_page+0x570/0x570
[ 438.640147]  [<ffffffff810525b6>] ? kthread+0x96/0xa0
[ 438.640196]  [<ffffffff813d1794>] ? kernel_thread_helper+0x4/0x10
[ 438.640247]  [<ffffffff81052520>] ? kthread_worker_fn+0x120/0x120
[ 438.640297]  [<ffffffff813d1790>] ? gs_change+0xb/0xb
[ 438.640349] Rebooting in 60 seconds..



[ 469.706013] general protection fault: 0000 [#1] SMP
[ 469.706125] last sysfs file: 
/sys/devices/pci0000:00/0000:00:18.3/temp1_input
[ 469.706162] CPU 1
[ 469.706195] Modules linked in: xt_iprange xt_DSCP xt_length 
xt_CLASSIFY sch_sfq xt_CHECKSUM ipt_REJECT ipt_MASQUERADE ipt_REDIRECT 
xt_recent xt_state iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack 
nf_defrag_ipv4 iptable_filter xt_TCPMSS xt_tcpmss xt_tcpudp 
iptable_mangle ip_tables x_tables pppoe pppox ppp_generic slhc cls_u32 
sch_htb deflate zlib_deflate des_generi cbc ecb crypto_blkcipher 
sha1_generic md5 hmac crypto_hash cryptomgr aead crypto_algapi af_key 
fuse netconsole configfs vhost_net powernow_k8 mperf i2c_nforce2 kvm_amd 
kvm pl2303 usbserial xhci_hcd i2c_piix4 k10temp ahci usb_storage 
usb_libusual ohci_hcd ehci_hcd r8169 libahci usbcore mii sata_mv 
megaraid_sas [last unloaded: scsi_wait_scan]
[ 469.708573]
[ 469.708608] Pid: 4942, comm: runbot Not tainted 2.6.39 #3 To Be Filled 
By O.E.M. To Be Filled By O.E.M. /880G Extreme3
[ 469.708753] RIP: 0010:[<ffffffff810db878>] [<ffffffff810db878>] 
dup_fd+0x168/0x300
[ 469.708829] RSP: 0018:ffff880417279dd0  EFLAGS: 00010202
[ 469.708870] RAX: 00000000000007f8 RBX: ffff88041c1658c0 RCX: 
bfffffffffffffff
[ 469.708907] RDX: 0000880413155540 RSI: 00000000000000ff RDI: 
0000000000000800
[ 469.708945] RBP: ffff88041bf22000 R08: ffff880412839dc0 R09: 
0000000000000003
[ 469.708981] R10: 0000000000000001 R11: 4000000000000000 R12: 
0000000000000100
[ 469.709018] R13: ffff88041d3cc000 R14: ffff880412839d40 R15: 
ffff88041c72f440
[ 469.709054] FS:  00007f510c0b4700(0000) GS:ffff88041fc40000(0000) 
knlGS:0000000000000000
[ 469.709094] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 469.709130] CR2: 00007f510ba56810 CR3: 0000000417c93000 CR4: 
00000000000006e0
[ 469.709167] DR0: 00000000000000a0 DR1: 0000000000000000 DR2: 
0000000000000003
[ 469.709204] DR3: 00000000000000b0 DR6: 00000000ffff0ff0 DR7: 
0000000000000400
[ 469.709240] Process runbot (pid: 4942, threadinfo ffff880417278000, 
task ffff880401351230)
[ 469.709279] Stack:
[ 469.709313]  ffff880401351230 0000000000000020 ffff880412e9ca80 
ffff88041c1658d0 ffff88041c1658d0
[ 469.709477]  ffff88041723f330 0000000001200011 ffff88041723f330 
0000000000000000
[ 469.709641]  00007f510c0b49d0 ffff880401351230 0000000000000000 
ffffffff81037645
[ 469.709812] Call Trace:
[ 469.709849]  [<ffffffff81037645>] ? copy_process+0xa75/0xfd0
[ 469.709885]  [<ffffffff81037c0d>] ? do_fork+0x6d/0x2b0
[ 469.709921]  [<ffffffff810dbc23>] ? alloc_fd+0x43/0x130
[ 469.709958]  [<ffffffff810c08c5>] ? fd_install+0x35/0x70
[ 469.709998]  [<ffffffff810457a9>] ? sigprocmask+0x69/0x100
[ 469.710042]  [<ffffffff813d0ca3>] ? stub_clone+0x13/0x20
[ 469.710079]  [<ffffffff813d0a3b>] ? system_call_fastpath+0x16/0x1b
[ 469.710114] Code:  4c 89 c2 e8 6b e5 0f 00 45 85 e4 74 78 41 8d 44 24 
ff 31 f6 41 ba 01 00 00 00 00 48 8d 3c c5 08 00 00 00 00 31 c0 eb 1a 0f 
1f 44 00 00 f0> 48 ff 42 30 48 89 54 05 00 48 83 c0
May 30 16:39:36 192.168.2.1 08 ff c6 48 39 f8 74 3b
[ 469.712247] RIP [<ffffffff810db878>] dup_fd+0x168/0x300
[ 469.712317]  RSP <ffff880417279dd0>
[ 469.712391] ---[ end trace b80596608cd6bb6b ]---
[ 469.712432] Kernel panic - not syncing: Fatal exception
[ 469.712470] Pid: 4942, comm: runbot Tainted: G      D     2.6.39 #3
[ 469.712508] Call Trace:
[ 469.712544]  [<ffffffff813cd6f5>] ? panic+0x92/0x18a
[ 469.712581]  [<ffffffff81038b61>] ? kmsg_dump+0x41/0xf0
[ 469.712618]  [<ffffffff810050ad>] ? oops_end+0x8d/0xa0
[ 469.712654]  [<ffffffff813d05ef>] ? general_protection+0x1f/0x30
[ 469.712691]  [<ffffffff810db878>] ? dup_fd+0x168/0x300
[ 469.712727]  [<ffffffff81037645>] ? copy_process+0xa75/0xfd0
[ 469.712767]  [<ffffffff81037c0d>] ? do_fork+0x6d/0x2b0
[ 469.712803]  [<ffffffff810dbc23>] ? alloc_fd+0x43/0x130
[ 469.712839]  [<ffffffff810c08c5>] ? fd_install+0x35/0x70
[ 469.712875]  [<ffffffff810457a9>] ? sigprocmask+0x69/0x100
[ 469.712912]  [<ffffffff813d0ca3>] ? stub_clone+0x13/0x20
[ 469.712948]  [<ffffffff813d0a3b>] ? system_call_fastpath+0x16/0x1b
[ 469.712989] Rebooting in 60 seconds..


[ 1100.473612] general protection fault: 0000 [#1] SMP
[ 1100.473718] last sysfs file: 
/sys/devices/pci0000:00/0000:00:18.3/temp1_input
[ 1100.473755] CPU 0
[ 1100.473788] Modules linked in: xt_iprange xt_DSCP xt_length 
xt_CLASSIFY sch_sfq xt_CHECKSUM ipt_REJECT ipt_MASQUERADE ipt_REDIRECT 
xt_recent xt_state iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack 
nf_defrag_ipv4 xt_TCPMSS xt_tcpmss xt_tcpudp iptable_mangle pppoe pppox 
iptable_filter ip_tables x_tables ppp_generic slhc cls_u32 sch_htb 
deflate zlib_deflate des_generic cbc ecb crypto_blkcipher sha1_generic 
md5 hmac crypto_hash cryptomgr aead crypto_algapi af_key fuse netconsole 
configfs vhost_net powernow_k8 mperf i2c_nforce2 kvm_amd kvm pl2303 
xhci_hcd usbserial k10temp i2c_piix4 usb_storage usb_libusual ohci_hcd 
ehci_hcd ahci usbcore sata_mv r8169 mii libahci megaraid_sas [last 
unloaded: scsi_wait_scan]
[ 1100.476102]
[ 1100.476137] Pid: 551, comm: ksmd Not tainted 2.6.39 #3 To Be Filled 
By O.E.M. To Be Filled By O.E.M. /880G Extreme3
[ 1100.476277] RIP: 0010:[<ffffffff810b5c71>] [<ffffffff810b5c71>] 
ksm_scan_thread+0x861/0xc20
[ 1100.476351] RSP: 0018:ffff88041c065e40  EFLAGS: 00010282
[ 1100.476387] RAX: 0000000000000000 RBX: ffffea000a5c8468 RCX: 
0000000000000f01
[ 1100.476424] RDX: 0000880403456828 RSI: ffff8802f5dcb002 RDI: 
ffffea000ce7eac8
[ 1100.476460] RBP: ffff880418377fc0 R08: 6db6db6db6db6db7 R09: 
0000160000000000
[ 1100.476498] R10: ffffea000ce7eac8 R11: 0000000080100073 R12: 
00000000000000ab
[ 1100.476535] R13: ffff880418377ff8 R14: ffffea000ce7eac8 R15: 
ffff880418377fe8
[ 1100.476572] FS:  00007f08c6498700(0000) GS:ffff88041fc00000(0000) 
knlGS:0000000000000000
[ 1100.476611] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 1100.476646] CR2: 00000000e1dd3000 CR3: 00000004025f7000 CR4: 
00000000000006f0
[ 1100.477007] DR0: 0000000000000045 DR1: 0000000000000000 DR2: 
0000000000000000
[ 1100.477042] DR3: 0000000000000005 DR6: 00000000ffff0ff0 DR7: 
0000000000000400
[ 1100.477078] Process ksmd (pid: 551, threadinfo ffff88041c064000, task 
ffff88041d0454e0)
[ 1100.477115] Stack:
[ 1100.477148]  ffff880403a8b100 00007fc1863cb000 ffff88041c064000 
00000000002f5dcb
[ 1100.477317]  ffff88041c064000 ffff88041c065e98 ffff88041d0454e0 
ffff88041267a580
[ 1100.477480]  0000000000000000 ffff88041d0454e0 ffffffff81052a20 
ffff88041c065e98
[ 1100.477644] Call Trace:
[ 1100.477681]  [<ffffffff81052a20>] ? wake_up_bit+0x40/0x40
[ 1100.477717]  [<ffffffff810b5410>] ? 
try_to_merge_with_ksm_page+0x570/0x570
[ 1100.477754]  [<ffffffff810b5410>] ? 
try_to_merge_with_ksm_page+0x570/0x570
[ 1100.477791]  [<ffffffff810525b6>] ? kthread+0x96/0xa0
[ 1100.477829]  [<ffffffff813d1794>] ? kernel_thread_helper+0x4/0x10
[ 1100.477865]  [<ffffffff81052520>] ? kthread_worker_fn+0x120/0x120
[ 1100.477902]  [<ffffffff813d1790>] ? gs_change+0xb/0xb
[ 1100.477937] Code: 05 99 6e 6e 00 4c 89 ff e8 ed 12 fa ff e9 6c f9 ff 
ff 48 89 6c 24 38 49 83 7d 00 00 0f 84 63 01 00 00 e8 b3 8b 31 00 49 8b 
55 00
[ 1100.479306] 8b 42 e8 4c 8b 72 f0 48 8d 6a d8 4c 8d 78 60 48 89 04 24 4c
[ 1100.480012] RIP [<ffffffff810b5c71>] ksm_scan_thread+0x861/0xc20
[ 1100.480015]  RSP <ffff88041c065e40>
[ 1100.480141] ---[ end trace 7382777d9e74ea23 ]---
[ 1100.480178] Kernel panic - not syncing: Fatal exception
[ 1100.480216] Pid: 551, comm: ksmd Tainted: G      D     2.6.39 #3
[ 1100.480257] Call Trace:
[ 1100.480297]  [<ffffffff813cd6f5>] ? panic+0x92/0x18a
[ 1100.480340]  [<ffffffff81038b61>] ? kmsg_dump+0x41/0xf0
[ 1100.480384]  [<ffffffff810050ad>] ? oops_end+0x8d/0xa0
[ 1100.480425]  [<ffffffff813d05ef>] ? general_protection+0x1f/0x30
[ 1100.480465]  [<ffffffff810b5c71>] ? ksm_scan_thread+0x861/0xc20
[ 1100.480507]  [<ffffffff81052a20>] ? wake_up_bit+0x40/0x40
[ 1100.480545]  [<ffffffff810b5410>] ? 
try_to_merge_with_ksm_page+0x570/0x570
[ 1100.480584]  [<ffffffff810b5410>] ? 
try_to_merge_with_ksm_page+0x570/0x570
[ 1100.480624]  [<ffffffff810525b6>] ? kthread+0x96/0xa0
[ 1100.480669]  [<ffffffff813d1794>] ? kernel_thread_helper+0x4/0x10
[ 1100.480708]  [<ffffffff81052520>] ? kthread_worker_fn+0x120/0x120
[ 1100.480748]  [<ffffffff813d1790>] ? gs_change+0xb/0xb
[ 1100.480794] Rebooting in 60 seconds..

brad@srv:~$ zcat /proc/config.gz | grep -v "#"
CONFIG_64BIT=y
CONFIG_X86_64=y
CONFIG_X86=y
CONFIG_INSTRUCTION_DECODER=y
CONFIG_OUTPUT_FORMAT="elf64-x86-64"
CONFIG_ARCH_DEFCONFIG="arch/x86/configs/x86_64_defconfig"
CONFIG_GENERIC_CMOS_UPDATE=y
CONFIG_CLOCKSOURCE_WATCHDOG=y
CONFIG_GENERIC_CLOCKEVENTS=y
CONFIG_GENERIC_CLOCKEVENTS_BROADCAST=y
CONFIG_LOCKDEP_SUPPORT=y
CONFIG_STACKTRACE_SUPPORT=y
CONFIG_HAVE_LATENCYTOP_SUPPORT=y
CONFIG_MMU=y
CONFIG_ZONE_DMA=y
CONFIG_NEED_DMA_MAP_STATE=y
CONFIG_NEED_SG_DMA_LENGTH=y
CONFIG_GENERIC_ISA_DMA=y
CONFIG_GENERIC_IOMAP=y
CONFIG_GENERIC_BUG=y
CONFIG_GENERIC_BUG_RELATIVE_POINTERS=y
CONFIG_GENERIC_HWEIGHT=y
CONFIG_ARCH_MAY_HAVE_PC_FDC=y
CONFIG_RWSEM_XCHGADD_ALGORITHM=y
CONFIG_ARCH_HAS_CPU_IDLE_WAIT=y
CONFIG_GENERIC_CALIBRATE_DELAY=y
CONFIG_GENERIC_TIME_VSYSCALL=y
CONFIG_ARCH_HAS_CPU_RELAX=y
CONFIG_ARCH_HAS_DEFAULT_IDLE=y
CONFIG_ARCH_HAS_CACHE_LINE_SIZE=y
CONFIG_HAVE_SETUP_PER_CPU_AREA=y
CONFIG_NEED_PER_CPU_EMBED_FIRST_CHUNK=y
CONFIG_NEED_PER_CPU_PAGE_FIRST_CHUNK=y
CONFIG_HAVE_CPUMASK_OF_CPU_MAP=y
CONFIG_ARCH_HIBERNATION_POSSIBLE=y
CONFIG_ARCH_SUSPEND_POSSIBLE=y
CONFIG_ZONE_DMA32=y
CONFIG_ARCH_POPULATES_NODE_MAP=y
CONFIG_AUDIT_ARCH=y
CONFIG_ARCH_SUPPORTS_OPTIMIZED_INLINING=y
CONFIG_ARCH_SUPPORTS_DEBUG_PAGEALLOC=y
CONFIG_X86_64_SMP=y
CONFIG_X86_HT=y
CONFIG_ARCH_HWEIGHT_CFLAGS="-fcall-saved-rdi -fcall-saved-rsi 
-fcall-saved-rdx -fcall-saved-rcx -fcall-saved-r8 -fcall-saved-r9 
-fcall-saved-r10 -fcall-saved-r11"
CONFIG_DEFCONFIG_LIST="/lib/modules/$UNAME_RELEASE/.config"
CONFIG_CONSTRUCTORS=y
CONFIG_HAVE_IRQ_WORK=y
CONFIG_IRQ_WORK=y

CONFIG_EXPERIMENTAL=y
CONFIG_INIT_ENV_ARG_LIMIT=32
CONFIG_CROSS_COMPILE=""
CONFIG_LOCALVERSION=""
CONFIG_LOCALVERSION_AUTO=y
CONFIG_HAVE_KERNEL_GZIP=y
CONFIG_HAVE_KERNEL_BZIP2=y
CONFIG_HAVE_KERNEL_LZMA=y
CONFIG_HAVE_KERNEL_XZ=y
CONFIG_HAVE_KERNEL_LZO=y
CONFIG_KERNEL_GZIP=y
CONFIG_SWAP=y
CONFIG_SYSVIPC=y
CONFIG_SYSVIPC_SYSCTL=y
CONFIG_BSD_PROCESS_ACCT=y
CONFIG_TASKSTATS=y
CONFIG_TASK_DELAY_ACCT=y
CONFIG_TASK_XACCT=y
CONFIG_TASK_IO_ACCOUNTING=y
CONFIG_HAVE_GENERIC_HARDIRQS=y

CONFIG_GENERIC_HARDIRQS=y
CONFIG_HAVE_SPARSE_IRQ=y
CONFIG_GENERIC_IRQ_PROBE=y
CONFIG_GENERIC_IRQ_SHOW=y
CONFIG_GENERIC_PENDING_IRQ=y
CONFIG_IRQ_FORCED_THREADING=y

CONFIG_TREE_RCU=y
CONFIG_RCU_FANOUT=64
CONFIG_IKCONFIG=y
CONFIG_IKCONFIG_PROC=y
CONFIG_LOG_BUF_SHIFT=20
CONFIG_HAVE_UNSTABLE_SCHED_CLOCK=y
CONFIG_NAMESPACES=y
CONFIG_BLK_DEV_INITRD=y
CONFIG_INITRAMFS_SOURCE=""
CONFIG_RD_GZIP=y
CONFIG_RD_BZIP2=y
CONFIG_RD_LZMA=y
CONFIG_RD_XZ=y
CONFIG_RD_LZO=y
CONFIG_SYSCTL=y
CONFIG_ANON_INODES=y
CONFIG_UID16=y
CONFIG_SYSCTL_SYSCALL=y
CONFIG_KALLSYMS=y
CONFIG_KALLSYMS_ALL=y
CONFIG_KALLSYMS_EXTRA_PASS=y
CONFIG_HOTPLUG=y
CONFIG_PRINTK=y
CONFIG_BUG=y
CONFIG_ELF_CORE=y
CONFIG_PCSPKR_PLATFORM=y
CONFIG_BASE_FULL=y
CONFIG_FUTEX=y
CONFIG_EPOLL=y
CONFIG_SIGNALFD=y
CONFIG_TIMERFD=y
CONFIG_EVENTFD=y
CONFIG_SHMEM=y
CONFIG_AIO=y
CONFIG_HAVE_PERF_EVENTS=y

CONFIG_PERF_EVENTS=y
CONFIG_VM_EVENT_COUNTERS=y
CONFIG_PCI_QUIRKS=y
CONFIG_SLUB_DEBUG=y
CONFIG_SLUB=y
CONFIG_HAVE_OPROFILE=y
CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS=y
CONFIG_USER_RETURN_NOTIFIER=y
CONFIG_HAVE_IOREMAP_PROT=y
CONFIG_HAVE_KPROBES=y
CONFIG_HAVE_KRETPROBES=y
CONFIG_HAVE_OPTPROBES=y
CONFIG_HAVE_ARCH_TRACEHOOK=y
CONFIG_HAVE_DMA_ATTRS=y
CONFIG_USE_GENERIC_SMP_HELPERS=y
CONFIG_HAVE_REGS_AND_STACK_ACCESS_API=y
CONFIG_HAVE_DMA_API_DEBUG=y
CONFIG_HAVE_HW_BREAKPOINT=y
CONFIG_HAVE_MIXED_BREAKPOINTS_REGS=y
CONFIG_HAVE_USER_RETURN_NOTIFIER=y
CONFIG_HAVE_PERF_EVENTS_NMI=y
CONFIG_HAVE_ARCH_JUMP_LABEL=y

CONFIG_SLABINFO=y
CONFIG_RT_MUTEXES=y
CONFIG_BASE_SMALL=0
CONFIG_MODULES=y
CONFIG_MODULE_UNLOAD=y
CONFIG_STOP_MACHINE=y
CONFIG_BLOCK=y
CONFIG_BLK_DEV_BSG=y
CONFIG_BLOCK_COMPAT=y

CONFIG_IOSCHED_NOOP=y
CONFIG_IOSCHED_DEADLINE=y
CONFIG_IOSCHED_CFQ=y
CONFIG_DEFAULT_CFQ=y
CONFIG_DEFAULT_IOSCHED="cfq"
CONFIG_PREEMPT_NOTIFIERS=y
CONFIG_INLINE_SPIN_UNLOCK=y
CONFIG_INLINE_SPIN_UNLOCK_IRQ=y
CONFIG_INLINE_READ_UNLOCK=y
CONFIG_INLINE_READ_UNLOCK_IRQ=y
CONFIG_INLINE_WRITE_UNLOCK=y
CONFIG_INLINE_WRITE_UNLOCK_IRQ=y
CONFIG_MUTEX_SPIN_ON_OWNER=y

CONFIG_TICK_ONESHOT=y
CONFIG_NO_HZ=y
CONFIG_HIGH_RES_TIMERS=y
CONFIG_GENERIC_CLOCKEVENTS_BUILD=y
CONFIG_SMP=y
CONFIG_X86_SUPPORTS_MEMORY_FAILURE=y
CONFIG_SCHED_OMIT_FRAME_POINTER=y
CONFIG_NO_BOOTMEM=y
CONFIG_MK8=y
CONFIG_X86_INTERNODE_CACHE_SHIFT=6
CONFIG_X86_CMPXCHG=y
CONFIG_CMPXCHG_LOCAL=y
CONFIG_X86_L1_CACHE_SHIFT=6
CONFIG_X86_XADD=y
CONFIG_X86_WP_WORKS_OK=y
CONFIG_X86_INTEL_USERCOPY=y
CONFIG_X86_USE_PPRO_CHECKSUM=y
CONFIG_X86_TSC=y
CONFIG_X86_CMPXCHG64=y
CONFIG_X86_CMOV=y
CONFIG_X86_MINIMUM_CPU_FAMILY=64
CONFIG_X86_DEBUGCTLMSR=y
CONFIG_CPU_SUP_INTEL=y
CONFIG_CPU_SUP_AMD=y
CONFIG_CPU_SUP_CENTAUR=y
CONFIG_HPET_TIMER=y
CONFIG_HPET_EMULATE_RTC=y
CONFIG_DMI=y
CONFIG_GART_IOMMU=y
CONFIG_AMD_IOMMU=y
CONFIG_SWIOTLB=y
CONFIG_IOMMU_HELPER=y
CONFIG_IOMMU_API=y
CONFIG_NR_CPUS=8
CONFIG_SCHED_MC=y
CONFIG_PREEMPT_NONE=y
CONFIG_X86_LOCAL_APIC=y
CONFIG_X86_IO_APIC=y
CONFIG_X86_MCE=y
CONFIG_X86_MCE_AMD=y
CONFIG_X86_MCE_THRESHOLD=y
CONFIG_ARCH_PHYS_ADDR_T_64BIT=y
CONFIG_ARCH_DMA_ADDR_T_64BIT=y
CONFIG_DIRECT_GBPAGES=y
CONFIG_ARCH_SPARSEMEM_DEFAULT=y
CONFIG_ARCH_SPARSEMEM_ENABLE=y
CONFIG_ARCH_SELECT_MEMORY_MODEL=y
CONFIG_ARCH_MEMORY_PROBE=y
CONFIG_ILLEGAL_POINTER_VALUE=0xdead000000000000
CONFIG_SELECT_MEMORY_MODEL=y
CONFIG_SPARSEMEM_MANUAL=y
CONFIG_SPARSEMEM=y
CONFIG_HAVE_MEMORY_PRESENT=y
CONFIG_SPARSEMEM_EXTREME=y
CONFIG_SPARSEMEM_VMEMMAP_ENABLE=y
CONFIG_SPARSEMEM_ALLOC_MEM_MAP_TOGETHER=y
CONFIG_SPARSEMEM_VMEMMAP=y
CONFIG_HAVE_MEMBLOCK=y
CONFIG_MEMORY_HOTPLUG=y
CONFIG_MEMORY_HOTPLUG_SPARSE=y
CONFIG_PAGEFLAGS_EXTENDED=y
CONFIG_SPLIT_PTLOCK_CPUS=4
CONFIG_COMPACTION=y
CONFIG_MIGRATION=y
CONFIG_PHYS_ADDR_T_64BIT=y
CONFIG_ZONE_DMA_FLAG=1
CONFIG_BOUNCE=y
CONFIG_VIRT_TO_BUS=y
CONFIG_MMU_NOTIFIER=y
CONFIG_KSM=y
CONFIG_DEFAULT_MMAP_MIN_ADDR=4096
CONFIG_ARCH_SUPPORTS_MEMORY_FAILURE=y
CONFIG_TRANSPARENT_HUGEPAGE=y
CONFIG_TRANSPARENT_HUGEPAGE_ALWAYS=y
CONFIG_X86_RESERVE_LOW=64
CONFIG_MTRR=y
CONFIG_MTRR_SANITIZER=y
CONFIG_MTRR_SANITIZER_ENABLE_DEFAULT=0
CONFIG_MTRR_SANITIZER_SPARE_REG_NR_DEFAULT=1
CONFIG_X86_PAT=y
CONFIG_ARCH_USES_PG_UNCACHED=y
CONFIG_SECCOMP=y
CONFIG_HZ_100=y
CONFIG_HZ=100
CONFIG_SCHED_HRTICK=y
CONFIG_PHYSICAL_START=0x1000000
CONFIG_PHYSICAL_ALIGN=0x1000000
CONFIG_ARCH_ENABLE_MEMORY_HOTPLUG=y
CONFIG_ARCH_ENABLE_MEMORY_HOTREMOVE=y

CONFIG_ACPI=y
CONFIG_ACPI_BUTTON=y
CONFIG_ACPI_DOCK=y
CONFIG_ACPI_PROCESSOR=y
CONFIG_ACPI_THERMAL=y
CONFIG_ACPI_BLACKLIST_YEAR=0
CONFIG_X86_PM_TIMER=y

CONFIG_CPU_FREQ=y
CONFIG_CPU_FREQ_TABLE=y
CONFIG_CPU_FREQ_STAT=y
CONFIG_CPU_FREQ_STAT_DETAILS=y
CONFIG_CPU_FREQ_DEFAULT_GOV_ONDEMAND=y
CONFIG_CPU_FREQ_GOV_PERFORMANCE=y
CONFIG_CPU_FREQ_GOV_POWERSAVE=y
CONFIG_CPU_FREQ_GOV_USERSPACE=y
CONFIG_CPU_FREQ_GOV_ONDEMAND=y
CONFIG_CPU_FREQ_GOV_CONSERVATIVE=y

CONFIG_X86_ACPI_CPUFREQ=m
CONFIG_X86_POWERNOW_K8=m

CONFIG_CPU_IDLE=y
CONFIG_CPU_IDLE_GOV_LADDER=y
CONFIG_CPU_IDLE_GOV_MENU=y


CONFIG_PCI=y
CONFIG_PCI_DIRECT=y
CONFIG_PCI_MMCONFIG=y
CONFIG_PCI_DOMAINS=y
CONFIG_PCIEPORTBUS=y
CONFIG_PCIEAER=y
CONFIG_PCIEASPM=y
CONFIG_ARCH_SUPPORTS_MSI=y
CONFIG_PCI_MSI=y
CONFIG_HT_IRQ=y
CONFIG_PCI_IOAPIC=y
CONFIG_PCI_LABEL=y
CONFIG_ISA_DMA_API=y
CONFIG_AMD_NB=y

CONFIG_BINFMT_ELF=y
CONFIG_COMPAT_BINFMT_ELF=y
CONFIG_BINFMT_MISC=m
CONFIG_IA32_EMULATION=y
CONFIG_IA32_AOUT=m
CONFIG_COMPAT=y
CONFIG_COMPAT_FOR_U64_ALIGNMENT=y
CONFIG_SYSVIPC_COMPAT=y
CONFIG_HAVE_TEXT_POKE_SMP=y
CONFIG_NET=y

CONFIG_PACKET=y
CONFIG_UNIX=y
CONFIG_XFRM=y
CONFIG_XFRM_MIGRATE=y
CONFIG_XFRM_IPCOMP=m
CONFIG_NET_KEY=m
CONFIG_NET_KEY_MIGRATE=y
CONFIG_INET=y
CONFIG_IP_MULTICAST=y
CONFIG_IP_ROUTE_CLASSID=y
CONFIG_NET_IPGRE_DEMUX=m
CONFIG_INET_AH=m
CONFIG_INET_ESP=m
CONFIG_INET_IPCOMP=m
CONFIG_INET_XFRM_TUNNEL=m
CONFIG_INET_TUNNEL=m
CONFIG_INET_XFRM_MODE_TRANSPORT=m
CONFIG_INET_XFRM_MODE_TUNNEL=m
CONFIG_INET_XFRM_MODE_BEET=m
CONFIG_INET_DIAG=y
CONFIG_INET_TCP_DIAG=y
CONFIG_TCP_CONG_CUBIC=y
CONFIG_DEFAULT_TCP_CONG="cubic"
CONFIG_NETFILTER=y
CONFIG_NETFILTER_ADVANCED=y
CONFIG_BRIDGE_NETFILTER=y

CONFIG_NETFILTER_NETLINK=m
CONFIG_NETFILTER_NETLINK_QUEUE=m
CONFIG_NETFILTER_NETLINK_LOG=m
CONFIG_NF_CONNTRACK=m
CONFIG_NF_CONNTRACK_MARK=y
CONFIG_NF_CONNTRACK_ZONES=y
CONFIG_NF_CT_PROTO_DCCP=m
CONFIG_NF_CT_PROTO_GRE=m
CONFIG_NF_CT_PROTO_SCTP=m
CONFIG_NF_CT_PROTO_UDPLITE=m
CONFIG_NF_CONNTRACK_AMANDA=m
CONFIG_NF_CONNTRACK_FTP=m
CONFIG_NF_CONNTRACK_H323=m
CONFIG_NF_CONNTRACK_IRC=m
CONFIG_NF_CONNTRACK_BROADCAST=m
CONFIG_NF_CONNTRACK_NETBIOS_NS=m
CONFIG_NF_CONNTRACK_SNMP=m
CONFIG_NF_CONNTRACK_PPTP=m
CONFIG_NF_CONNTRACK_SANE=m
CONFIG_NF_CONNTRACK_SIP=m
CONFIG_NF_CONNTRACK_TFTP=m
CONFIG_NF_CT_NETLINK=m
CONFIG_NETFILTER_TPROXY=m
CONFIG_NETFILTER_XTABLES=m

CONFIG_NETFILTER_XT_MARK=m
CONFIG_NETFILTER_XT_CONNMARK=m

CONFIG_NETFILTER_XT_TARGET_CHECKSUM=m
CONFIG_NETFILTER_XT_TARGET_CLASSIFY=m
CONFIG_NETFILTER_XT_TARGET_CONNMARK=m
CONFIG_NETFILTER_XT_TARGET_CT=m
CONFIG_NETFILTER_XT_TARGET_DSCP=m
CONFIG_NETFILTER_XT_TARGET_HL=m
CONFIG_NETFILTER_XT_TARGET_IDLETIMER=m
CONFIG_NETFILTER_XT_TARGET_MARK=m
CONFIG_NETFILTER_XT_TARGET_NFLOG=m
CONFIG_NETFILTER_XT_TARGET_NFQUEUE=m
CONFIG_NETFILTER_XT_TARGET_NOTRACK=m
CONFIG_NETFILTER_XT_TARGET_RATEEST=m
CONFIG_NETFILTER_XT_TARGET_TEE=m
CONFIG_NETFILTER_XT_TARGET_TPROXY=m
CONFIG_NETFILTER_XT_TARGET_TRACE=m
CONFIG_NETFILTER_XT_TARGET_TCPMSS=m
CONFIG_NETFILTER_XT_TARGET_TCPOPTSTRIP=m

CONFIG_NETFILTER_XT_MATCH_ADDRTYPE=m
CONFIG_NETFILTER_XT_MATCH_CLUSTER=m
CONFIG_NETFILTER_XT_MATCH_COMMENT=m
CONFIG_NETFILTER_XT_MATCH_CONNBYTES=m
CONFIG_NETFILTER_XT_MATCH_CONNLIMIT=m
CONFIG_NETFILTER_XT_MATCH_CONNMARK=m
CONFIG_NETFILTER_XT_MATCH_CONNTRACK=m
CONFIG_NETFILTER_XT_MATCH_CPU=m
CONFIG_NETFILTER_XT_MATCH_DCCP=m
CONFIG_NETFILTER_XT_MATCH_DEVGROUP=m
CONFIG_NETFILTER_XT_MATCH_DSCP=m
CONFIG_NETFILTER_XT_MATCH_ESP=m
CONFIG_NETFILTER_XT_MATCH_HASHLIMIT=m
CONFIG_NETFILTER_XT_MATCH_HELPER=m
CONFIG_NETFILTER_XT_MATCH_HL=m
CONFIG_NETFILTER_XT_MATCH_IPRANGE=m
CONFIG_NETFILTER_XT_MATCH_LENGTH=m
CONFIG_NETFILTER_XT_MATCH_LIMIT=m
CONFIG_NETFILTER_XT_MATCH_MAC=m
CONFIG_NETFILTER_XT_MATCH_MARK=m
CONFIG_NETFILTER_XT_MATCH_MULTIPORT=m
CONFIG_NETFILTER_XT_MATCH_OSF=m
CONFIG_NETFILTER_XT_MATCH_OWNER=m
CONFIG_NETFILTER_XT_MATCH_POLICY=m
CONFIG_NETFILTER_XT_MATCH_PHYSDEV=m
CONFIG_NETFILTER_XT_MATCH_PKTTYPE=m
CONFIG_NETFILTER_XT_MATCH_QUOTA=m
CONFIG_NETFILTER_XT_MATCH_RATEEST=m
CONFIG_NETFILTER_XT_MATCH_REALM=m
CONFIG_NETFILTER_XT_MATCH_RECENT=m
CONFIG_NETFILTER_XT_MATCH_SCTP=m
CONFIG_NETFILTER_XT_MATCH_SOCKET=m
CONFIG_NETFILTER_XT_MATCH_STATE=m
CONFIG_NETFILTER_XT_MATCH_STATISTIC=m
CONFIG_NETFILTER_XT_MATCH_STRING=m
CONFIG_NETFILTER_XT_MATCH_TCPMSS=m
CONFIG_NETFILTER_XT_MATCH_TIME=m
CONFIG_NETFILTER_XT_MATCH_U32=m

CONFIG_NF_DEFRAG_IPV4=m
CONFIG_NF_CONNTRACK_IPV4=m
CONFIG_NF_CONNTRACK_PROC_COMPAT=y
CONFIG_IP_NF_QUEUE=m
CONFIG_IP_NF_IPTABLES=m
CONFIG_IP_NF_MATCH_AH=m
CONFIG_IP_NF_MATCH_ECN=m
CONFIG_IP_NF_MATCH_TTL=m
CONFIG_IP_NF_FILTER=m
CONFIG_IP_NF_TARGET_REJECT=m
CONFIG_IP_NF_TARGET_LOG=m
CONFIG_IP_NF_TARGET_ULOG=m
CONFIG_NF_NAT=m
CONFIG_NF_NAT_NEEDED=y
CONFIG_IP_NF_TARGET_MASQUERADE=m
CONFIG_IP_NF_TARGET_NETMAP=m
CONFIG_IP_NF_TARGET_REDIRECT=m
CONFIG_NF_NAT_SNMP_BASIC=m
CONFIG_NF_NAT_PROTO_DCCP=m
CONFIG_NF_NAT_PROTO_GRE=m
CONFIG_NF_NAT_PROTO_UDPLITE=m
CONFIG_NF_NAT_PROTO_SCTP=m
CONFIG_NF_NAT_FTP=m
CONFIG_NF_NAT_IRC=m
CONFIG_NF_NAT_TFTP=m
CONFIG_NF_NAT_AMANDA=m
CONFIG_NF_NAT_PPTP=m
CONFIG_NF_NAT_H323=m
CONFIG_NF_NAT_SIP=m
CONFIG_IP_NF_MANGLE=m
CONFIG_IP_NF_TARGET_CLUSTERIP=m
CONFIG_IP_NF_TARGET_ECN=m
CONFIG_IP_NF_TARGET_TTL=m
CONFIG_IP_NF_RAW=m
CONFIG_IP_NF_ARPTABLES=m
CONFIG_IP_NF_ARPFILTER=m
CONFIG_IP_NF_ARP_MANGLE=m
CONFIG_IP_SCTP=m
CONFIG_SCTP_HMAC_MD5=y
CONFIG_STP=y
CONFIG_BRIDGE=y
CONFIG_BRIDGE_IGMP_SNOOPING=y
CONFIG_VLAN_8021Q=y
CONFIG_LLC=y
CONFIG_NET_SCHED=y

CONFIG_NET_SCH_CBQ=m
CONFIG_NET_SCH_HTB=m
CONFIG_NET_SCH_HFSC=m
CONFIG_NET_SCH_PRIO=m
CONFIG_NET_SCH_RED=m
CONFIG_NET_SCH_SFB=m
CONFIG_NET_SCH_SFQ=m
CONFIG_NET_SCH_TEQL=m
CONFIG_NET_SCH_TBF=m
CONFIG_NET_SCH_GRED=m
CONFIG_NET_SCH_DSMARK=m
CONFIG_NET_SCH_NETEM=m
CONFIG_NET_SCH_MQPRIO=m
CONFIG_NET_SCH_CHOKE=m
CONFIG_NET_SCH_INGRESS=m

CONFIG_NET_CLS=y
CONFIG_NET_CLS_BASIC=m
CONFIG_NET_CLS_TCINDEX=m
CONFIG_NET_CLS_ROUTE4=m
CONFIG_NET_CLS_FW=m
CONFIG_NET_CLS_U32=m
CONFIG_NET_CLS_FLOW=m
CONFIG_NET_EMATCH=y
CONFIG_NET_EMATCH_STACK=32
CONFIG_NET_EMATCH_CMP=m
CONFIG_NET_EMATCH_NBYTE=m
CONFIG_NET_EMATCH_U32=m
CONFIG_NET_EMATCH_META=m
CONFIG_NET_EMATCH_TEXT=m
CONFIG_NET_CLS_ACT=y
CONFIG_NET_ACT_POLICE=m
CONFIG_NET_ACT_GACT=m
CONFIG_GACT_PROB=y
CONFIG_NET_ACT_MIRRED=m
CONFIG_NET_ACT_IPT=m
CONFIG_NET_ACT_NAT=m
CONFIG_NET_ACT_PEDIT=m
CONFIG_NET_ACT_SIMP=m
CONFIG_NET_ACT_SKBEDIT=m
CONFIG_NET_ACT_CSUM=m
CONFIG_NET_SCH_FIFO=y
CONFIG_RPS=y
CONFIG_RFS_ACCEL=y
CONFIG_XPS=y



CONFIG_UEVENT_HELPER_PATH="/sbin/hotplug"
CONFIG_DEVTMPFS=y
CONFIG_STANDALONE=y
CONFIG_PREVENT_FIRMWARE_BUILD=y
CONFIG_FW_LOADER=y
CONFIG_FIRMWARE_IN_KERNEL=y
CONFIG_EXTRA_FIRMWARE=""
CONFIG_ARCH_NO_SYSDEV_OPS=y
CONFIG_PNP=y
CONFIG_PNP_DEBUG_MESSAGES=y

CONFIG_PNPACPI=y
CONFIG_BLK_DEV=y
CONFIG_BLK_DEV_LOOP=y

CONFIG_HAVE_IDE=y

CONFIG_SCSI_MOD=y
CONFIG_RAID_ATTRS=m
CONFIG_SCSI=y
CONFIG_SCSI_DMA=y
CONFIG_SCSI_PROC_FS=y

CONFIG_BLK_DEV_SD=y
CONFIG_BLK_DEV_SR=y
CONFIG_CHR_DEV_SG=y
CONFIG_SCSI_WAIT_SCAN=m

CONFIG_SCSI_SAS_ATTRS=m
CONFIG_SCSI_LOWLEVEL=y
CONFIG_MEGARAID_SAS=m
CONFIG_ATA=y
CONFIG_ATA_VERBOSE_ERROR=y
CONFIG_ATA_ACPI=y
CONFIG_SATA_PMP=y

CONFIG_SATA_AHCI=m
CONFIG_SATA_SIL24=m
CONFIG_ATA_SFF=y

CONFIG_ATA_BMDMA=y

CONFIG_SATA_MV=m
CONFIG_SATA_NV=m
CONFIG_SATA_PROMISE=m
CONFIG_SATA_SIL=m



CONFIG_MD=y
CONFIG_BLK_DEV_MD=y
CONFIG_MD_LINEAR=y
CONFIG_MD_RAID0=y
CONFIG_MD_RAID1=y
CONFIG_MD_RAID10=y
CONFIG_MD_RAID456=y
CONFIG_BLK_DEV_DM=y

CONFIG_NETDEVICES=y
CONFIG_DUMMY=m
CONFIG_BONDING=m
CONFIG_TUN=y
CONFIG_MII=m
CONFIG_NET_ETHERNET=y
CONFIG_NET_PCI=y
CONFIG_FORCEDETH=m
CONFIG_NETDEV_1000=y
CONFIG_R8169=m



CONFIG_PPP=m
CONFIG_PPP_MULTILINK=y
CONFIG_PPP_FILTER=y
CONFIG_PPP_ASYNC=m
CONFIG_PPP_SYNC_TTY=m
CONFIG_PPP_DEFLATE=m
CONFIG_PPP_BSDCOMP=m
CONFIG_PPP_MPPE=m
CONFIG_PPPOE=m
CONFIG_SLHC=m
CONFIG_NETCONSOLE=m
CONFIG_NETCONSOLE_DYNAMIC=y
CONFIG_NETPOLL=y
CONFIG_NET_POLL_CONTROLLER=y

CONFIG_INPUT=y

CONFIG_INPUT_MOUSEDEV=y
CONFIG_INPUT_MOUSEDEV_SCREEN_X=1024
CONFIG_INPUT_MOUSEDEV_SCREEN_Y=768

CONFIG_INPUT_KEYBOARD=y
CONFIG_KEYBOARD_ATKBD=y

CONFIG_SERIO=y
CONFIG_SERIO_I8042=y
CONFIG_SERIO_LIBPS2=y

CONFIG_VT=y
CONFIG_CONSOLE_TRANSLATIONS=y
CONFIG_VT_CONSOLE=y
CONFIG_HW_CONSOLE=y
CONFIG_UNIX98_PTYS=y
CONFIG_LEGACY_PTYS=y
CONFIG_LEGACY_PTY_COUNT=256

CONFIG_SERIAL_8250=y
CONFIG_SERIAL_8250_CONSOLE=y
CONFIG_FIX_EARLYCON_MEM=y
CONFIG_SERIAL_8250_PCI=y
CONFIG_SERIAL_8250_PNP=y
CONFIG_SERIAL_8250_NR_UARTS=4
CONFIG_SERIAL_8250_RUNTIME_UARTS=4

CONFIG_SERIAL_CORE=y
CONFIG_SERIAL_CORE_CONSOLE=y
CONFIG_RTC=y
CONFIG_HPET=y
CONFIG_HPET_MMAP=y
CONFIG_DEVPORT=y
CONFIG_I2C=y
CONFIG_I2C_BOARDINFO=y
CONFIG_I2C_COMPAT=y
CONFIG_I2C_CHARDEV=y
CONFIG_I2C_HELPER_AUTO=y
CONFIG_I2C_ALGOBIT=y


CONFIG_I2C_AMD756=m
CONFIG_I2C_PIIX4=m
CONFIG_I2C_NFORCE2=m
CONFIG_I2C_VIA=m
CONFIG_I2C_VIAPRO=m

CONFIG_I2C_SCMI=m





CONFIG_ARCH_WANT_OPTIONAL_GPIOLIB=y
CONFIG_HWMON=y
CONFIG_HWMON_VID=m

CONFIG_SENSORS_K8TEMP=m
CONFIG_SENSORS_K10TEMP=m
CONFIG_SENSORS_W83781D=m
CONFIG_SENSORS_W83791D=m
CONFIG_SENSORS_W83792D=m
CONFIG_SENSORS_W83793=m
CONFIG_SENSORS_W83795=m
CONFIG_SENSORS_W83795_FANCTRL=y
CONFIG_SENSORS_W83L785TS=m
CONFIG_SENSORS_W83L786NG=m
CONFIG_SENSORS_W83627HF=m
CONFIG_SENSORS_W83627EHF=m

CONFIG_SENSORS_ATK0110=m
CONFIG_THERMAL=y
CONFIG_THERMAL_HWMON=y
CONFIG_SSB_POSSIBLE=y


CONFIG_VGA_ARB=y
CONFIG_VGA_ARB_MAX_GPUS=16
CONFIG_VGASTATE=y
CONFIG_FB=y
CONFIG_FB_DDC=y
CONFIG_FB_CFB_FILLRECT=y
CONFIG_FB_CFB_COPYAREA=y
CONFIG_FB_CFB_IMAGEBLIT=y
CONFIG_FB_MODE_HELPERS=y

CONFIG_FB_NVIDIA=y
CONFIG_FB_NVIDIA_I2C=y
CONFIG_FB_RADEON=y
CONFIG_FB_RADEON_I2C=y


CONFIG_VGA_CONSOLE=y
CONFIG_DUMMY_CONSOLE=y
CONFIG_FRAMEBUFFER_CONSOLE=y
CONFIG_FONT_8x8=y
CONFIG_FONT_8x16=y
CONFIG_LOGO=y
CONFIG_LOGO_LINUX_MONO=y
CONFIG_LOGO_LINUX_VGA16=y
CONFIG_LOGO_LINUX_CLUT224=y
CONFIG_HID_SUPPORT=y
CONFIG_HID=y
CONFIG_HIDRAW=y

CONFIG_USB_HID=m
CONFIG_USB_HIDDEV=y

CONFIG_HID_A4TECH=m
CONFIG_HID_APPLE=m
CONFIG_HID_BELKIN=m
CONFIG_HID_CHERRY=m
CONFIG_HID_CHICONY=m
CONFIG_HID_CYPRESS=m
CONFIG_HID_EZKEY=m
CONFIG_HID_KYE=m
CONFIG_HID_KENSINGTON=m
CONFIG_HID_LOGITECH=m
CONFIG_HID_MICROSOFT=m
CONFIG_HID_MONTEREY=m
CONFIG_USB_SUPPORT=y
CONFIG_USB_ARCH_HAS_HCD=y
CONFIG_USB_ARCH_HAS_OHCI=y
CONFIG_USB_ARCH_HAS_EHCI=y
CONFIG_USB=m


CONFIG_USB_XHCI_HCD=m
CONFIG_USB_EHCI_HCD=m
CONFIG_USB_OHCI_HCD=m
CONFIG_USB_OHCI_LITTLE_ENDIAN=y
CONFIG_USB_UHCI_HCD=m


CONFIG_USB_ACM=m


CONFIG_USB_STORAGE=m
CONFIG_USB_LIBUSUAL=y


CONFIG_USB_SERIAL=m
CONFIG_USB_SERIAL_GENERIC=y
CONFIG_USB_SERIAL_FTDI_SIO=m
CONFIG_USB_SERIAL_IPAQ=m
CONFIG_USB_SERIAL_PL2303=m



CONFIG_FIRMWARE_MEMMAP=y
CONFIG_DMIID=y

CONFIG_EXT2_FS=y
CONFIG_EXT2_FS_XATTR=y
CONFIG_EXT3_FS=y
CONFIG_EXT3_FS_XATTR=y
CONFIG_EXT4_FS=y
CONFIG_EXT4_FS_XATTR=y
CONFIG_JBD=y
CONFIG_JBD2=y
CONFIG_FS_MBCACHE=y
CONFIG_EXPORTFS=y
CONFIG_FILE_LOCKING=y
CONFIG_FSNOTIFY=y
CONFIG_DNOTIFY=y
CONFIG_INOTIFY_USER=y
CONFIG_FANOTIFY=y
CONFIG_FUSE_FS=m
CONFIG_CUSE=m

CONFIG_FSCACHE=m
CONFIG_FSCACHE_DEBUG=y
CONFIG_CACHEFILES=m

CONFIG_ISO9660_FS=m
CONFIG_JOLIET=y
CONFIG_ZISOFS=y
CONFIG_UDF_FS=m
CONFIG_UDF_NLS=y

CONFIG_FAT_FS=m
CONFIG_MSDOS_FS=m
CONFIG_VFAT_FS=m
CONFIG_FAT_DEFAULT_CODEPAGE=437
CONFIG_FAT_DEFAULT_IOCHARSET="iso8859-1"
CONFIG_NTFS_FS=m

CONFIG_PROC_FS=y
CONFIG_PROC_SYSCTL=y
CONFIG_PROC_PAGE_MONITOR=y
CONFIG_SYSFS=y
CONFIG_TMPFS=y
CONFIG_HUGETLBFS=y
CONFIG_HUGETLB_PAGE=y
CONFIG_CONFIGFS_FS=m
CONFIG_NETWORK_FILESYSTEMS=y
CONFIG_NFS_FS=y
CONFIG_NFS_V3=y
CONFIG_NFSD=y
CONFIG_NFSD_DEPRECATED=y
CONFIG_NFSD_V3=y
CONFIG_LOCKD=y
CONFIG_LOCKD_V4=y
CONFIG_NFS_COMMON=y
CONFIG_SUNRPC=y

CONFIG_PARTITION_ADVANCED=y
CONFIG_MSDOS_PARTITION=y
CONFIG_EFI_PARTITION=y
CONFIG_NLS=y
CONFIG_NLS_DEFAULT="iso8859-1"
CONFIG_NLS_CODEPAGE_437=y
CONFIG_NLS_ISO8859_1=y
CONFIG_DLM=m

CONFIG_TRACE_IRQFLAGS_SUPPORT=y
CONFIG_PRINTK_TIME=y
CONFIG_DEFAULT_MESSAGE_LOGLEVEL=4
CONFIG_ENABLE_WARN_DEPRECATED=y
CONFIG_ENABLE_MUST_CHECK=y
CONFIG_FRAME_WARN=2048
CONFIG_MAGIC_SYSRQ=y
CONFIG_DEBUG_FS=y
CONFIG_DEBUG_KERNEL=y
CONFIG_DETECT_HUNG_TASK=y
CONFIG_BOOTPARAM_HUNG_TASK_PANIC_VALUE=0
CONFIG_TIMER_STATS=y
CONFIG_DEBUG_BUGVERBOSE=y
CONFIG_DEBUG_MEMORY_INIT=y
CONFIG_ARCH_WANT_FRAME_POINTERS=y
CONFIG_SYSCTL_SYSCALL_CHECK=y
CONFIG_USER_STACKTRACE_SUPPORT=y
CONFIG_HAVE_FUNCTION_TRACER=y
CONFIG_HAVE_FUNCTION_GRAPH_TRACER=y
CONFIG_HAVE_FUNCTION_GRAPH_FP_TEST=y
CONFIG_HAVE_FUNCTION_TRACE_MCOUNT_TEST=y
CONFIG_HAVE_DYNAMIC_FTRACE=y
CONFIG_HAVE_FTRACE_MCOUNT_RECORD=y
CONFIG_HAVE_SYSCALL_TRACEPOINTS=y
CONFIG_HAVE_C_RECORDMCOUNT=y
CONFIG_TRACING_SUPPORT=y
CONFIG_HAVE_ARCH_KGDB=y
CONFIG_HAVE_ARCH_KMEMCHECK=y
CONFIG_X86_VERBOSE_BOOTUP=y
CONFIG_EARLY_PRINTK=y
CONFIG_HAVE_MMIOTRACE_SUPPORT=y
CONFIG_IO_DELAY_TYPE_0X80=0
CONFIG_IO_DELAY_TYPE_0XED=1
CONFIG_IO_DELAY_TYPE_UDELAY=2
CONFIG_IO_DELAY_TYPE_NONE=3
CONFIG_IO_DELAY_0X80=y
CONFIG_DEFAULT_IO_DELAY_TYPE=0

CONFIG_DEFAULT_SECURITY_DAC=y
CONFIG_DEFAULT_SECURITY=""
CONFIG_XOR_BLOCKS=y
CONFIG_ASYNC_CORE=y
CONFIG_ASYNC_MEMCPY=y
CONFIG_ASYNC_XOR=y
CONFIG_ASYNC_PQ=y
CONFIG_ASYNC_RAID6_RECOV=y
CONFIG_CRYPTO=y

CONFIG_CRYPTO_ALGAPI=m
CONFIG_CRYPTO_ALGAPI2=m
CONFIG_CRYPTO_AEAD=m
CONFIG_CRYPTO_AEAD2=m
CONFIG_CRYPTO_BLKCIPHER=m
CONFIG_CRYPTO_BLKCIPHER2=m
CONFIG_CRYPTO_HASH=m
CONFIG_CRYPTO_HASH2=m
CONFIG_CRYPTO_RNG2=m
CONFIG_CRYPTO_PCOMP2=m
CONFIG_CRYPTO_MANAGER=m
CONFIG_CRYPTO_MANAGER2=m
CONFIG_CRYPTO_MANAGER_DISABLE_TESTS=y
CONFIG_CRYPTO_WORKQUEUE=m
CONFIG_CRYPTO_AUTHENC=m


CONFIG_CRYPTO_CBC=m
CONFIG_CRYPTO_ECB=m

CONFIG_CRYPTO_HMAC=m

CONFIG_CRYPTO_CRC32C=m
CONFIG_CRYPTO_MD5=m
CONFIG_CRYPTO_SHA1=m

CONFIG_CRYPTO_ARC4=m
CONFIG_CRYPTO_DES=m

CONFIG_CRYPTO_DEFLATE=m

CONFIG_HAVE_KVM=y
CONFIG_HAVE_KVM_IRQCHIP=y
CONFIG_HAVE_KVM_EVENTFD=y
CONFIG_KVM_APIC_ARCHITECTURE=y
CONFIG_KVM_MMIO=y
CONFIG_KVM_ASYNC_PF=y
CONFIG_VIRTUALIZATION=y
CONFIG_KVM=m
CONFIG_KVM_INTEL=m
CONFIG_KVM_AMD=m
CONFIG_VHOST_NET=m

CONFIG_RAID6_PQ=y
CONFIG_BITREVERSE=y
CONFIG_GENERIC_FIND_FIRST_BIT=y
CONFIG_GENERIC_FIND_NEXT_BIT=y
CONFIG_GENERIC_FIND_LAST_BIT=y
CONFIG_CRC_CCITT=m
CONFIG_CRC16=y
CONFIG_CRC_T10DIF=y
CONFIG_CRC_ITU_T=y
CONFIG_CRC32=y
CONFIG_LIBCRC32C=m
CONFIG_ZLIB_INFLATE=y
CONFIG_ZLIB_DEFLATE=m
CONFIG_LZO_DECOMPRESS=y
CONFIG_XZ_DEC=y
CONFIG_XZ_DEC_X86=y
CONFIG_XZ_DEC_POWERPC=y
CONFIG_XZ_DEC_IA64=y
CONFIG_XZ_DEC_ARM=y
CONFIG_XZ_DEC_ARMTHUMB=y
CONFIG_XZ_DEC_SPARC=y
CONFIG_XZ_DEC_BCJ=y
CONFIG_DECOMPRESS_GZIP=y
CONFIG_DECOMPRESS_BZIP2=y
CONFIG_DECOMPRESS_LZMA=y
CONFIG_DECOMPRESS_XZ=y
CONFIG_DECOMPRESS_LZO=y
CONFIG_TEXTSEARCH=y
CONFIG_TEXTSEARCH_KMP=m
CONFIG_TEXTSEARCH_BM=m
CONFIG_TEXTSEARCH_FSM=m
CONFIG_HAS_IOMEM=y
CONFIG_HAS_IOPORT=y
CONFIG_HAS_DMA=y
CONFIG_CPU_RMAP=y
CONFIG_NLATTR=y

00:00.0 Host bridge: Advanced Micro Devices [AMD] RS880 Host Bridge
00:01.0 PCI bridge: ASRock Incorporation Device 9602
00:02.0 PCI bridge: Advanced Micro Devices [AMD] RS780 PCI to PCI bridge 
(ext gfx port 0)
00:03.0 PCI bridge: Advanced Micro Devices [AMD] RS780 PCI to PCI bridge 
(ext gfx port 1)
00:04.0 PCI bridge: Advanced Micro Devices [AMD] RS780 PCI to PCI bridge 
(PCIE port 0)
00:09.0 PCI bridge: Advanced Micro Devices [AMD] RS780 PCI to PCI bridge 
(PCIE port 4)
00:0a.0 PCI bridge: Advanced Micro Devices [AMD] RS780 PCI to PCI bridge 
(PCIE port 5)
00:11.0 SATA controller: ATI Technologies Inc SB700/SB800 SATA 
Controller [AHCI mode] (rev 40)
00:12.0 USB Controller: ATI Technologies Inc SB700/SB800 USB OHCI0 
Controller
00:12.2 USB Controller: ATI Technologies Inc SB700/SB800 USB EHCI Controller
00:13.0 USB Controller: ATI Technologies Inc SB700/SB800 USB OHCI0 
Controller
00:13.2 USB Controller: ATI Technologies Inc SB700/SB800 USB EHCI Controller
00:14.0 SMBus: ATI Technologies Inc SBx00 SMBus Controller (rev 41)
00:14.2 Audio device: ATI Technologies Inc SBx00 Azalia (Intel HDA) (rev 40)
00:14.3 ISA bridge: ATI Technologies Inc SB700/SB800 LPC host controller 
(rev 40)
00:14.4 PCI bridge: ATI Technologies Inc SBx00 PCI to PCI Bridge (rev 40)
00:14.5 USB Controller: ATI Technologies Inc SB700/SB800 USB OHCI2 
Controller
00:16.0 USB Controller: ATI Technologies Inc SB700/SB800 USB OHCI0 
Controller
00:16.2 USB Controller: ATI Technologies Inc SB700/SB800 USB EHCI Controller
00:18.0 Host bridge: Advanced Micro Devices [AMD] Family 10h Processor 
HyperTransport Configuration
00:18.1 Host bridge: Advanced Micro Devices [AMD] Family 10h Processor 
Address Map
00:18.2 Host bridge: Advanced Micro Devices [AMD] Family 10h Processor 
DRAM Controller
00:18.3 Host bridge: Advanced Micro Devices [AMD] Family 10h Processor 
Miscellaneous Control
00:18.4 Host bridge: Advanced Micro Devices [AMD] Family 10h Processor 
Link Control
01:05.0 VGA compatible controller: ATI Technologies Inc RS880 [Radeon HD 
4250]
02:00.0 USB Controller: NEC Corporation uPD720200 USB 3.0 Host 
Controller (rev 03)
03:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. 
RTL8111/8168B PCI Express Gigabit Ethernet controller (rev 06)
04:00.0 RAID bus controller: LSI Logic / Symbios Logic MegaRAID SAS 9240 
(rev 02)
05:00.0 RAID bus controller: LSI Logic / Symbios Logic MegaRAID SAS 9240 
(rev 02)
07:00.0 SCSI storage controller: Marvell Technology Group Ltd. 88SX7042 
PCI-e 4-port SATA-II (rev 02)

^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
  2011-05-31  1:24 KVM induced panic on 2.6.38[2367] & 2.6.39 Brad Campbell
@ 2011-05-31  5:47   ` Borislav Petkov
  0 siblings, 0 replies; 111+ messages in thread
From: Borislav Petkov @ 2011-05-31  5:47 UTC (permalink / raw)
  To: Brad Campbell; +Cc: linux-kernel, kvm, linux-mm

Looks like a KSM issue. Disabling CONFIG_KSM should at least stop your
machine from oopsing.

Adding linux-mm.

On Tue, May 31, 2011 at 09:24:03AM +0800, Brad Campbell wrote:
> G'day all,
> 
> I'm running a pretty standard home server
> x86_64 Phenom-II 6 Core
> 16GB DDR 3
> 
> I run some virtual machines under that. 3 x Debian 64 Bit, 1 x XP 32
> Bit. These run at boot.
> 
> When I fire up another XP 32 bit instance and play with it for more
> than about 2 minutes, I get the panics included in this mail.
> 
> I've included three of them here. The first and third are as booted.
> The second was with ksmd disabled just as a data point.
> 
> The machine passes every load test and memory test I can throw at
> it, but I still can't rule out this being a hardware issue.
> 
> Provided I don't start this XP VM the machine is quite stable, but
> running this VM will kill it within minutes.
> 
> This was tested with qemu-kvm.
> The last commit in the git tree was
> commit c007db193eb6b2557acb5caf2dc4d7023639e6f3
> Author: Avi Kivity <avi@redhat.com>
> Date:   Sun May 29 09:00:42 2011 -0400
> (I pulled it yesterday)
> 
> These panics were captured with netconsole to a remote syslog
> daemon, and the formatting was ruined, so I've reformatted them by
> hand prior to posting.
> 
> I've tested and reproduced this on 2.6.38.[2,3,6 & 7] and 2.6.39 obviously.
> 
> Can anyone help shed some light on this?
> 
> Regards,
> Brad
> 
> [ 438.632061] general protection fault: 0000 [#1] SMP
> [ 438.632196] last sysfs file: /sys/module/x_tables/initstate
> [ 438.632242] CPU 4
> [ 438.632282] Modules linked in: xt_iprange xt_DSCP xt_length
> xt_CLASSIFY sch_sfq xt_CHECKSUM ipt_REJECT ipt_MASQUERADE
> ipt_REDIRECT xt_recent xt_state iptable_nat nf_nat nf_conntrack_ipv4
> nf_conntrack nf_defrag_ipv4 iptable_filter xt_TCPMSS xt_tcpmss
> xt_tcpudp iptable_mangle ip_tables x_tables pppoe pppox ppp_generic
> slhc cls_u32 sch_htb deflate zlib_deflate des_generic cbc ecb
> crypto_blkcipher sha1_generic md5 hmac crypto_hash cryptomgr aead
> crypto_algapi af_key fuse netconsole configfs= vhost_net powernow_k8
> mperf i2c_nforce2 kvm_amd kvm pl2303 usbserial xhci_hcd k10temp
> i2c_piix4 usb_storage usb_libusual ohci_hcd ehci_hcd usbcore ahci
> libahci r8169 mii sata_mv megaraid_sas [last unloaded:
> scsi_wait_scan]
> [ 438.634960]
> [ 438.635006] Pid: 551, comm: ksmd Not tainted 2.6.39 #3 To Be
> Filled By O.E.M. To Be Filled By O.E.M. /880G Extreme3
> [ 438.635170] RIP: 0010:[<ffffffff810b4596>] [<ffffffff810b4596>]
> remove_rmap_item_from_tree+0x96/0x150
> [ 438.635268] RSP: 0018:ffff88041c065e20  EFLAGS: 00010282
> [ 438.635314] RAX: ffff8804153bd8b0 RBX: ffff8804176c3fc0 RCX:
> 0000000000057754
> [ 438.635362] RDX: 0000880415418030 RSI: ffff880414b65003 RDI:
> ffffea000dde6030
> [ 438.635410] RBP: ffff880414b65000 R08: 0000000000057755 R09:
> 000000001bbde1bd
> [ 438.635458] R10: 000000001c28e0c3 R11: 0000000000000002 R12:
> ffffea000dde6030
> [ 438.635506] R13: ffff8804176c3f80 R14: ffff8804151ed7b0 R15:
> ffff88041bf23be0
> [ 438.635555] FS:  00007f617772e700(0000) GS:ffff88041fd00000(0000)
> knlGS:0000000000000000
> [ 438.635607] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [ 438.635654] CR2: 0000000000e70000 CR3: 0000000001583000 CR4:
> 00000000000006e0
> [ 438.635703] DR0: 0000000000000045 DR1: 0000000000000000 DR2:
> 0000000000000000
> [ 438.635750] DR3: 0000000000000005 DR6: 00000000ffff0ff0 DR7:
> 0000000000000400
> [ 438.635799] Process ksmd (pid: 551, threadinfo ffff88041c064000,
> task ffff88041d8caa70)
> [ 438.635851] Stack:
> [ 438.635893]  ffffea000c43c408 ffff8804176c3fc0 000000000000036c
> ffffffff810b58f2
> [ 438.636088]  ffff88041c782a00 00007fc2d81b5000 ffff88041c064000
> 00000000003808ff
> [ 438.636281]  ffff88041c064000 ffff88041c065e98 ffff88041d8caa70
> ffff8804165d4480
> [ 438.636479] Call Trace:
> [ 438.636528]  [<ffffffff810b58f2>] ? ksm_scan_thread+0x4e2/0xc20
> [ 438.636580]  [<ffffffff81052a20>] ? wake_up_bit+0x40/0x40
> [ 438.636628]  [<ffffffff810b5410>] ? try_to_merge_with_ksm_page+0x570/0x570
> [ 438.636679]  [<ffffffff810b5410>] ? try_to_merge_with_ksm_page+0x570/0x570
> [ 438.636730]  [<ffffffff810525b6>] ? kthread+0x96/0xa0
> [ 438.636781]  [<ffffffff813d1794>] ? kernel_thread_helper+0x4/0x10
> [ 438.636832]  [<ffffffff81052520>] ? kthread_worker_fn+0x120/0x120
> [ 438.636882]  [<ffffffff813d1790>] ? gs_change+0xb/0xb
> [ 438.636926] Code: 28 48 89 ef e8 6c fe ff ff 48 85 c0 49 89 c4 74
> d2 f0 0f ba 28 00 19 c0 85 c0 0f 85 ae 00 00 00 00 48 8b 43 30 48 8b
> 53 38 48 85 c0
> [ 438.638504] 89 02 74 04 48 89 50 08 48 ba 00 01 10 00 00 00 00 ad de 48 b8
> [ 438.639329] RIP [<ffffffff810b4596>] remove_rmap_item_from_tree+0x96/0x150
> [ 438.639414]  RSP <ffff88041c065e20>
> [ 438.639460] ---[ end trace c29fb871f6b874e3 ]---
> [ 438.639506] Kernel panic - not syncing: Fatal exception
> [ 438.639553] Pid: 551, comm: ksmd Tainted: G      D     2.6.39 #3
> [ 438.639598] Call Trace:
> [ 438.639644]  [<ffffffff813cd6f5>] ? panic+0x92/0x18a
> [ 438.639693]  [<ffffffff81038b61>] ? kmsg_dump+0x41/0xf0
> [ 438.639745]  [<ffffffff810050ad>] ? oops_end+0x8d/0xa0
> [ 438.639794]  [<ffffffff813d05ef>] ? general_protection+0x1f/0x30
> [ 438.639842]  [<ffffffff810b4596>] ? remove_rmap_item_from_tree+0x96/0x150
> [ 438.639891]  [<ffffffff810b58f2>] ? ksm_scan_thread+0x4e2/0xc20
> [ 438.639941]  [<ffffffff81052a20>] ? wake_up_bit+0x40/0x40
> [ 438.640045]  [<ffffffff810b5410>] ? try_to_merge_with_ksm_page+0x570/0x570
> [ 438.640096]  [<ffffffff810b5410>] ? try_to_merge_with_ksm_page+0x570/0x570
> [ 438.640147]  [<ffffffff810525b6>] ? kthread+0x96/0xa0
> [ 438.640196]  [<ffffffff813d1794>] ? kernel_thread_helper+0x4/0x10
> [ 438.640247]  [<ffffffff81052520>] ? kthread_worker_fn+0x120/0x120
> [ 438.640297]  [<ffffffff813d1790>] ? gs_change+0xb/0xb
> [ 438.640349] Rebooting in 60 seconds..
> 
> 
> 
> [ 469.706013] general protection fault: 0000 [#1] SMP
> [ 469.706125] last sysfs file:
> /sys/devices/pci0000:00/0000:00:18.3/temp1_input
> [ 469.706162] CPU 1
> [ 469.706195] Modules linked in: xt_iprange xt_DSCP xt_length
> xt_CLASSIFY sch_sfq xt_CHECKSUM ipt_REJECT ipt_MASQUERADE
> ipt_REDIRECT xt_recent xt_state iptable_nat nf_nat nf_conntrack_ipv4
> nf_conntrack nf_defrag_ipv4 iptable_filter xt_TCPMSS xt_tcpmss
> xt_tcpudp iptable_mangle ip_tables x_tables pppoe pppox ppp_generic
> slhc cls_u32 sch_htb deflate zlib_deflate des_generi cbc ecb
> crypto_blkcipher sha1_generic md5 hmac crypto_hash cryptomgr aead
> crypto_algapi af_key fuse netconsole configfs vhost_net powernow_k8
> mperf i2c_nforce2 kvm_amd kvm pl2303 usbserial xhci_hcd i2c_piix4
> k10temp ahci usb_storage usb_libusual ohci_hcd ehci_hcd r8169
> libahci usbcore mii sata_mv megaraid_sas [last unloaded:
> scsi_wait_scan]
> [ 469.708573]
> [ 469.708608] Pid: 4942, comm: runbot Not tainted 2.6.39 #3 To Be
> Filled By O.E.M. To Be Filled By O.E.M. /880G Extreme3
> [ 469.708753] RIP: 0010:[<ffffffff810db878>] [<ffffffff810db878>]
> dup_fd+0x168/0x300
> [ 469.708829] RSP: 0018:ffff880417279dd0  EFLAGS: 00010202
> [ 469.708870] RAX: 00000000000007f8 RBX: ffff88041c1658c0 RCX:
> bfffffffffffffff
> [ 469.708907] RDX: 0000880413155540 RSI: 00000000000000ff RDI:
> 0000000000000800
> [ 469.708945] RBP: ffff88041bf22000 R08: ffff880412839dc0 R09:
> 0000000000000003
> [ 469.708981] R10: 0000000000000001 R11: 4000000000000000 R12:
> 0000000000000100
> [ 469.709018] R13: ffff88041d3cc000 R14: ffff880412839d40 R15:
> ffff88041c72f440
> [ 469.709054] FS:  00007f510c0b4700(0000) GS:ffff88041fc40000(0000)
> knlGS:0000000000000000
> [ 469.709094] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [ 469.709130] CR2: 00007f510ba56810 CR3: 0000000417c93000 CR4:
> 00000000000006e0
> [ 469.709167] DR0: 00000000000000a0 DR1: 0000000000000000 DR2:
> 0000000000000003
> [ 469.709204] DR3: 00000000000000b0 DR6: 00000000ffff0ff0 DR7:
> 0000000000000400
> [ 469.709240] Process runbot (pid: 4942, threadinfo
> ffff880417278000, task ffff880401351230)
> [ 469.709279] Stack:
> [ 469.709313]  ffff880401351230 0000000000000020 ffff880412e9ca80
> ffff88041c1658d0 ffff88041c1658d0
> [ 469.709477]  ffff88041723f330 0000000001200011 ffff88041723f330
> 0000000000000000
> [ 469.709641]  00007f510c0b49d0 ffff880401351230 0000000000000000
> ffffffff81037645
> [ 469.709812] Call Trace:
> [ 469.709849]  [<ffffffff81037645>] ? copy_process+0xa75/0xfd0
> [ 469.709885]  [<ffffffff81037c0d>] ? do_fork+0x6d/0x2b0
> [ 469.709921]  [<ffffffff810dbc23>] ? alloc_fd+0x43/0x130
> [ 469.709958]  [<ffffffff810c08c5>] ? fd_install+0x35/0x70
> [ 469.709998]  [<ffffffff810457a9>] ? sigprocmask+0x69/0x100
> [ 469.710042]  [<ffffffff813d0ca3>] ? stub_clone+0x13/0x20
> [ 469.710079]  [<ffffffff813d0a3b>] ? system_call_fastpath+0x16/0x1b
> [ 469.710114] Code:  4c 89 c2 e8 6b e5 0f 00 45 85 e4 74 78 41 8d 44
> 24 ff 31 f6 41 ba 01 00 00 00 00 48 8d 3c c5 08 00 00 00 00 31 c0 eb
> 1a 0f 1f 44 00 00 f0> 48 ff 42 30 48 89 54 05 00 48 83 c0
> May 30 16:39:36 192.168.2.1 08 ff c6 48 39 f8 74 3b
> [ 469.712247] RIP [<ffffffff810db878>] dup_fd+0x168/0x300
> [ 469.712317]  RSP <ffff880417279dd0>
> [ 469.712391] ---[ end trace b80596608cd6bb6b ]---
> [ 469.712432] Kernel panic - not syncing: Fatal exception
> [ 469.712470] Pid: 4942, comm: runbot Tainted: G      D     2.6.39 #3
> [ 469.712508] Call Trace:
> [ 469.712544]  [<ffffffff813cd6f5>] ? panic+0x92/0x18a
> [ 469.712581]  [<ffffffff81038b61>] ? kmsg_dump+0x41/0xf0
> [ 469.712618]  [<ffffffff810050ad>] ? oops_end+0x8d/0xa0
> [ 469.712654]  [<ffffffff813d05ef>] ? general_protection+0x1f/0x30
> [ 469.712691]  [<ffffffff810db878>] ? dup_fd+0x168/0x300
> [ 469.712727]  [<ffffffff81037645>] ? copy_process+0xa75/0xfd0
> [ 469.712767]  [<ffffffff81037c0d>] ? do_fork+0x6d/0x2b0
> [ 469.712803]  [<ffffffff810dbc23>] ? alloc_fd+0x43/0x130
> [ 469.712839]  [<ffffffff810c08c5>] ? fd_install+0x35/0x70
> [ 469.712875]  [<ffffffff810457a9>] ? sigprocmask+0x69/0x100
> [ 469.712912]  [<ffffffff813d0ca3>] ? stub_clone+0x13/0x20
> [ 469.712948]  [<ffffffff813d0a3b>] ? system_call_fastpath+0x16/0x1b
> [ 469.712989] Rebooting in 60 seconds..
> 
> 
> [ 1100.473612] general protection fault: 0000 [#1] SMP
> [ 1100.473718] last sysfs file:
> /sys/devices/pci0000:00/0000:00:18.3/temp1_input
> [ 1100.473755] CPU 0
> [ 1100.473788] Modules linked in: xt_iprange xt_DSCP xt_length
> xt_CLASSIFY sch_sfq xt_CHECKSUM ipt_REJECT ipt_MASQUERADE
> ipt_REDIRECT xt_recent xt_state iptable_nat nf_nat nf_conntrack_ipv4
> nf_conntrack nf_defrag_ipv4 xt_TCPMSS xt_tcpmss xt_tcpudp
> iptable_mangle pppoe pppox iptable_filter ip_tables x_tables
> ppp_generic slhc cls_u32 sch_htb deflate zlib_deflate des_generic
> cbc ecb crypto_blkcipher sha1_generic md5 hmac crypto_hash cryptomgr
> aead crypto_algapi af_key fuse netconsole configfs vhost_net
> powernow_k8 mperf i2c_nforce2 kvm_amd kvm pl2303 xhci_hcd usbserial
> k10temp i2c_piix4 usb_storage usb_libusual ohci_hcd ehci_hcd ahci
> usbcore sata_mv r8169 mii libahci megaraid_sas [last unloaded:
> scsi_wait_scan]
> [ 1100.476102]
> [ 1100.476137] Pid: 551, comm: ksmd Not tainted 2.6.39 #3 To Be
> Filled By O.E.M. To Be Filled By O.E.M. /880G Extreme3
> [ 1100.476277] RIP: 0010:[<ffffffff810b5c71>] [<ffffffff810b5c71>]
> ksm_scan_thread+0x861/0xc20
> [ 1100.476351] RSP: 0018:ffff88041c065e40  EFLAGS: 00010282
> [ 1100.476387] RAX: 0000000000000000 RBX: ffffea000a5c8468 RCX:
> 0000000000000f01
> [ 1100.476424] RDX: 0000880403456828 RSI: ffff8802f5dcb002 RDI:
> ffffea000ce7eac8
> [ 1100.476460] RBP: ffff880418377fc0 R08: 6db6db6db6db6db7 R09:
> 0000160000000000
> [ 1100.476498] R10: ffffea000ce7eac8 R11: 0000000080100073 R12:
> 00000000000000ab
> [ 1100.476535] R13: ffff880418377ff8 R14: ffffea000ce7eac8 R15:
> ffff880418377fe8
> [ 1100.476572] FS:  00007f08c6498700(0000) GS:ffff88041fc00000(0000)
> knlGS:0000000000000000
> [ 1100.476611] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [ 1100.476646] CR2: 00000000e1dd3000 CR3: 00000004025f7000 CR4:
> 00000000000006f0
> [ 1100.477007] DR0: 0000000000000045 DR1: 0000000000000000 DR2:
> 0000000000000000
> [ 1100.477042] DR3: 0000000000000005 DR6: 00000000ffff0ff0 DR7:
> 0000000000000400
> [ 1100.477078] Process ksmd (pid: 551, threadinfo ffff88041c064000,
> task ffff88041d0454e0)
> [ 1100.477115] Stack:
> [ 1100.477148]  ffff880403a8b100 00007fc1863cb000 ffff88041c064000
> 00000000002f5dcb
> [ 1100.477317]  ffff88041c064000 ffff88041c065e98 ffff88041d0454e0
> ffff88041267a580
> [ 1100.477480]  0000000000000000 ffff88041d0454e0 ffffffff81052a20
> ffff88041c065e98
> [ 1100.477644] Call Trace:
> [ 1100.477681]  [<ffffffff81052a20>] ? wake_up_bit+0x40/0x40
> [ 1100.477717]  [<ffffffff810b5410>] ?
> try_to_merge_with_ksm_page+0x570/0x570
> [ 1100.477754]  [<ffffffff810b5410>] ?
> try_to_merge_with_ksm_page+0x570/0x570
> [ 1100.477791]  [<ffffffff810525b6>] ? kthread+0x96/0xa0
> [ 1100.477829]  [<ffffffff813d1794>] ? kernel_thread_helper+0x4/0x10
> [ 1100.477865]  [<ffffffff81052520>] ? kthread_worker_fn+0x120/0x120
> [ 1100.477902]  [<ffffffff813d1790>] ? gs_change+0xb/0xb
> [ 1100.477937] Code: 05 99 6e 6e 00 4c 89 ff e8 ed 12 fa ff e9 6c f9
> ff ff 48 89 6c 24 38 49 83 7d 00 00 0f 84 63 01 00 00 e8 b3 8b 31 00
> 49 8b 55 00
> [ 1100.479306] 8b 42 e8 4c 8b 72 f0 48 8d 6a d8 4c 8d 78 60 48 89 04 24 4c
> [ 1100.480012] RIP [<ffffffff810b5c71>] ksm_scan_thread+0x861/0xc20
> [ 1100.480015]  RSP <ffff88041c065e40>
> [ 1100.480141] ---[ end trace 7382777d9e74ea23 ]---
> [ 1100.480178] Kernel panic - not syncing: Fatal exception
> [ 1100.480216] Pid: 551, comm: ksmd Tainted: G      D     2.6.39 #3
> [ 1100.480257] Call Trace:
> [ 1100.480297]  [<ffffffff813cd6f5>] ? panic+0x92/0x18a
> [ 1100.480340]  [<ffffffff81038b61>] ? kmsg_dump+0x41/0xf0
> [ 1100.480384]  [<ffffffff810050ad>] ? oops_end+0x8d/0xa0
> [ 1100.480425]  [<ffffffff813d05ef>] ? general_protection+0x1f/0x30
> [ 1100.480465]  [<ffffffff810b5c71>] ? ksm_scan_thread+0x861/0xc20
> [ 1100.480507]  [<ffffffff81052a20>] ? wake_up_bit+0x40/0x40
> [ 1100.480545]  [<ffffffff810b5410>] ?
> try_to_merge_with_ksm_page+0x570/0x570
> [ 1100.480584]  [<ffffffff810b5410>] ?
> try_to_merge_with_ksm_page+0x570/0x570
> [ 1100.480624]  [<ffffffff810525b6>] ? kthread+0x96/0xa0
> [ 1100.480669]  [<ffffffff813d1794>] ? kernel_thread_helper+0x4/0x10
> [ 1100.480708]  [<ffffffff81052520>] ? kthread_worker_fn+0x120/0x120
> [ 1100.480748]  [<ffffffff813d1790>] ? gs_change+0xb/0xb
> [ 1100.480794] Rebooting in 60 seconds..
> 
> brad@srv:~$ zcat /proc/config.gz | grep -v "#"
> CONFIG_64BIT=y
> CONFIG_X86_64=y
> CONFIG_X86=y
> CONFIG_INSTRUCTION_DECODER=y
> CONFIG_OUTPUT_FORMAT="elf64-x86-64"
> CONFIG_ARCH_DEFCONFIG="arch/x86/configs/x86_64_defconfig"
> CONFIG_GENERIC_CMOS_UPDATE=y
> CONFIG_CLOCKSOURCE_WATCHDOG=y
> CONFIG_GENERIC_CLOCKEVENTS=y
> CONFIG_GENERIC_CLOCKEVENTS_BROADCAST=y
> CONFIG_LOCKDEP_SUPPORT=y
> CONFIG_STACKTRACE_SUPPORT=y
> CONFIG_HAVE_LATENCYTOP_SUPPORT=y
> CONFIG_MMU=y
> CONFIG_ZONE_DMA=y
> CONFIG_NEED_DMA_MAP_STATE=y
> CONFIG_NEED_SG_DMA_LENGTH=y
> CONFIG_GENERIC_ISA_DMA=y
> CONFIG_GENERIC_IOMAP=y
> CONFIG_GENERIC_BUG=y
> CONFIG_GENERIC_BUG_RELATIVE_POINTERS=y
> CONFIG_GENERIC_HWEIGHT=y
> CONFIG_ARCH_MAY_HAVE_PC_FDC=y
> CONFIG_RWSEM_XCHGADD_ALGORITHM=y
> CONFIG_ARCH_HAS_CPU_IDLE_WAIT=y
> CONFIG_GENERIC_CALIBRATE_DELAY=y
> CONFIG_GENERIC_TIME_VSYSCALL=y
> CONFIG_ARCH_HAS_CPU_RELAX=y
> CONFIG_ARCH_HAS_DEFAULT_IDLE=y
> CONFIG_ARCH_HAS_CACHE_LINE_SIZE=y
> CONFIG_HAVE_SETUP_PER_CPU_AREA=y
> CONFIG_NEED_PER_CPU_EMBED_FIRST_CHUNK=y
> CONFIG_NEED_PER_CPU_PAGE_FIRST_CHUNK=y
> CONFIG_HAVE_CPUMASK_OF_CPU_MAP=y
> CONFIG_ARCH_HIBERNATION_POSSIBLE=y
> CONFIG_ARCH_SUSPEND_POSSIBLE=y
> CONFIG_ZONE_DMA32=y
> CONFIG_ARCH_POPULATES_NODE_MAP=y
> CONFIG_AUDIT_ARCH=y
> CONFIG_ARCH_SUPPORTS_OPTIMIZED_INLINING=y
> CONFIG_ARCH_SUPPORTS_DEBUG_PAGEALLOC=y
> CONFIG_X86_64_SMP=y
> CONFIG_X86_HT=y
> CONFIG_ARCH_HWEIGHT_CFLAGS="-fcall-saved-rdi -fcall-saved-rsi
> -fcall-saved-rdx -fcall-saved-rcx -fcall-saved-r8 -fcall-saved-r9
> -fcall-saved-r10 -fcall-saved-r11"
> CONFIG_DEFCONFIG_LIST="/lib/modules/$UNAME_RELEASE/.config"
> CONFIG_CONSTRUCTORS=y
> CONFIG_HAVE_IRQ_WORK=y
> CONFIG_IRQ_WORK=y
> 
> CONFIG_EXPERIMENTAL=y
> CONFIG_INIT_ENV_ARG_LIMIT=32
> CONFIG_CROSS_COMPILE=""
> CONFIG_LOCALVERSION=""
> CONFIG_LOCALVERSION_AUTO=y
> CONFIG_HAVE_KERNEL_GZIP=y
> CONFIG_HAVE_KERNEL_BZIP2=y
> CONFIG_HAVE_KERNEL_LZMA=y
> CONFIG_HAVE_KERNEL_XZ=y
> CONFIG_HAVE_KERNEL_LZO=y
> CONFIG_KERNEL_GZIP=y
> CONFIG_SWAP=y
> CONFIG_SYSVIPC=y
> CONFIG_SYSVIPC_SYSCTL=y
> CONFIG_BSD_PROCESS_ACCT=y
> CONFIG_TASKSTATS=y
> CONFIG_TASK_DELAY_ACCT=y
> CONFIG_TASK_XACCT=y
> CONFIG_TASK_IO_ACCOUNTING=y
> CONFIG_HAVE_GENERIC_HARDIRQS=y
> 
> CONFIG_GENERIC_HARDIRQS=y
> CONFIG_HAVE_SPARSE_IRQ=y
> CONFIG_GENERIC_IRQ_PROBE=y
> CONFIG_GENERIC_IRQ_SHOW=y
> CONFIG_GENERIC_PENDING_IRQ=y
> CONFIG_IRQ_FORCED_THREADING=y
> 
> CONFIG_TREE_RCU=y
> CONFIG_RCU_FANOUT=64
> CONFIG_IKCONFIG=y
> CONFIG_IKCONFIG_PROC=y
> CONFIG_LOG_BUF_SHIFT=20
> CONFIG_HAVE_UNSTABLE_SCHED_CLOCK=y
> CONFIG_NAMESPACES=y
> CONFIG_BLK_DEV_INITRD=y
> CONFIG_INITRAMFS_SOURCE=""
> CONFIG_RD_GZIP=y
> CONFIG_RD_BZIP2=y
> CONFIG_RD_LZMA=y
> CONFIG_RD_XZ=y
> CONFIG_RD_LZO=y
> CONFIG_SYSCTL=y
> CONFIG_ANON_INODES=y
> CONFIG_UID16=y
> CONFIG_SYSCTL_SYSCALL=y
> CONFIG_KALLSYMS=y
> CONFIG_KALLSYMS_ALL=y
> CONFIG_KALLSYMS_EXTRA_PASS=y
> CONFIG_HOTPLUG=y
> CONFIG_PRINTK=y
> CONFIG_BUG=y
> CONFIG_ELF_CORE=y
> CONFIG_PCSPKR_PLATFORM=y
> CONFIG_BASE_FULL=y
> CONFIG_FUTEX=y
> CONFIG_EPOLL=y
> CONFIG_SIGNALFD=y
> CONFIG_TIMERFD=y
> CONFIG_EVENTFD=y
> CONFIG_SHMEM=y
> CONFIG_AIO=y
> CONFIG_HAVE_PERF_EVENTS=y
> 
> CONFIG_PERF_EVENTS=y
> CONFIG_VM_EVENT_COUNTERS=y
> CONFIG_PCI_QUIRKS=y
> CONFIG_SLUB_DEBUG=y
> CONFIG_SLUB=y
> CONFIG_HAVE_OPROFILE=y
> CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS=y
> CONFIG_USER_RETURN_NOTIFIER=y
> CONFIG_HAVE_IOREMAP_PROT=y
> CONFIG_HAVE_KPROBES=y
> CONFIG_HAVE_KRETPROBES=y
> CONFIG_HAVE_OPTPROBES=y
> CONFIG_HAVE_ARCH_TRACEHOOK=y
> CONFIG_HAVE_DMA_ATTRS=y
> CONFIG_USE_GENERIC_SMP_HELPERS=y
> CONFIG_HAVE_REGS_AND_STACK_ACCESS_API=y
> CONFIG_HAVE_DMA_API_DEBUG=y
> CONFIG_HAVE_HW_BREAKPOINT=y
> CONFIG_HAVE_MIXED_BREAKPOINTS_REGS=y
> CONFIG_HAVE_USER_RETURN_NOTIFIER=y
> CONFIG_HAVE_PERF_EVENTS_NMI=y
> CONFIG_HAVE_ARCH_JUMP_LABEL=y
> 
> CONFIG_SLABINFO=y
> CONFIG_RT_MUTEXES=y
> CONFIG_BASE_SMALL=0
> CONFIG_MODULES=y
> CONFIG_MODULE_UNLOAD=y
> CONFIG_STOP_MACHINE=y
> CONFIG_BLOCK=y
> CONFIG_BLK_DEV_BSG=y
> CONFIG_BLOCK_COMPAT=y
> 
> CONFIG_IOSCHED_NOOP=y
> CONFIG_IOSCHED_DEADLINE=y
> CONFIG_IOSCHED_CFQ=y
> CONFIG_DEFAULT_CFQ=y
> CONFIG_DEFAULT_IOSCHED="cfq"
> CONFIG_PREEMPT_NOTIFIERS=y
> CONFIG_INLINE_SPIN_UNLOCK=y
> CONFIG_INLINE_SPIN_UNLOCK_IRQ=y
> CONFIG_INLINE_READ_UNLOCK=y
> CONFIG_INLINE_READ_UNLOCK_IRQ=y
> CONFIG_INLINE_WRITE_UNLOCK=y
> CONFIG_INLINE_WRITE_UNLOCK_IRQ=y
> CONFIG_MUTEX_SPIN_ON_OWNER=y
> 
> CONFIG_TICK_ONESHOT=y
> CONFIG_NO_HZ=y
> CONFIG_HIGH_RES_TIMERS=y
> CONFIG_GENERIC_CLOCKEVENTS_BUILD=y
> CONFIG_SMP=y
> CONFIG_X86_SUPPORTS_MEMORY_FAILURE=y
> CONFIG_SCHED_OMIT_FRAME_POINTER=y
> CONFIG_NO_BOOTMEM=y
> CONFIG_MK8=y
> CONFIG_X86_INTERNODE_CACHE_SHIFT=6
> CONFIG_X86_CMPXCHG=y
> CONFIG_CMPXCHG_LOCAL=y
> CONFIG_X86_L1_CACHE_SHIFT=6
> CONFIG_X86_XADD=y
> CONFIG_X86_WP_WORKS_OK=y
> CONFIG_X86_INTEL_USERCOPY=y
> CONFIG_X86_USE_PPRO_CHECKSUM=y
> CONFIG_X86_TSC=y
> CONFIG_X86_CMPXCHG64=y
> CONFIG_X86_CMOV=y
> CONFIG_X86_MINIMUM_CPU_FAMILY=64
> CONFIG_X86_DEBUGCTLMSR=y
> CONFIG_CPU_SUP_INTEL=y
> CONFIG_CPU_SUP_AMD=y
> CONFIG_CPU_SUP_CENTAUR=y
> CONFIG_HPET_TIMER=y
> CONFIG_HPET_EMULATE_RTC=y
> CONFIG_DMI=y
> CONFIG_GART_IOMMU=y
> CONFIG_AMD_IOMMU=y
> CONFIG_SWIOTLB=y
> CONFIG_IOMMU_HELPER=y
> CONFIG_IOMMU_API=y
> CONFIG_NR_CPUS=8
> CONFIG_SCHED_MC=y
> CONFIG_PREEMPT_NONE=y
> CONFIG_X86_LOCAL_APIC=y
> CONFIG_X86_IO_APIC=y
> CONFIG_X86_MCE=y
> CONFIG_X86_MCE_AMD=y
> CONFIG_X86_MCE_THRESHOLD=y
> CONFIG_ARCH_PHYS_ADDR_T_64BIT=y
> CONFIG_ARCH_DMA_ADDR_T_64BIT=y
> CONFIG_DIRECT_GBPAGES=y
> CONFIG_ARCH_SPARSEMEM_DEFAULT=y
> CONFIG_ARCH_SPARSEMEM_ENABLE=y
> CONFIG_ARCH_SELECT_MEMORY_MODEL=y
> CONFIG_ARCH_MEMORY_PROBE=y
> CONFIG_ILLEGAL_POINTER_VALUE=0xdead000000000000
> CONFIG_SELECT_MEMORY_MODEL=y
> CONFIG_SPARSEMEM_MANUAL=y
> CONFIG_SPARSEMEM=y
> CONFIG_HAVE_MEMORY_PRESENT=y
> CONFIG_SPARSEMEM_EXTREME=y
> CONFIG_SPARSEMEM_VMEMMAP_ENABLE=y
> CONFIG_SPARSEMEM_ALLOC_MEM_MAP_TOGETHER=y
> CONFIG_SPARSEMEM_VMEMMAP=y
> CONFIG_HAVE_MEMBLOCK=y
> CONFIG_MEMORY_HOTPLUG=y
> CONFIG_MEMORY_HOTPLUG_SPARSE=y
> CONFIG_PAGEFLAGS_EXTENDED=y
> CONFIG_SPLIT_PTLOCK_CPUS=4
> CONFIG_COMPACTION=y
> CONFIG_MIGRATION=y
> CONFIG_PHYS_ADDR_T_64BIT=y
> CONFIG_ZONE_DMA_FLAG=1
> CONFIG_BOUNCE=y
> CONFIG_VIRT_TO_BUS=y
> CONFIG_MMU_NOTIFIER=y
> CONFIG_KSM=y
> CONFIG_DEFAULT_MMAP_MIN_ADDR=4096
> CONFIG_ARCH_SUPPORTS_MEMORY_FAILURE=y
> CONFIG_TRANSPARENT_HUGEPAGE=y
> CONFIG_TRANSPARENT_HUGEPAGE_ALWAYS=y
> CONFIG_X86_RESERVE_LOW=64
> CONFIG_MTRR=y
> CONFIG_MTRR_SANITIZER=y
> CONFIG_MTRR_SANITIZER_ENABLE_DEFAULT=0
> CONFIG_MTRR_SANITIZER_SPARE_REG_NR_DEFAULT=1
> CONFIG_X86_PAT=y
> CONFIG_ARCH_USES_PG_UNCACHED=y
> CONFIG_SECCOMP=y
> CONFIG_HZ_100=y
> CONFIG_HZ=100
> CONFIG_SCHED_HRTICK=y
> CONFIG_PHYSICAL_START=0x1000000
> CONFIG_PHYSICAL_ALIGN=0x1000000
> CONFIG_ARCH_ENABLE_MEMORY_HOTPLUG=y
> CONFIG_ARCH_ENABLE_MEMORY_HOTREMOVE=y
> 
> CONFIG_ACPI=y
> CONFIG_ACPI_BUTTON=y
> CONFIG_ACPI_DOCK=y
> CONFIG_ACPI_PROCESSOR=y
> CONFIG_ACPI_THERMAL=y
> CONFIG_ACPI_BLACKLIST_YEAR=0
> CONFIG_X86_PM_TIMER=y
> 
> CONFIG_CPU_FREQ=y
> CONFIG_CPU_FREQ_TABLE=y
> CONFIG_CPU_FREQ_STAT=y
> CONFIG_CPU_FREQ_STAT_DETAILS=y
> CONFIG_CPU_FREQ_DEFAULT_GOV_ONDEMAND=y
> CONFIG_CPU_FREQ_GOV_PERFORMANCE=y
> CONFIG_CPU_FREQ_GOV_POWERSAVE=y
> CONFIG_CPU_FREQ_GOV_USERSPACE=y
> CONFIG_CPU_FREQ_GOV_ONDEMAND=y
> CONFIG_CPU_FREQ_GOV_CONSERVATIVE=y
> 
> CONFIG_X86_ACPI_CPUFREQ=m
> CONFIG_X86_POWERNOW_K8=m
> 
> CONFIG_CPU_IDLE=y
> CONFIG_CPU_IDLE_GOV_LADDER=y
> CONFIG_CPU_IDLE_GOV_MENU=y
> 
> 
> CONFIG_PCI=y
> CONFIG_PCI_DIRECT=y
> CONFIG_PCI_MMCONFIG=y
> CONFIG_PCI_DOMAINS=y
> CONFIG_PCIEPORTBUS=y
> CONFIG_PCIEAER=y
> CONFIG_PCIEASPM=y
> CONFIG_ARCH_SUPPORTS_MSI=y
> CONFIG_PCI_MSI=y
> CONFIG_HT_IRQ=y
> CONFIG_PCI_IOAPIC=y
> CONFIG_PCI_LABEL=y
> CONFIG_ISA_DMA_API=y
> CONFIG_AMD_NB=y
> 
> CONFIG_BINFMT_ELF=y
> CONFIG_COMPAT_BINFMT_ELF=y
> CONFIG_BINFMT_MISC=m
> CONFIG_IA32_EMULATION=y
> CONFIG_IA32_AOUT=m
> CONFIG_COMPAT=y
> CONFIG_COMPAT_FOR_U64_ALIGNMENT=y
> CONFIG_SYSVIPC_COMPAT=y
> CONFIG_HAVE_TEXT_POKE_SMP=y
> CONFIG_NET=y
> 
> CONFIG_PACKET=y
> CONFIG_UNIX=y
> CONFIG_XFRM=y
> CONFIG_XFRM_MIGRATE=y
> CONFIG_XFRM_IPCOMP=m
> CONFIG_NET_KEY=m
> CONFIG_NET_KEY_MIGRATE=y
> CONFIG_INET=y
> CONFIG_IP_MULTICAST=y
> CONFIG_IP_ROUTE_CLASSID=y
> CONFIG_NET_IPGRE_DEMUX=m
> CONFIG_INET_AH=m
> CONFIG_INET_ESP=m
> CONFIG_INET_IPCOMP=m
> CONFIG_INET_XFRM_TUNNEL=m
> CONFIG_INET_TUNNEL=m
> CONFIG_INET_XFRM_MODE_TRANSPORT=m
> CONFIG_INET_XFRM_MODE_TUNNEL=m
> CONFIG_INET_XFRM_MODE_BEET=m
> CONFIG_INET_DIAG=y
> CONFIG_INET_TCP_DIAG=y
> CONFIG_TCP_CONG_CUBIC=y
> CONFIG_DEFAULT_TCP_CONG="cubic"
> CONFIG_NETFILTER=y
> CONFIG_NETFILTER_ADVANCED=y
> CONFIG_BRIDGE_NETFILTER=y
> 
> CONFIG_NETFILTER_NETLINK=m
> CONFIG_NETFILTER_NETLINK_QUEUE=m
> CONFIG_NETFILTER_NETLINK_LOG=m
> CONFIG_NF_CONNTRACK=m
> CONFIG_NF_CONNTRACK_MARK=y
> CONFIG_NF_CONNTRACK_ZONES=y
> CONFIG_NF_CT_PROTO_DCCP=m
> CONFIG_NF_CT_PROTO_GRE=m
> CONFIG_NF_CT_PROTO_SCTP=m
> CONFIG_NF_CT_PROTO_UDPLITE=m
> CONFIG_NF_CONNTRACK_AMANDA=m
> CONFIG_NF_CONNTRACK_FTP=m
> CONFIG_NF_CONNTRACK_H323=m
> CONFIG_NF_CONNTRACK_IRC=m
> CONFIG_NF_CONNTRACK_BROADCAST=m
> CONFIG_NF_CONNTRACK_NETBIOS_NS=m
> CONFIG_NF_CONNTRACK_SNMP=m
> CONFIG_NF_CONNTRACK_PPTP=m
> CONFIG_NF_CONNTRACK_SANE=m
> CONFIG_NF_CONNTRACK_SIP=m
> CONFIG_NF_CONNTRACK_TFTP=m
> CONFIG_NF_CT_NETLINK=m
> CONFIG_NETFILTER_TPROXY=m
> CONFIG_NETFILTER_XTABLES=m
> 
> CONFIG_NETFILTER_XT_MARK=m
> CONFIG_NETFILTER_XT_CONNMARK=m
> 
> CONFIG_NETFILTER_XT_TARGET_CHECKSUM=m
> CONFIG_NETFILTER_XT_TARGET_CLASSIFY=m
> CONFIG_NETFILTER_XT_TARGET_CONNMARK=m
> CONFIG_NETFILTER_XT_TARGET_CT=m
> CONFIG_NETFILTER_XT_TARGET_DSCP=m
> CONFIG_NETFILTER_XT_TARGET_HL=m
> CONFIG_NETFILTER_XT_TARGET_IDLETIMER=m
> CONFIG_NETFILTER_XT_TARGET_MARK=m
> CONFIG_NETFILTER_XT_TARGET_NFLOG=m
> CONFIG_NETFILTER_XT_TARGET_NFQUEUE=m
> CONFIG_NETFILTER_XT_TARGET_NOTRACK=m
> CONFIG_NETFILTER_XT_TARGET_RATEEST=m
> CONFIG_NETFILTER_XT_TARGET_TEE=m
> CONFIG_NETFILTER_XT_TARGET_TPROXY=m
> CONFIG_NETFILTER_XT_TARGET_TRACE=m
> CONFIG_NETFILTER_XT_TARGET_TCPMSS=m
> CONFIG_NETFILTER_XT_TARGET_TCPOPTSTRIP=m
> 
> CONFIG_NETFILTER_XT_MATCH_ADDRTYPE=m
> CONFIG_NETFILTER_XT_MATCH_CLUSTER=m
> CONFIG_NETFILTER_XT_MATCH_COMMENT=m
> CONFIG_NETFILTER_XT_MATCH_CONNBYTES=m
> CONFIG_NETFILTER_XT_MATCH_CONNLIMIT=m
> CONFIG_NETFILTER_XT_MATCH_CONNMARK=m
> CONFIG_NETFILTER_XT_MATCH_CONNTRACK=m
> CONFIG_NETFILTER_XT_MATCH_CPU=m
> CONFIG_NETFILTER_XT_MATCH_DCCP=m
> CONFIG_NETFILTER_XT_MATCH_DEVGROUP=m
> CONFIG_NETFILTER_XT_MATCH_DSCP=m
> CONFIG_NETFILTER_XT_MATCH_ESP=m
> CONFIG_NETFILTER_XT_MATCH_HASHLIMIT=m
> CONFIG_NETFILTER_XT_MATCH_HELPER=m
> CONFIG_NETFILTER_XT_MATCH_HL=m
> CONFIG_NETFILTER_XT_MATCH_IPRANGE=m
> CONFIG_NETFILTER_XT_MATCH_LENGTH=m
> CONFIG_NETFILTER_XT_MATCH_LIMIT=m
> CONFIG_NETFILTER_XT_MATCH_MAC=m
> CONFIG_NETFILTER_XT_MATCH_MARK=m
> CONFIG_NETFILTER_XT_MATCH_MULTIPORT=m
> CONFIG_NETFILTER_XT_MATCH_OSF=m
> CONFIG_NETFILTER_XT_MATCH_OWNER=m
> CONFIG_NETFILTER_XT_MATCH_POLICY=m
> CONFIG_NETFILTER_XT_MATCH_PHYSDEV=m
> CONFIG_NETFILTER_XT_MATCH_PKTTYPE=m
> CONFIG_NETFILTER_XT_MATCH_QUOTA=m
> CONFIG_NETFILTER_XT_MATCH_RATEEST=m
> CONFIG_NETFILTER_XT_MATCH_REALM=m
> CONFIG_NETFILTER_XT_MATCH_RECENT=m
> CONFIG_NETFILTER_XT_MATCH_SCTP=m
> CONFIG_NETFILTER_XT_MATCH_SOCKET=m
> CONFIG_NETFILTER_XT_MATCH_STATE=m
> CONFIG_NETFILTER_XT_MATCH_STATISTIC=m
> CONFIG_NETFILTER_XT_MATCH_STRING=m
> CONFIG_NETFILTER_XT_MATCH_TCPMSS=m
> CONFIG_NETFILTER_XT_MATCH_TIME=m
> CONFIG_NETFILTER_XT_MATCH_U32=m
> 
> CONFIG_NF_DEFRAG_IPV4=m
> CONFIG_NF_CONNTRACK_IPV4=m
> CONFIG_NF_CONNTRACK_PROC_COMPAT=y
> CONFIG_IP_NF_QUEUE=m
> CONFIG_IP_NF_IPTABLES=m
> CONFIG_IP_NF_MATCH_AH=m
> CONFIG_IP_NF_MATCH_ECN=m
> CONFIG_IP_NF_MATCH_TTL=m
> CONFIG_IP_NF_FILTER=m
> CONFIG_IP_NF_TARGET_REJECT=m
> CONFIG_IP_NF_TARGET_LOG=m
> CONFIG_IP_NF_TARGET_ULOG=m
> CONFIG_NF_NAT=m
> CONFIG_NF_NAT_NEEDED=y
> CONFIG_IP_NF_TARGET_MASQUERADE=m
> CONFIG_IP_NF_TARGET_NETMAP=m
> CONFIG_IP_NF_TARGET_REDIRECT=m
> CONFIG_NF_NAT_SNMP_BASIC=m
> CONFIG_NF_NAT_PROTO_DCCP=m
> CONFIG_NF_NAT_PROTO_GRE=m
> CONFIG_NF_NAT_PROTO_UDPLITE=m
> CONFIG_NF_NAT_PROTO_SCTP=m
> CONFIG_NF_NAT_FTP=m
> CONFIG_NF_NAT_IRC=m
> CONFIG_NF_NAT_TFTP=m
> CONFIG_NF_NAT_AMANDA=m
> CONFIG_NF_NAT_PPTP=m
> CONFIG_NF_NAT_H323=m
> CONFIG_NF_NAT_SIP=m
> CONFIG_IP_NF_MANGLE=m
> CONFIG_IP_NF_TARGET_CLUSTERIP=m
> CONFIG_IP_NF_TARGET_ECN=m
> CONFIG_IP_NF_TARGET_TTL=m
> CONFIG_IP_NF_RAW=m
> CONFIG_IP_NF_ARPTABLES=m
> CONFIG_IP_NF_ARPFILTER=m
> CONFIG_IP_NF_ARP_MANGLE=m
> CONFIG_IP_SCTP=m
> CONFIG_SCTP_HMAC_MD5=y
> CONFIG_STP=y
> CONFIG_BRIDGE=y
> CONFIG_BRIDGE_IGMP_SNOOPING=y
> CONFIG_VLAN_8021Q=y
> CONFIG_LLC=y
> CONFIG_NET_SCHED=y
> 
> CONFIG_NET_SCH_CBQ=m
> CONFIG_NET_SCH_HTB=m
> CONFIG_NET_SCH_HFSC=m
> CONFIG_NET_SCH_PRIO=m
> CONFIG_NET_SCH_RED=m
> CONFIG_NET_SCH_SFB=m
> CONFIG_NET_SCH_SFQ=m
> CONFIG_NET_SCH_TEQL=m
> CONFIG_NET_SCH_TBF=m
> CONFIG_NET_SCH_GRED=m
> CONFIG_NET_SCH_DSMARK=m
> CONFIG_NET_SCH_NETEM=m
> CONFIG_NET_SCH_MQPRIO=m
> CONFIG_NET_SCH_CHOKE=m
> CONFIG_NET_SCH_INGRESS=m
> 
> CONFIG_NET_CLS=y
> CONFIG_NET_CLS_BASIC=m
> CONFIG_NET_CLS_TCINDEX=m
> CONFIG_NET_CLS_ROUTE4=m
> CONFIG_NET_CLS_FW=m
> CONFIG_NET_CLS_U32=m
> CONFIG_NET_CLS_FLOW=m
> CONFIG_NET_EMATCH=y
> CONFIG_NET_EMATCH_STACK=32
> CONFIG_NET_EMATCH_CMP=m
> CONFIG_NET_EMATCH_NBYTE=m
> CONFIG_NET_EMATCH_U32=m
> CONFIG_NET_EMATCH_META=m
> CONFIG_NET_EMATCH_TEXT=m
> CONFIG_NET_CLS_ACT=y
> CONFIG_NET_ACT_POLICE=m
> CONFIG_NET_ACT_GACT=m
> CONFIG_GACT_PROB=y
> CONFIG_NET_ACT_MIRRED=m
> CONFIG_NET_ACT_IPT=m
> CONFIG_NET_ACT_NAT=m
> CONFIG_NET_ACT_PEDIT=m
> CONFIG_NET_ACT_SIMP=m
> CONFIG_NET_ACT_SKBEDIT=m
> CONFIG_NET_ACT_CSUM=m
> CONFIG_NET_SCH_FIFO=y
> CONFIG_RPS=y
> CONFIG_RFS_ACCEL=y
> CONFIG_XPS=y
> 
> 
> 
> CONFIG_UEVENT_HELPER_PATH="/sbin/hotplug"
> CONFIG_DEVTMPFS=y
> CONFIG_STANDALONE=y
> CONFIG_PREVENT_FIRMWARE_BUILD=y
> CONFIG_FW_LOADER=y
> CONFIG_FIRMWARE_IN_KERNEL=y
> CONFIG_EXTRA_FIRMWARE=""
> CONFIG_ARCH_NO_SYSDEV_OPS=y
> CONFIG_PNP=y
> CONFIG_PNP_DEBUG_MESSAGES=y
> 
> CONFIG_PNPACPI=y
> CONFIG_BLK_DEV=y
> CONFIG_BLK_DEV_LOOP=y
> 
> CONFIG_HAVE_IDE=y
> 
> CONFIG_SCSI_MOD=y
> CONFIG_RAID_ATTRS=m
> CONFIG_SCSI=y
> CONFIG_SCSI_DMA=y
> CONFIG_SCSI_PROC_FS=y
> 
> CONFIG_BLK_DEV_SD=y
> CONFIG_BLK_DEV_SR=y
> CONFIG_CHR_DEV_SG=y
> CONFIG_SCSI_WAIT_SCAN=m
> 
> CONFIG_SCSI_SAS_ATTRS=m
> CONFIG_SCSI_LOWLEVEL=y
> CONFIG_MEGARAID_SAS=m
> CONFIG_ATA=y
> CONFIG_ATA_VERBOSE_ERROR=y
> CONFIG_ATA_ACPI=y
> CONFIG_SATA_PMP=y
> 
> CONFIG_SATA_AHCI=m
> CONFIG_SATA_SIL24=m
> CONFIG_ATA_SFF=y
> 
> CONFIG_ATA_BMDMA=y
> 
> CONFIG_SATA_MV=m
> CONFIG_SATA_NV=m
> CONFIG_SATA_PROMISE=m
> CONFIG_SATA_SIL=m
> 
> 
> 
> CONFIG_MD=y
> CONFIG_BLK_DEV_MD=y
> CONFIG_MD_LINEAR=y
> CONFIG_MD_RAID0=y
> CONFIG_MD_RAID1=y
> CONFIG_MD_RAID10=y
> CONFIG_MD_RAID456=y
> CONFIG_BLK_DEV_DM=y
> 
> CONFIG_NETDEVICES=y
> CONFIG_DUMMY=m
> CONFIG_BONDING=m
> CONFIG_TUN=y
> CONFIG_MII=m
> CONFIG_NET_ETHERNET=y
> CONFIG_NET_PCI=y
> CONFIG_FORCEDETH=m
> CONFIG_NETDEV_1000=y
> CONFIG_R8169=m
> 
> 
> 
> CONFIG_PPP=m
> CONFIG_PPP_MULTILINK=y
> CONFIG_PPP_FILTER=y
> CONFIG_PPP_ASYNC=m
> CONFIG_PPP_SYNC_TTY=m
> CONFIG_PPP_DEFLATE=m
> CONFIG_PPP_BSDCOMP=m
> CONFIG_PPP_MPPE=m
> CONFIG_PPPOE=m
> CONFIG_SLHC=m
> CONFIG_NETCONSOLE=m
> CONFIG_NETCONSOLE_DYNAMIC=y
> CONFIG_NETPOLL=y
> CONFIG_NET_POLL_CONTROLLER=y
> 
> CONFIG_INPUT=y
> 
> CONFIG_INPUT_MOUSEDEV=y
> CONFIG_INPUT_MOUSEDEV_SCREEN_X=1024
> CONFIG_INPUT_MOUSEDEV_SCREEN_Y=768
> 
> CONFIG_INPUT_KEYBOARD=y
> CONFIG_KEYBOARD_ATKBD=y
> 
> CONFIG_SERIO=y
> CONFIG_SERIO_I8042=y
> CONFIG_SERIO_LIBPS2=y
> 
> CONFIG_VT=y
> CONFIG_CONSOLE_TRANSLATIONS=y
> CONFIG_VT_CONSOLE=y
> CONFIG_HW_CONSOLE=y
> CONFIG_UNIX98_PTYS=y
> CONFIG_LEGACY_PTYS=y
> CONFIG_LEGACY_PTY_COUNT=256
> 
> CONFIG_SERIAL_8250=y
> CONFIG_SERIAL_8250_CONSOLE=y
> CONFIG_FIX_EARLYCON_MEM=y
> CONFIG_SERIAL_8250_PCI=y
> CONFIG_SERIAL_8250_PNP=y
> CONFIG_SERIAL_8250_NR_UARTS=4
> CONFIG_SERIAL_8250_RUNTIME_UARTS=4
> 
> CONFIG_SERIAL_CORE=y
> CONFIG_SERIAL_CORE_CONSOLE=y
> CONFIG_RTC=y
> CONFIG_HPET=y
> CONFIG_HPET_MMAP=y
> CONFIG_DEVPORT=y
> CONFIG_I2C=y
> CONFIG_I2C_BOARDINFO=y
> CONFIG_I2C_COMPAT=y
> CONFIG_I2C_CHARDEV=y
> CONFIG_I2C_HELPER_AUTO=y
> CONFIG_I2C_ALGOBIT=y
> 
> 
> CONFIG_I2C_AMD756=m
> CONFIG_I2C_PIIX4=m
> CONFIG_I2C_NFORCE2=m
> CONFIG_I2C_VIA=m
> CONFIG_I2C_VIAPRO=m
> 
> CONFIG_I2C_SCMI=m
> 
> 
> 
> 
> 
> CONFIG_ARCH_WANT_OPTIONAL_GPIOLIB=y
> CONFIG_HWMON=y
> CONFIG_HWMON_VID=m
> 
> CONFIG_SENSORS_K8TEMP=m
> CONFIG_SENSORS_K10TEMP=m
> CONFIG_SENSORS_W83781D=m
> CONFIG_SENSORS_W83791D=m
> CONFIG_SENSORS_W83792D=m
> CONFIG_SENSORS_W83793=m
> CONFIG_SENSORS_W83795=m
> CONFIG_SENSORS_W83795_FANCTRL=y
> CONFIG_SENSORS_W83L785TS=m
> CONFIG_SENSORS_W83L786NG=m
> CONFIG_SENSORS_W83627HF=m
> CONFIG_SENSORS_W83627EHF=m
> 
> CONFIG_SENSORS_ATK0110=m
> CONFIG_THERMAL=y
> CONFIG_THERMAL_HWMON=y
> CONFIG_SSB_POSSIBLE=y
> 
> 
> CONFIG_VGA_ARB=y
> CONFIG_VGA_ARB_MAX_GPUS=16
> CONFIG_VGASTATE=y
> CONFIG_FB=y
> CONFIG_FB_DDC=y
> CONFIG_FB_CFB_FILLRECT=y
> CONFIG_FB_CFB_COPYAREA=y
> CONFIG_FB_CFB_IMAGEBLIT=y
> CONFIG_FB_MODE_HELPERS=y
> 
> CONFIG_FB_NVIDIA=y
> CONFIG_FB_NVIDIA_I2C=y
> CONFIG_FB_RADEON=y
> CONFIG_FB_RADEON_I2C=y
> 
> 
> CONFIG_VGA_CONSOLE=y
> CONFIG_DUMMY_CONSOLE=y
> CONFIG_FRAMEBUFFER_CONSOLE=y
> CONFIG_FONT_8x8=y
> CONFIG_FONT_8x16=y
> CONFIG_LOGO=y
> CONFIG_LOGO_LINUX_MONO=y
> CONFIG_LOGO_LINUX_VGA16=y
> CONFIG_LOGO_LINUX_CLUT224=y
> CONFIG_HID_SUPPORT=y
> CONFIG_HID=y
> CONFIG_HIDRAW=y
> 
> CONFIG_USB_HID=m
> CONFIG_USB_HIDDEV=y
> 
> CONFIG_HID_A4TECH=m
> CONFIG_HID_APPLE=m
> CONFIG_HID_BELKIN=m
> CONFIG_HID_CHERRY=m
> CONFIG_HID_CHICONY=m
> CONFIG_HID_CYPRESS=m
> CONFIG_HID_EZKEY=m
> CONFIG_HID_KYE=m
> CONFIG_HID_KENSINGTON=m
> CONFIG_HID_LOGITECH=m
> CONFIG_HID_MICROSOFT=m
> CONFIG_HID_MONTEREY=m
> CONFIG_USB_SUPPORT=y
> CONFIG_USB_ARCH_HAS_HCD=y
> CONFIG_USB_ARCH_HAS_OHCI=y
> CONFIG_USB_ARCH_HAS_EHCI=y
> CONFIG_USB=m
> 
> 
> CONFIG_USB_XHCI_HCD=m
> CONFIG_USB_EHCI_HCD=m
> CONFIG_USB_OHCI_HCD=m
> CONFIG_USB_OHCI_LITTLE_ENDIAN=y
> CONFIG_USB_UHCI_HCD=m
> 
> 
> CONFIG_USB_ACM=m
> 
> 
> CONFIG_USB_STORAGE=m
> CONFIG_USB_LIBUSUAL=y
> 
> 
> CONFIG_USB_SERIAL=m
> CONFIG_USB_SERIAL_GENERIC=y
> CONFIG_USB_SERIAL_FTDI_SIO=m
> CONFIG_USB_SERIAL_IPAQ=m
> CONFIG_USB_SERIAL_PL2303=m
> 
> 
> 
> CONFIG_FIRMWARE_MEMMAP=y
> CONFIG_DMIID=y
> 
> CONFIG_EXT2_FS=y
> CONFIG_EXT2_FS_XATTR=y
> CONFIG_EXT3_FS=y
> CONFIG_EXT3_FS_XATTR=y
> CONFIG_EXT4_FS=y
> CONFIG_EXT4_FS_XATTR=y
> CONFIG_JBD=y
> CONFIG_JBD2=y
> CONFIG_FS_MBCACHE=y
> CONFIG_EXPORTFS=y
> CONFIG_FILE_LOCKING=y
> CONFIG_FSNOTIFY=y
> CONFIG_DNOTIFY=y
> CONFIG_INOTIFY_USER=y
> CONFIG_FANOTIFY=y
> CONFIG_FUSE_FS=m
> CONFIG_CUSE=m
> 
> CONFIG_FSCACHE=m
> CONFIG_FSCACHE_DEBUG=y
> CONFIG_CACHEFILES=m
> 
> CONFIG_ISO9660_FS=m
> CONFIG_JOLIET=y
> CONFIG_ZISOFS=y
> CONFIG_UDF_FS=m
> CONFIG_UDF_NLS=y
> 
> CONFIG_FAT_FS=m
> CONFIG_MSDOS_FS=m
> CONFIG_VFAT_FS=m
> CONFIG_FAT_DEFAULT_CODEPAGE=437
> CONFIG_FAT_DEFAULT_IOCHARSET="iso8859-1"
> CONFIG_NTFS_FS=m
> 
> CONFIG_PROC_FS=y
> CONFIG_PROC_SYSCTL=y
> CONFIG_PROC_PAGE_MONITOR=y
> CONFIG_SYSFS=y
> CONFIG_TMPFS=y
> CONFIG_HUGETLBFS=y
> CONFIG_HUGETLB_PAGE=y
> CONFIG_CONFIGFS_FS=m
> CONFIG_NETWORK_FILESYSTEMS=y
> CONFIG_NFS_FS=y
> CONFIG_NFS_V3=y
> CONFIG_NFSD=y
> CONFIG_NFSD_DEPRECATED=y
> CONFIG_NFSD_V3=y
> CONFIG_LOCKD=y
> CONFIG_LOCKD_V4=y
> CONFIG_NFS_COMMON=y
> CONFIG_SUNRPC=y
> 
> CONFIG_PARTITION_ADVANCED=y
> CONFIG_MSDOS_PARTITION=y
> CONFIG_EFI_PARTITION=y
> CONFIG_NLS=y
> CONFIG_NLS_DEFAULT="iso8859-1"
> CONFIG_NLS_CODEPAGE_437=y
> CONFIG_NLS_ISO8859_1=y
> CONFIG_DLM=m
> 
> CONFIG_TRACE_IRQFLAGS_SUPPORT=y
> CONFIG_PRINTK_TIME=y
> CONFIG_DEFAULT_MESSAGE_LOGLEVEL=4
> CONFIG_ENABLE_WARN_DEPRECATED=y
> CONFIG_ENABLE_MUST_CHECK=y
> CONFIG_FRAME_WARN=2048
> CONFIG_MAGIC_SYSRQ=y
> CONFIG_DEBUG_FS=y
> CONFIG_DEBUG_KERNEL=y
> CONFIG_DETECT_HUNG_TASK=y
> CONFIG_BOOTPARAM_HUNG_TASK_PANIC_VALUE=0
> CONFIG_TIMER_STATS=y
> CONFIG_DEBUG_BUGVERBOSE=y
> CONFIG_DEBUG_MEMORY_INIT=y
> CONFIG_ARCH_WANT_FRAME_POINTERS=y
> CONFIG_SYSCTL_SYSCALL_CHECK=y
> CONFIG_USER_STACKTRACE_SUPPORT=y
> CONFIG_HAVE_FUNCTION_TRACER=y
> CONFIG_HAVE_FUNCTION_GRAPH_TRACER=y
> CONFIG_HAVE_FUNCTION_GRAPH_FP_TEST=y
> CONFIG_HAVE_FUNCTION_TRACE_MCOUNT_TEST=y
> CONFIG_HAVE_DYNAMIC_FTRACE=y
> CONFIG_HAVE_FTRACE_MCOUNT_RECORD=y
> CONFIG_HAVE_SYSCALL_TRACEPOINTS=y
> CONFIG_HAVE_C_RECORDMCOUNT=y
> CONFIG_TRACING_SUPPORT=y
> CONFIG_HAVE_ARCH_KGDB=y
> CONFIG_HAVE_ARCH_KMEMCHECK=y
> CONFIG_X86_VERBOSE_BOOTUP=y
> CONFIG_EARLY_PRINTK=y
> CONFIG_HAVE_MMIOTRACE_SUPPORT=y
> CONFIG_IO_DELAY_TYPE_0X80=0
> CONFIG_IO_DELAY_TYPE_0XED=1
> CONFIG_IO_DELAY_TYPE_UDELAY=2
> CONFIG_IO_DELAY_TYPE_NONE=3
> CONFIG_IO_DELAY_0X80=y
> CONFIG_DEFAULT_IO_DELAY_TYPE=0
> 
> CONFIG_DEFAULT_SECURITY_DAC=y
> CONFIG_DEFAULT_SECURITY=""
> CONFIG_XOR_BLOCKS=y
> CONFIG_ASYNC_CORE=y
> CONFIG_ASYNC_MEMCPY=y
> CONFIG_ASYNC_XOR=y
> CONFIG_ASYNC_PQ=y
> CONFIG_ASYNC_RAID6_RECOV=y
> CONFIG_CRYPTO=y
> 
> CONFIG_CRYPTO_ALGAPI=m
> CONFIG_CRYPTO_ALGAPI2=m
> CONFIG_CRYPTO_AEAD=m
> CONFIG_CRYPTO_AEAD2=m
> CONFIG_CRYPTO_BLKCIPHER=m
> CONFIG_CRYPTO_BLKCIPHER2=m
> CONFIG_CRYPTO_HASH=m
> CONFIG_CRYPTO_HASH2=m
> CONFIG_CRYPTO_RNG2=m
> CONFIG_CRYPTO_PCOMP2=m
> CONFIG_CRYPTO_MANAGER=m
> CONFIG_CRYPTO_MANAGER2=m
> CONFIG_CRYPTO_MANAGER_DISABLE_TESTS=y
> CONFIG_CRYPTO_WORKQUEUE=m
> CONFIG_CRYPTO_AUTHENC=m
> 
> 
> CONFIG_CRYPTO_CBC=m
> CONFIG_CRYPTO_ECB=m
> 
> CONFIG_CRYPTO_HMAC=m
> 
> CONFIG_CRYPTO_CRC32C=m
> CONFIG_CRYPTO_MD5=m
> CONFIG_CRYPTO_SHA1=m
> 
> CONFIG_CRYPTO_ARC4=m
> CONFIG_CRYPTO_DES=m
> 
> CONFIG_CRYPTO_DEFLATE=m
> 
> CONFIG_HAVE_KVM=y
> CONFIG_HAVE_KVM_IRQCHIP=y
> CONFIG_HAVE_KVM_EVENTFD=y
> CONFIG_KVM_APIC_ARCHITECTURE=y
> CONFIG_KVM_MMIO=y
> CONFIG_KVM_ASYNC_PF=y
> CONFIG_VIRTUALIZATION=y
> CONFIG_KVM=m
> CONFIG_KVM_INTEL=m
> CONFIG_KVM_AMD=m
> CONFIG_VHOST_NET=m
> 
> CONFIG_RAID6_PQ=y
> CONFIG_BITREVERSE=y
> CONFIG_GENERIC_FIND_FIRST_BIT=y
> CONFIG_GENERIC_FIND_NEXT_BIT=y
> CONFIG_GENERIC_FIND_LAST_BIT=y
> CONFIG_CRC_CCITT=m
> CONFIG_CRC16=y
> CONFIG_CRC_T10DIF=y
> CONFIG_CRC_ITU_T=y
> CONFIG_CRC32=y
> CONFIG_LIBCRC32C=m
> CONFIG_ZLIB_INFLATE=y
> CONFIG_ZLIB_DEFLATE=m
> CONFIG_LZO_DECOMPRESS=y
> CONFIG_XZ_DEC=y
> CONFIG_XZ_DEC_X86=y
> CONFIG_XZ_DEC_POWERPC=y
> CONFIG_XZ_DEC_IA64=y
> CONFIG_XZ_DEC_ARM=y
> CONFIG_XZ_DEC_ARMTHUMB=y
> CONFIG_XZ_DEC_SPARC=y
> CONFIG_XZ_DEC_BCJ=y
> CONFIG_DECOMPRESS_GZIP=y
> CONFIG_DECOMPRESS_BZIP2=y
> CONFIG_DECOMPRESS_LZMA=y
> CONFIG_DECOMPRESS_XZ=y
> CONFIG_DECOMPRESS_LZO=y
> CONFIG_TEXTSEARCH=y
> CONFIG_TEXTSEARCH_KMP=m
> CONFIG_TEXTSEARCH_BM=m
> CONFIG_TEXTSEARCH_FSM=m
> CONFIG_HAS_IOMEM=y
> CONFIG_HAS_IOPORT=y
> CONFIG_HAS_DMA=y
> CONFIG_CPU_RMAP=y
> CONFIG_NLATTR=y
> 
> 00:00.0 Host bridge: Advanced Micro Devices [AMD] RS880 Host Bridge
> 00:01.0 PCI bridge: ASRock Incorporation Device 9602
> 00:02.0 PCI bridge: Advanced Micro Devices [AMD] RS780 PCI to PCI
> bridge (ext gfx port 0)
> 00:03.0 PCI bridge: Advanced Micro Devices [AMD] RS780 PCI to PCI
> bridge (ext gfx port 1)
> 00:04.0 PCI bridge: Advanced Micro Devices [AMD] RS780 PCI to PCI
> bridge (PCIE port 0)
> 00:09.0 PCI bridge: Advanced Micro Devices [AMD] RS780 PCI to PCI
> bridge (PCIE port 4)
> 00:0a.0 PCI bridge: Advanced Micro Devices [AMD] RS780 PCI to PCI
> bridge (PCIE port 5)
> 00:11.0 SATA controller: ATI Technologies Inc SB700/SB800 SATA
> Controller [AHCI mode] (rev 40)
> 00:12.0 USB Controller: ATI Technologies Inc SB700/SB800 USB OHCI0
> Controller
> 00:12.2 USB Controller: ATI Technologies Inc SB700/SB800 USB EHCI Controller
> 00:13.0 USB Controller: ATI Technologies Inc SB700/SB800 USB OHCI0
> Controller
> 00:13.2 USB Controller: ATI Technologies Inc SB700/SB800 USB EHCI Controller
> 00:14.0 SMBus: ATI Technologies Inc SBx00 SMBus Controller (rev 41)
> 00:14.2 Audio device: ATI Technologies Inc SBx00 Azalia (Intel HDA) (rev 40)
> 00:14.3 ISA bridge: ATI Technologies Inc SB700/SB800 LPC host
> controller (rev 40)
> 00:14.4 PCI bridge: ATI Technologies Inc SBx00 PCI to PCI Bridge (rev 40)
> 00:14.5 USB Controller: ATI Technologies Inc SB700/SB800 USB OHCI2
> Controller
> 00:16.0 USB Controller: ATI Technologies Inc SB700/SB800 USB OHCI0
> Controller
> 00:16.2 USB Controller: ATI Technologies Inc SB700/SB800 USB EHCI Controller
> 00:18.0 Host bridge: Advanced Micro Devices [AMD] Family 10h
> Processor HyperTransport Configuration
> 00:18.1 Host bridge: Advanced Micro Devices [AMD] Family 10h
> Processor Address Map
> 00:18.2 Host bridge: Advanced Micro Devices [AMD] Family 10h
> Processor DRAM Controller
> 00:18.3 Host bridge: Advanced Micro Devices [AMD] Family 10h
> Processor Miscellaneous Control
> 00:18.4 Host bridge: Advanced Micro Devices [AMD] Family 10h
> Processor Link Control
> 01:05.0 VGA compatible controller: ATI Technologies Inc RS880
> [Radeon HD 4250]
> 02:00.0 USB Controller: NEC Corporation uPD720200 USB 3.0 Host
> Controller (rev 03)
> 03:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd.
> RTL8111/8168B PCI Express Gigabit Ethernet controller (rev 06)
> 04:00.0 RAID bus controller: LSI Logic / Symbios Logic MegaRAID SAS
> 9240 (rev 02)
> 05:00.0 RAID bus controller: LSI Logic / Symbios Logic MegaRAID SAS
> 9240 (rev 02)
> 07:00.0 SCSI storage controller: Marvell Technology Group Ltd.
> 88SX7042 PCI-e 4-port SATA-II (rev 02)
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

-- 
Regards/Gruss,
    Boris.

^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
@ 2011-05-31  5:47   ` Borislav Petkov
  0 siblings, 0 replies; 111+ messages in thread
From: Borislav Petkov @ 2011-05-31  5:47 UTC (permalink / raw)
  To: Brad Campbell; +Cc: linux-kernel, kvm, linux-mm

Looks like a KSM issue. Disabling CONFIG_KSM should at least stop your
machine from oopsing.

Adding linux-mm.

On Tue, May 31, 2011 at 09:24:03AM +0800, Brad Campbell wrote:
> G'day all,
> 
> I'm running a pretty standard home server
> x86_64 Phenom-II 6 Core
> 16GB DDR 3
> 
> I run some virtual machines under that. 3 x Debian 64 Bit, 1 x XP 32
> Bit. These run at boot.
> 
> When I fire up another XP 32 bit instance and play with it for more
> than about 2 minutes, I get the panics included in this mail.
> 
> I've included three of them here. The first and third are as booted.
> The second was with ksmd disabled just as a data point.
> 
> The machine passes every load test and memory test I can throw at
> it, but I still can't rule out this being a hardware issue.
> 
> Provided I don't start this XP VM the machine is quite stable, but
> running this VM will kill it within minutes.
> 
> This was tested with qemu-kvm.
> The last commit in the git tree was
> commit c007db193eb6b2557acb5caf2dc4d7023639e6f3
> Author: Avi Kivity <avi@redhat.com>
> Date:   Sun May 29 09:00:42 2011 -0400
> (I pulled it yesterday)
> 
> These panics were captured with netconsole to a remote syslog
> daemon, and the formatting was ruined, so I've reformatted them by
> hand prior to posting.
> 
> I've tested and reproduced this on 2.6.38.[2,3,6 & 7] and 2.6.39 obviously.
> 
> Can anyone help shed some light on this?
> 
> Regards,
> Brad
> 
> [ 438.632061] general protection fault: 0000 [#1] SMP
> [ 438.632196] last sysfs file: /sys/module/x_tables/initstate
> [ 438.632242] CPU 4
> [ 438.632282] Modules linked in: xt_iprange xt_DSCP xt_length
> xt_CLASSIFY sch_sfq xt_CHECKSUM ipt_REJECT ipt_MASQUERADE
> ipt_REDIRECT xt_recent xt_state iptable_nat nf_nat nf_conntrack_ipv4
> nf_conntrack nf_defrag_ipv4 iptable_filter xt_TCPMSS xt_tcpmss
> xt_tcpudp iptable_mangle ip_tables x_tables pppoe pppox ppp_generic
> slhc cls_u32 sch_htb deflate zlib_deflate des_generic cbc ecb
> crypto_blkcipher sha1_generic md5 hmac crypto_hash cryptomgr aead
> crypto_algapi af_key fuse netconsole configfs= vhost_net powernow_k8
> mperf i2c_nforce2 kvm_amd kvm pl2303 usbserial xhci_hcd k10temp
> i2c_piix4 usb_storage usb_libusual ohci_hcd ehci_hcd usbcore ahci
> libahci r8169 mii sata_mv megaraid_sas [last unloaded:
> scsi_wait_scan]
> [ 438.634960]
> [ 438.635006] Pid: 551, comm: ksmd Not tainted 2.6.39 #3 To Be
> Filled By O.E.M. To Be Filled By O.E.M. /880G Extreme3
> [ 438.635170] RIP: 0010:[<ffffffff810b4596>] [<ffffffff810b4596>]
> remove_rmap_item_from_tree+0x96/0x150
> [ 438.635268] RSP: 0018:ffff88041c065e20  EFLAGS: 00010282
> [ 438.635314] RAX: ffff8804153bd8b0 RBX: ffff8804176c3fc0 RCX:
> 0000000000057754
> [ 438.635362] RDX: 0000880415418030 RSI: ffff880414b65003 RDI:
> ffffea000dde6030
> [ 438.635410] RBP: ffff880414b65000 R08: 0000000000057755 R09:
> 000000001bbde1bd
> [ 438.635458] R10: 000000001c28e0c3 R11: 0000000000000002 R12:
> ffffea000dde6030
> [ 438.635506] R13: ffff8804176c3f80 R14: ffff8804151ed7b0 R15:
> ffff88041bf23be0
> [ 438.635555] FS:  00007f617772e700(0000) GS:ffff88041fd00000(0000)
> knlGS:0000000000000000
> [ 438.635607] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [ 438.635654] CR2: 0000000000e70000 CR3: 0000000001583000 CR4:
> 00000000000006e0
> [ 438.635703] DR0: 0000000000000045 DR1: 0000000000000000 DR2:
> 0000000000000000
> [ 438.635750] DR3: 0000000000000005 DR6: 00000000ffff0ff0 DR7:
> 0000000000000400
> [ 438.635799] Process ksmd (pid: 551, threadinfo ffff88041c064000,
> task ffff88041d8caa70)
> [ 438.635851] Stack:
> [ 438.635893]  ffffea000c43c408 ffff8804176c3fc0 000000000000036c
> ffffffff810b58f2
> [ 438.636088]  ffff88041c782a00 00007fc2d81b5000 ffff88041c064000
> 00000000003808ff
> [ 438.636281]  ffff88041c064000 ffff88041c065e98 ffff88041d8caa70
> ffff8804165d4480
> [ 438.636479] Call Trace:
> [ 438.636528]  [<ffffffff810b58f2>] ? ksm_scan_thread+0x4e2/0xc20
> [ 438.636580]  [<ffffffff81052a20>] ? wake_up_bit+0x40/0x40
> [ 438.636628]  [<ffffffff810b5410>] ? try_to_merge_with_ksm_page+0x570/0x570
> [ 438.636679]  [<ffffffff810b5410>] ? try_to_merge_with_ksm_page+0x570/0x570
> [ 438.636730]  [<ffffffff810525b6>] ? kthread+0x96/0xa0
> [ 438.636781]  [<ffffffff813d1794>] ? kernel_thread_helper+0x4/0x10
> [ 438.636832]  [<ffffffff81052520>] ? kthread_worker_fn+0x120/0x120
> [ 438.636882]  [<ffffffff813d1790>] ? gs_change+0xb/0xb
> [ 438.636926] Code: 28 48 89 ef e8 6c fe ff ff 48 85 c0 49 89 c4 74
> d2 f0 0f ba 28 00 19 c0 85 c0 0f 85 ae 00 00 00 00 48 8b 43 30 48 8b
> 53 38 48 85 c0
> [ 438.638504] 89 02 74 04 48 89 50 08 48 ba 00 01 10 00 00 00 00 ad de 48 b8
> [ 438.639329] RIP [<ffffffff810b4596>] remove_rmap_item_from_tree+0x96/0x150
> [ 438.639414]  RSP <ffff88041c065e20>
> [ 438.639460] ---[ end trace c29fb871f6b874e3 ]---
> [ 438.639506] Kernel panic - not syncing: Fatal exception
> [ 438.639553] Pid: 551, comm: ksmd Tainted: G      D     2.6.39 #3
> [ 438.639598] Call Trace:
> [ 438.639644]  [<ffffffff813cd6f5>] ? panic+0x92/0x18a
> [ 438.639693]  [<ffffffff81038b61>] ? kmsg_dump+0x41/0xf0
> [ 438.639745]  [<ffffffff810050ad>] ? oops_end+0x8d/0xa0
> [ 438.639794]  [<ffffffff813d05ef>] ? general_protection+0x1f/0x30
> [ 438.639842]  [<ffffffff810b4596>] ? remove_rmap_item_from_tree+0x96/0x150
> [ 438.639891]  [<ffffffff810b58f2>] ? ksm_scan_thread+0x4e2/0xc20
> [ 438.639941]  [<ffffffff81052a20>] ? wake_up_bit+0x40/0x40
> [ 438.640045]  [<ffffffff810b5410>] ? try_to_merge_with_ksm_page+0x570/0x570
> [ 438.640096]  [<ffffffff810b5410>] ? try_to_merge_with_ksm_page+0x570/0x570
> [ 438.640147]  [<ffffffff810525b6>] ? kthread+0x96/0xa0
> [ 438.640196]  [<ffffffff813d1794>] ? kernel_thread_helper+0x4/0x10
> [ 438.640247]  [<ffffffff81052520>] ? kthread_worker_fn+0x120/0x120
> [ 438.640297]  [<ffffffff813d1790>] ? gs_change+0xb/0xb
> [ 438.640349] Rebooting in 60 seconds..
> 
> 
> 
> [ 469.706013] general protection fault: 0000 [#1] SMP
> [ 469.706125] last sysfs file:
> /sys/devices/pci0000:00/0000:00:18.3/temp1_input
> [ 469.706162] CPU 1
> [ 469.706195] Modules linked in: xt_iprange xt_DSCP xt_length
> xt_CLASSIFY sch_sfq xt_CHECKSUM ipt_REJECT ipt_MASQUERADE
> ipt_REDIRECT xt_recent xt_state iptable_nat nf_nat nf_conntrack_ipv4
> nf_conntrack nf_defrag_ipv4 iptable_filter xt_TCPMSS xt_tcpmss
> xt_tcpudp iptable_mangle ip_tables x_tables pppoe pppox ppp_generic
> slhc cls_u32 sch_htb deflate zlib_deflate des_generi cbc ecb
> crypto_blkcipher sha1_generic md5 hmac crypto_hash cryptomgr aead
> crypto_algapi af_key fuse netconsole configfs vhost_net powernow_k8
> mperf i2c_nforce2 kvm_amd kvm pl2303 usbserial xhci_hcd i2c_piix4
> k10temp ahci usb_storage usb_libusual ohci_hcd ehci_hcd r8169
> libahci usbcore mii sata_mv megaraid_sas [last unloaded:
> scsi_wait_scan]
> [ 469.708573]
> [ 469.708608] Pid: 4942, comm: runbot Not tainted 2.6.39 #3 To Be
> Filled By O.E.M. To Be Filled By O.E.M. /880G Extreme3
> [ 469.708753] RIP: 0010:[<ffffffff810db878>] [<ffffffff810db878>]
> dup_fd+0x168/0x300
> [ 469.708829] RSP: 0018:ffff880417279dd0  EFLAGS: 00010202
> [ 469.708870] RAX: 00000000000007f8 RBX: ffff88041c1658c0 RCX:
> bfffffffffffffff
> [ 469.708907] RDX: 0000880413155540 RSI: 00000000000000ff RDI:
> 0000000000000800
> [ 469.708945] RBP: ffff88041bf22000 R08: ffff880412839dc0 R09:
> 0000000000000003
> [ 469.708981] R10: 0000000000000001 R11: 4000000000000000 R12:
> 0000000000000100
> [ 469.709018] R13: ffff88041d3cc000 R14: ffff880412839d40 R15:
> ffff88041c72f440
> [ 469.709054] FS:  00007f510c0b4700(0000) GS:ffff88041fc40000(0000)
> knlGS:0000000000000000
> [ 469.709094] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [ 469.709130] CR2: 00007f510ba56810 CR3: 0000000417c93000 CR4:
> 00000000000006e0
> [ 469.709167] DR0: 00000000000000a0 DR1: 0000000000000000 DR2:
> 0000000000000003
> [ 469.709204] DR3: 00000000000000b0 DR6: 00000000ffff0ff0 DR7:
> 0000000000000400
> [ 469.709240] Process runbot (pid: 4942, threadinfo
> ffff880417278000, task ffff880401351230)
> [ 469.709279] Stack:
> [ 469.709313]  ffff880401351230 0000000000000020 ffff880412e9ca80
> ffff88041c1658d0 ffff88041c1658d0
> [ 469.709477]  ffff88041723f330 0000000001200011 ffff88041723f330
> 0000000000000000
> [ 469.709641]  00007f510c0b49d0 ffff880401351230 0000000000000000
> ffffffff81037645
> [ 469.709812] Call Trace:
> [ 469.709849]  [<ffffffff81037645>] ? copy_process+0xa75/0xfd0
> [ 469.709885]  [<ffffffff81037c0d>] ? do_fork+0x6d/0x2b0
> [ 469.709921]  [<ffffffff810dbc23>] ? alloc_fd+0x43/0x130
> [ 469.709958]  [<ffffffff810c08c5>] ? fd_install+0x35/0x70
> [ 469.709998]  [<ffffffff810457a9>] ? sigprocmask+0x69/0x100
> [ 469.710042]  [<ffffffff813d0ca3>] ? stub_clone+0x13/0x20
> [ 469.710079]  [<ffffffff813d0a3b>] ? system_call_fastpath+0x16/0x1b
> [ 469.710114] Code:  4c 89 c2 e8 6b e5 0f 00 45 85 e4 74 78 41 8d 44
> 24 ff 31 f6 41 ba 01 00 00 00 00 48 8d 3c c5 08 00 00 00 00 31 c0 eb
> 1a 0f 1f 44 00 00 f0> 48 ff 42 30 48 89 54 05 00 48 83 c0
> May 30 16:39:36 192.168.2.1 08 ff c6 48 39 f8 74 3b
> [ 469.712247] RIP [<ffffffff810db878>] dup_fd+0x168/0x300
> [ 469.712317]  RSP <ffff880417279dd0>
> [ 469.712391] ---[ end trace b80596608cd6bb6b ]---
> [ 469.712432] Kernel panic - not syncing: Fatal exception
> [ 469.712470] Pid: 4942, comm: runbot Tainted: G      D     2.6.39 #3
> [ 469.712508] Call Trace:
> [ 469.712544]  [<ffffffff813cd6f5>] ? panic+0x92/0x18a
> [ 469.712581]  [<ffffffff81038b61>] ? kmsg_dump+0x41/0xf0
> [ 469.712618]  [<ffffffff810050ad>] ? oops_end+0x8d/0xa0
> [ 469.712654]  [<ffffffff813d05ef>] ? general_protection+0x1f/0x30
> [ 469.712691]  [<ffffffff810db878>] ? dup_fd+0x168/0x300
> [ 469.712727]  [<ffffffff81037645>] ? copy_process+0xa75/0xfd0
> [ 469.712767]  [<ffffffff81037c0d>] ? do_fork+0x6d/0x2b0
> [ 469.712803]  [<ffffffff810dbc23>] ? alloc_fd+0x43/0x130
> [ 469.712839]  [<ffffffff810c08c5>] ? fd_install+0x35/0x70
> [ 469.712875]  [<ffffffff810457a9>] ? sigprocmask+0x69/0x100
> [ 469.712912]  [<ffffffff813d0ca3>] ? stub_clone+0x13/0x20
> [ 469.712948]  [<ffffffff813d0a3b>] ? system_call_fastpath+0x16/0x1b
> [ 469.712989] Rebooting in 60 seconds..
> 
> 
> [ 1100.473612] general protection fault: 0000 [#1] SMP
> [ 1100.473718] last sysfs file:
> /sys/devices/pci0000:00/0000:00:18.3/temp1_input
> [ 1100.473755] CPU 0
> [ 1100.473788] Modules linked in: xt_iprange xt_DSCP xt_length
> xt_CLASSIFY sch_sfq xt_CHECKSUM ipt_REJECT ipt_MASQUERADE
> ipt_REDIRECT xt_recent xt_state iptable_nat nf_nat nf_conntrack_ipv4
> nf_conntrack nf_defrag_ipv4 xt_TCPMSS xt_tcpmss xt_tcpudp
> iptable_mangle pppoe pppox iptable_filter ip_tables x_tables
> ppp_generic slhc cls_u32 sch_htb deflate zlib_deflate des_generic
> cbc ecb crypto_blkcipher sha1_generic md5 hmac crypto_hash cryptomgr
> aead crypto_algapi af_key fuse netconsole configfs vhost_net
> powernow_k8 mperf i2c_nforce2 kvm_amd kvm pl2303 xhci_hcd usbserial
> k10temp i2c_piix4 usb_storage usb_libusual ohci_hcd ehci_hcd ahci
> usbcore sata_mv r8169 mii libahci megaraid_sas [last unloaded:
> scsi_wait_scan]
> [ 1100.476102]
> [ 1100.476137] Pid: 551, comm: ksmd Not tainted 2.6.39 #3 To Be
> Filled By O.E.M. To Be Filled By O.E.M. /880G Extreme3
> [ 1100.476277] RIP: 0010:[<ffffffff810b5c71>] [<ffffffff810b5c71>]
> ksm_scan_thread+0x861/0xc20
> [ 1100.476351] RSP: 0018:ffff88041c065e40  EFLAGS: 00010282
> [ 1100.476387] RAX: 0000000000000000 RBX: ffffea000a5c8468 RCX:
> 0000000000000f01
> [ 1100.476424] RDX: 0000880403456828 RSI: ffff8802f5dcb002 RDI:
> ffffea000ce7eac8
> [ 1100.476460] RBP: ffff880418377fc0 R08: 6db6db6db6db6db7 R09:
> 0000160000000000
> [ 1100.476498] R10: ffffea000ce7eac8 R11: 0000000080100073 R12:
> 00000000000000ab
> [ 1100.476535] R13: ffff880418377ff8 R14: ffffea000ce7eac8 R15:
> ffff880418377fe8
> [ 1100.476572] FS:  00007f08c6498700(0000) GS:ffff88041fc00000(0000)
> knlGS:0000000000000000
> [ 1100.476611] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [ 1100.476646] CR2: 00000000e1dd3000 CR3: 00000004025f7000 CR4:
> 00000000000006f0
> [ 1100.477007] DR0: 0000000000000045 DR1: 0000000000000000 DR2:
> 0000000000000000
> [ 1100.477042] DR3: 0000000000000005 DR6: 00000000ffff0ff0 DR7:
> 0000000000000400
> [ 1100.477078] Process ksmd (pid: 551, threadinfo ffff88041c064000,
> task ffff88041d0454e0)
> [ 1100.477115] Stack:
> [ 1100.477148]  ffff880403a8b100 00007fc1863cb000 ffff88041c064000
> 00000000002f5dcb
> [ 1100.477317]  ffff88041c064000 ffff88041c065e98 ffff88041d0454e0
> ffff88041267a580
> [ 1100.477480]  0000000000000000 ffff88041d0454e0 ffffffff81052a20
> ffff88041c065e98
> [ 1100.477644] Call Trace:
> [ 1100.477681]  [<ffffffff81052a20>] ? wake_up_bit+0x40/0x40
> [ 1100.477717]  [<ffffffff810b5410>] ?
> try_to_merge_with_ksm_page+0x570/0x570
> [ 1100.477754]  [<ffffffff810b5410>] ?
> try_to_merge_with_ksm_page+0x570/0x570
> [ 1100.477791]  [<ffffffff810525b6>] ? kthread+0x96/0xa0
> [ 1100.477829]  [<ffffffff813d1794>] ? kernel_thread_helper+0x4/0x10
> [ 1100.477865]  [<ffffffff81052520>] ? kthread_worker_fn+0x120/0x120
> [ 1100.477902]  [<ffffffff813d1790>] ? gs_change+0xb/0xb
> [ 1100.477937] Code: 05 99 6e 6e 00 4c 89 ff e8 ed 12 fa ff e9 6c f9
> ff ff 48 89 6c 24 38 49 83 7d 00 00 0f 84 63 01 00 00 e8 b3 8b 31 00
> 49 8b 55 00
> [ 1100.479306] 8b 42 e8 4c 8b 72 f0 48 8d 6a d8 4c 8d 78 60 48 89 04 24 4c
> [ 1100.480012] RIP [<ffffffff810b5c71>] ksm_scan_thread+0x861/0xc20
> [ 1100.480015]  RSP <ffff88041c065e40>
> [ 1100.480141] ---[ end trace 7382777d9e74ea23 ]---
> [ 1100.480178] Kernel panic - not syncing: Fatal exception
> [ 1100.480216] Pid: 551, comm: ksmd Tainted: G      D     2.6.39 #3
> [ 1100.480257] Call Trace:
> [ 1100.480297]  [<ffffffff813cd6f5>] ? panic+0x92/0x18a
> [ 1100.480340]  [<ffffffff81038b61>] ? kmsg_dump+0x41/0xf0
> [ 1100.480384]  [<ffffffff810050ad>] ? oops_end+0x8d/0xa0
> [ 1100.480425]  [<ffffffff813d05ef>] ? general_protection+0x1f/0x30
> [ 1100.480465]  [<ffffffff810b5c71>] ? ksm_scan_thread+0x861/0xc20
> [ 1100.480507]  [<ffffffff81052a20>] ? wake_up_bit+0x40/0x40
> [ 1100.480545]  [<ffffffff810b5410>] ?
> try_to_merge_with_ksm_page+0x570/0x570
> [ 1100.480584]  [<ffffffff810b5410>] ?
> try_to_merge_with_ksm_page+0x570/0x570
> [ 1100.480624]  [<ffffffff810525b6>] ? kthread+0x96/0xa0
> [ 1100.480669]  [<ffffffff813d1794>] ? kernel_thread_helper+0x4/0x10
> [ 1100.480708]  [<ffffffff81052520>] ? kthread_worker_fn+0x120/0x120
> [ 1100.480748]  [<ffffffff813d1790>] ? gs_change+0xb/0xb
> [ 1100.480794] Rebooting in 60 seconds..
> 
> brad@srv:~$ zcat /proc/config.gz | grep -v "#"
> CONFIG_64BIT=y
> CONFIG_X86_64=y
> CONFIG_X86=y
> CONFIG_INSTRUCTION_DECODER=y
> CONFIG_OUTPUT_FORMAT="elf64-x86-64"
> CONFIG_ARCH_DEFCONFIG="arch/x86/configs/x86_64_defconfig"
> CONFIG_GENERIC_CMOS_UPDATE=y
> CONFIG_CLOCKSOURCE_WATCHDOG=y
> CONFIG_GENERIC_CLOCKEVENTS=y
> CONFIG_GENERIC_CLOCKEVENTS_BROADCAST=y
> CONFIG_LOCKDEP_SUPPORT=y
> CONFIG_STACKTRACE_SUPPORT=y
> CONFIG_HAVE_LATENCYTOP_SUPPORT=y
> CONFIG_MMU=y
> CONFIG_ZONE_DMA=y
> CONFIG_NEED_DMA_MAP_STATE=y
> CONFIG_NEED_SG_DMA_LENGTH=y
> CONFIG_GENERIC_ISA_DMA=y
> CONFIG_GENERIC_IOMAP=y
> CONFIG_GENERIC_BUG=y
> CONFIG_GENERIC_BUG_RELATIVE_POINTERS=y
> CONFIG_GENERIC_HWEIGHT=y
> CONFIG_ARCH_MAY_HAVE_PC_FDC=y
> CONFIG_RWSEM_XCHGADD_ALGORITHM=y
> CONFIG_ARCH_HAS_CPU_IDLE_WAIT=y
> CONFIG_GENERIC_CALIBRATE_DELAY=y
> CONFIG_GENERIC_TIME_VSYSCALL=y
> CONFIG_ARCH_HAS_CPU_RELAX=y
> CONFIG_ARCH_HAS_DEFAULT_IDLE=y
> CONFIG_ARCH_HAS_CACHE_LINE_SIZE=y
> CONFIG_HAVE_SETUP_PER_CPU_AREA=y
> CONFIG_NEED_PER_CPU_EMBED_FIRST_CHUNK=y
> CONFIG_NEED_PER_CPU_PAGE_FIRST_CHUNK=y
> CONFIG_HAVE_CPUMASK_OF_CPU_MAP=y
> CONFIG_ARCH_HIBERNATION_POSSIBLE=y
> CONFIG_ARCH_SUSPEND_POSSIBLE=y
> CONFIG_ZONE_DMA32=y
> CONFIG_ARCH_POPULATES_NODE_MAP=y
> CONFIG_AUDIT_ARCH=y
> CONFIG_ARCH_SUPPORTS_OPTIMIZED_INLINING=y
> CONFIG_ARCH_SUPPORTS_DEBUG_PAGEALLOC=y
> CONFIG_X86_64_SMP=y
> CONFIG_X86_HT=y
> CONFIG_ARCH_HWEIGHT_CFLAGS="-fcall-saved-rdi -fcall-saved-rsi
> -fcall-saved-rdx -fcall-saved-rcx -fcall-saved-r8 -fcall-saved-r9
> -fcall-saved-r10 -fcall-saved-r11"
> CONFIG_DEFCONFIG_LIST="/lib/modules/$UNAME_RELEASE/.config"
> CONFIG_CONSTRUCTORS=y
> CONFIG_HAVE_IRQ_WORK=y
> CONFIG_IRQ_WORK=y
> 
> CONFIG_EXPERIMENTAL=y
> CONFIG_INIT_ENV_ARG_LIMIT=32
> CONFIG_CROSS_COMPILE=""
> CONFIG_LOCALVERSION=""
> CONFIG_LOCALVERSION_AUTO=y
> CONFIG_HAVE_KERNEL_GZIP=y
> CONFIG_HAVE_KERNEL_BZIP2=y
> CONFIG_HAVE_KERNEL_LZMA=y
> CONFIG_HAVE_KERNEL_XZ=y
> CONFIG_HAVE_KERNEL_LZO=y
> CONFIG_KERNEL_GZIP=y
> CONFIG_SWAP=y
> CONFIG_SYSVIPC=y
> CONFIG_SYSVIPC_SYSCTL=y
> CONFIG_BSD_PROCESS_ACCT=y
> CONFIG_TASKSTATS=y
> CONFIG_TASK_DELAY_ACCT=y
> CONFIG_TASK_XACCT=y
> CONFIG_TASK_IO_ACCOUNTING=y
> CONFIG_HAVE_GENERIC_HARDIRQS=y
> 
> CONFIG_GENERIC_HARDIRQS=y
> CONFIG_HAVE_SPARSE_IRQ=y
> CONFIG_GENERIC_IRQ_PROBE=y
> CONFIG_GENERIC_IRQ_SHOW=y
> CONFIG_GENERIC_PENDING_IRQ=y
> CONFIG_IRQ_FORCED_THREADING=y
> 
> CONFIG_TREE_RCU=y
> CONFIG_RCU_FANOUT=64
> CONFIG_IKCONFIG=y
> CONFIG_IKCONFIG_PROC=y
> CONFIG_LOG_BUF_SHIFT=20
> CONFIG_HAVE_UNSTABLE_SCHED_CLOCK=y
> CONFIG_NAMESPACES=y
> CONFIG_BLK_DEV_INITRD=y
> CONFIG_INITRAMFS_SOURCE=""
> CONFIG_RD_GZIP=y
> CONFIG_RD_BZIP2=y
> CONFIG_RD_LZMA=y
> CONFIG_RD_XZ=y
> CONFIG_RD_LZO=y
> CONFIG_SYSCTL=y
> CONFIG_ANON_INODES=y
> CONFIG_UID16=y
> CONFIG_SYSCTL_SYSCALL=y
> CONFIG_KALLSYMS=y
> CONFIG_KALLSYMS_ALL=y
> CONFIG_KALLSYMS_EXTRA_PASS=y
> CONFIG_HOTPLUG=y
> CONFIG_PRINTK=y
> CONFIG_BUG=y
> CONFIG_ELF_CORE=y
> CONFIG_PCSPKR_PLATFORM=y
> CONFIG_BASE_FULL=y
> CONFIG_FUTEX=y
> CONFIG_EPOLL=y
> CONFIG_SIGNALFD=y
> CONFIG_TIMERFD=y
> CONFIG_EVENTFD=y
> CONFIG_SHMEM=y
> CONFIG_AIO=y
> CONFIG_HAVE_PERF_EVENTS=y
> 
> CONFIG_PERF_EVENTS=y
> CONFIG_VM_EVENT_COUNTERS=y
> CONFIG_PCI_QUIRKS=y
> CONFIG_SLUB_DEBUG=y
> CONFIG_SLUB=y
> CONFIG_HAVE_OPROFILE=y
> CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS=y
> CONFIG_USER_RETURN_NOTIFIER=y
> CONFIG_HAVE_IOREMAP_PROT=y
> CONFIG_HAVE_KPROBES=y
> CONFIG_HAVE_KRETPROBES=y
> CONFIG_HAVE_OPTPROBES=y
> CONFIG_HAVE_ARCH_TRACEHOOK=y
> CONFIG_HAVE_DMA_ATTRS=y
> CONFIG_USE_GENERIC_SMP_HELPERS=y
> CONFIG_HAVE_REGS_AND_STACK_ACCESS_API=y
> CONFIG_HAVE_DMA_API_DEBUG=y
> CONFIG_HAVE_HW_BREAKPOINT=y
> CONFIG_HAVE_MIXED_BREAKPOINTS_REGS=y
> CONFIG_HAVE_USER_RETURN_NOTIFIER=y
> CONFIG_HAVE_PERF_EVENTS_NMI=y
> CONFIG_HAVE_ARCH_JUMP_LABEL=y
> 
> CONFIG_SLABINFO=y
> CONFIG_RT_MUTEXES=y
> CONFIG_BASE_SMALL=0
> CONFIG_MODULES=y
> CONFIG_MODULE_UNLOAD=y
> CONFIG_STOP_MACHINE=y
> CONFIG_BLOCK=y
> CONFIG_BLK_DEV_BSG=y
> CONFIG_BLOCK_COMPAT=y
> 
> CONFIG_IOSCHED_NOOP=y
> CONFIG_IOSCHED_DEADLINE=y
> CONFIG_IOSCHED_CFQ=y
> CONFIG_DEFAULT_CFQ=y
> CONFIG_DEFAULT_IOSCHED="cfq"
> CONFIG_PREEMPT_NOTIFIERS=y
> CONFIG_INLINE_SPIN_UNLOCK=y
> CONFIG_INLINE_SPIN_UNLOCK_IRQ=y
> CONFIG_INLINE_READ_UNLOCK=y
> CONFIG_INLINE_READ_UNLOCK_IRQ=y
> CONFIG_INLINE_WRITE_UNLOCK=y
> CONFIG_INLINE_WRITE_UNLOCK_IRQ=y
> CONFIG_MUTEX_SPIN_ON_OWNER=y
> 
> CONFIG_TICK_ONESHOT=y
> CONFIG_NO_HZ=y
> CONFIG_HIGH_RES_TIMERS=y
> CONFIG_GENERIC_CLOCKEVENTS_BUILD=y
> CONFIG_SMP=y
> CONFIG_X86_SUPPORTS_MEMORY_FAILURE=y
> CONFIG_SCHED_OMIT_FRAME_POINTER=y
> CONFIG_NO_BOOTMEM=y
> CONFIG_MK8=y
> CONFIG_X86_INTERNODE_CACHE_SHIFT=6
> CONFIG_X86_CMPXCHG=y
> CONFIG_CMPXCHG_LOCAL=y
> CONFIG_X86_L1_CACHE_SHIFT=6
> CONFIG_X86_XADD=y
> CONFIG_X86_WP_WORKS_OK=y
> CONFIG_X86_INTEL_USERCOPY=y
> CONFIG_X86_USE_PPRO_CHECKSUM=y
> CONFIG_X86_TSC=y
> CONFIG_X86_CMPXCHG64=y
> CONFIG_X86_CMOV=y
> CONFIG_X86_MINIMUM_CPU_FAMILY=64
> CONFIG_X86_DEBUGCTLMSR=y
> CONFIG_CPU_SUP_INTEL=y
> CONFIG_CPU_SUP_AMD=y
> CONFIG_CPU_SUP_CENTAUR=y
> CONFIG_HPET_TIMER=y
> CONFIG_HPET_EMULATE_RTC=y
> CONFIG_DMI=y
> CONFIG_GART_IOMMU=y
> CONFIG_AMD_IOMMU=y
> CONFIG_SWIOTLB=y
> CONFIG_IOMMU_HELPER=y
> CONFIG_IOMMU_API=y
> CONFIG_NR_CPUS=8
> CONFIG_SCHED_MC=y
> CONFIG_PREEMPT_NONE=y
> CONFIG_X86_LOCAL_APIC=y
> CONFIG_X86_IO_APIC=y
> CONFIG_X86_MCE=y
> CONFIG_X86_MCE_AMD=y
> CONFIG_X86_MCE_THRESHOLD=y
> CONFIG_ARCH_PHYS_ADDR_T_64BIT=y
> CONFIG_ARCH_DMA_ADDR_T_64BIT=y
> CONFIG_DIRECT_GBPAGES=y
> CONFIG_ARCH_SPARSEMEM_DEFAULT=y
> CONFIG_ARCH_SPARSEMEM_ENABLE=y
> CONFIG_ARCH_SELECT_MEMORY_MODEL=y
> CONFIG_ARCH_MEMORY_PROBE=y
> CONFIG_ILLEGAL_POINTER_VALUE=0xdead000000000000
> CONFIG_SELECT_MEMORY_MODEL=y
> CONFIG_SPARSEMEM_MANUAL=y
> CONFIG_SPARSEMEM=y
> CONFIG_HAVE_MEMORY_PRESENT=y
> CONFIG_SPARSEMEM_EXTREME=y
> CONFIG_SPARSEMEM_VMEMMAP_ENABLE=y
> CONFIG_SPARSEMEM_ALLOC_MEM_MAP_TOGETHER=y
> CONFIG_SPARSEMEM_VMEMMAP=y
> CONFIG_HAVE_MEMBLOCK=y
> CONFIG_MEMORY_HOTPLUG=y
> CONFIG_MEMORY_HOTPLUG_SPARSE=y
> CONFIG_PAGEFLAGS_EXTENDED=y
> CONFIG_SPLIT_PTLOCK_CPUS=4
> CONFIG_COMPACTION=y
> CONFIG_MIGRATION=y
> CONFIG_PHYS_ADDR_T_64BIT=y
> CONFIG_ZONE_DMA_FLAG=1
> CONFIG_BOUNCE=y
> CONFIG_VIRT_TO_BUS=y
> CONFIG_MMU_NOTIFIER=y
> CONFIG_KSM=y
> CONFIG_DEFAULT_MMAP_MIN_ADDR=4096
> CONFIG_ARCH_SUPPORTS_MEMORY_FAILURE=y
> CONFIG_TRANSPARENT_HUGEPAGE=y
> CONFIG_TRANSPARENT_HUGEPAGE_ALWAYS=y
> CONFIG_X86_RESERVE_LOW=64
> CONFIG_MTRR=y
> CONFIG_MTRR_SANITIZER=y
> CONFIG_MTRR_SANITIZER_ENABLE_DEFAULT=0
> CONFIG_MTRR_SANITIZER_SPARE_REG_NR_DEFAULT=1
> CONFIG_X86_PAT=y
> CONFIG_ARCH_USES_PG_UNCACHED=y
> CONFIG_SECCOMP=y
> CONFIG_HZ_100=y
> CONFIG_HZ=100
> CONFIG_SCHED_HRTICK=y
> CONFIG_PHYSICAL_START=0x1000000
> CONFIG_PHYSICAL_ALIGN=0x1000000
> CONFIG_ARCH_ENABLE_MEMORY_HOTPLUG=y
> CONFIG_ARCH_ENABLE_MEMORY_HOTREMOVE=y
> 
> CONFIG_ACPI=y
> CONFIG_ACPI_BUTTON=y
> CONFIG_ACPI_DOCK=y
> CONFIG_ACPI_PROCESSOR=y
> CONFIG_ACPI_THERMAL=y
> CONFIG_ACPI_BLACKLIST_YEAR=0
> CONFIG_X86_PM_TIMER=y
> 
> CONFIG_CPU_FREQ=y
> CONFIG_CPU_FREQ_TABLE=y
> CONFIG_CPU_FREQ_STAT=y
> CONFIG_CPU_FREQ_STAT_DETAILS=y
> CONFIG_CPU_FREQ_DEFAULT_GOV_ONDEMAND=y
> CONFIG_CPU_FREQ_GOV_PERFORMANCE=y
> CONFIG_CPU_FREQ_GOV_POWERSAVE=y
> CONFIG_CPU_FREQ_GOV_USERSPACE=y
> CONFIG_CPU_FREQ_GOV_ONDEMAND=y
> CONFIG_CPU_FREQ_GOV_CONSERVATIVE=y
> 
> CONFIG_X86_ACPI_CPUFREQ=m
> CONFIG_X86_POWERNOW_K8=m
> 
> CONFIG_CPU_IDLE=y
> CONFIG_CPU_IDLE_GOV_LADDER=y
> CONFIG_CPU_IDLE_GOV_MENU=y
> 
> 
> CONFIG_PCI=y
> CONFIG_PCI_DIRECT=y
> CONFIG_PCI_MMCONFIG=y
> CONFIG_PCI_DOMAINS=y
> CONFIG_PCIEPORTBUS=y
> CONFIG_PCIEAER=y
> CONFIG_PCIEASPM=y
> CONFIG_ARCH_SUPPORTS_MSI=y
> CONFIG_PCI_MSI=y
> CONFIG_HT_IRQ=y
> CONFIG_PCI_IOAPIC=y
> CONFIG_PCI_LABEL=y
> CONFIG_ISA_DMA_API=y
> CONFIG_AMD_NB=y
> 
> CONFIG_BINFMT_ELF=y
> CONFIG_COMPAT_BINFMT_ELF=y
> CONFIG_BINFMT_MISC=m
> CONFIG_IA32_EMULATION=y
> CONFIG_IA32_AOUT=m
> CONFIG_COMPAT=y
> CONFIG_COMPAT_FOR_U64_ALIGNMENT=y
> CONFIG_SYSVIPC_COMPAT=y
> CONFIG_HAVE_TEXT_POKE_SMP=y
> CONFIG_NET=y
> 
> CONFIG_PACKET=y
> CONFIG_UNIX=y
> CONFIG_XFRM=y
> CONFIG_XFRM_MIGRATE=y
> CONFIG_XFRM_IPCOMP=m
> CONFIG_NET_KEY=m
> CONFIG_NET_KEY_MIGRATE=y
> CONFIG_INET=y
> CONFIG_IP_MULTICAST=y
> CONFIG_IP_ROUTE_CLASSID=y
> CONFIG_NET_IPGRE_DEMUX=m
> CONFIG_INET_AH=m
> CONFIG_INET_ESP=m
> CONFIG_INET_IPCOMP=m
> CONFIG_INET_XFRM_TUNNEL=m
> CONFIG_INET_TUNNEL=m
> CONFIG_INET_XFRM_MODE_TRANSPORT=m
> CONFIG_INET_XFRM_MODE_TUNNEL=m
> CONFIG_INET_XFRM_MODE_BEET=m
> CONFIG_INET_DIAG=y
> CONFIG_INET_TCP_DIAG=y
> CONFIG_TCP_CONG_CUBIC=y
> CONFIG_DEFAULT_TCP_CONG="cubic"
> CONFIG_NETFILTER=y
> CONFIG_NETFILTER_ADVANCED=y
> CONFIG_BRIDGE_NETFILTER=y
> 
> CONFIG_NETFILTER_NETLINK=m
> CONFIG_NETFILTER_NETLINK_QUEUE=m
> CONFIG_NETFILTER_NETLINK_LOG=m
> CONFIG_NF_CONNTRACK=m
> CONFIG_NF_CONNTRACK_MARK=y
> CONFIG_NF_CONNTRACK_ZONES=y
> CONFIG_NF_CT_PROTO_DCCP=m
> CONFIG_NF_CT_PROTO_GRE=m
> CONFIG_NF_CT_PROTO_SCTP=m
> CONFIG_NF_CT_PROTO_UDPLITE=m
> CONFIG_NF_CONNTRACK_AMANDA=m
> CONFIG_NF_CONNTRACK_FTP=m
> CONFIG_NF_CONNTRACK_H323=m
> CONFIG_NF_CONNTRACK_IRC=m
> CONFIG_NF_CONNTRACK_BROADCAST=m
> CONFIG_NF_CONNTRACK_NETBIOS_NS=m
> CONFIG_NF_CONNTRACK_SNMP=m
> CONFIG_NF_CONNTRACK_PPTP=m
> CONFIG_NF_CONNTRACK_SANE=m
> CONFIG_NF_CONNTRACK_SIP=m
> CONFIG_NF_CONNTRACK_TFTP=m
> CONFIG_NF_CT_NETLINK=m
> CONFIG_NETFILTER_TPROXY=m
> CONFIG_NETFILTER_XTABLES=m
> 
> CONFIG_NETFILTER_XT_MARK=m
> CONFIG_NETFILTER_XT_CONNMARK=m
> 
> CONFIG_NETFILTER_XT_TARGET_CHECKSUM=m
> CONFIG_NETFILTER_XT_TARGET_CLASSIFY=m
> CONFIG_NETFILTER_XT_TARGET_CONNMARK=m
> CONFIG_NETFILTER_XT_TARGET_CT=m
> CONFIG_NETFILTER_XT_TARGET_DSCP=m
> CONFIG_NETFILTER_XT_TARGET_HL=m
> CONFIG_NETFILTER_XT_TARGET_IDLETIMER=m
> CONFIG_NETFILTER_XT_TARGET_MARK=m
> CONFIG_NETFILTER_XT_TARGET_NFLOG=m
> CONFIG_NETFILTER_XT_TARGET_NFQUEUE=m
> CONFIG_NETFILTER_XT_TARGET_NOTRACK=m
> CONFIG_NETFILTER_XT_TARGET_RATEEST=m
> CONFIG_NETFILTER_XT_TARGET_TEE=m
> CONFIG_NETFILTER_XT_TARGET_TPROXY=m
> CONFIG_NETFILTER_XT_TARGET_TRACE=m
> CONFIG_NETFILTER_XT_TARGET_TCPMSS=m
> CONFIG_NETFILTER_XT_TARGET_TCPOPTSTRIP=m
> 
> CONFIG_NETFILTER_XT_MATCH_ADDRTYPE=m
> CONFIG_NETFILTER_XT_MATCH_CLUSTER=m
> CONFIG_NETFILTER_XT_MATCH_COMMENT=m
> CONFIG_NETFILTER_XT_MATCH_CONNBYTES=m
> CONFIG_NETFILTER_XT_MATCH_CONNLIMIT=m
> CONFIG_NETFILTER_XT_MATCH_CONNMARK=m
> CONFIG_NETFILTER_XT_MATCH_CONNTRACK=m
> CONFIG_NETFILTER_XT_MATCH_CPU=m
> CONFIG_NETFILTER_XT_MATCH_DCCP=m
> CONFIG_NETFILTER_XT_MATCH_DEVGROUP=m
> CONFIG_NETFILTER_XT_MATCH_DSCP=m
> CONFIG_NETFILTER_XT_MATCH_ESP=m
> CONFIG_NETFILTER_XT_MATCH_HASHLIMIT=m
> CONFIG_NETFILTER_XT_MATCH_HELPER=m
> CONFIG_NETFILTER_XT_MATCH_HL=m
> CONFIG_NETFILTER_XT_MATCH_IPRANGE=m
> CONFIG_NETFILTER_XT_MATCH_LENGTH=m
> CONFIG_NETFILTER_XT_MATCH_LIMIT=m
> CONFIG_NETFILTER_XT_MATCH_MAC=m
> CONFIG_NETFILTER_XT_MATCH_MARK=m
> CONFIG_NETFILTER_XT_MATCH_MULTIPORT=m
> CONFIG_NETFILTER_XT_MATCH_OSF=m
> CONFIG_NETFILTER_XT_MATCH_OWNER=m
> CONFIG_NETFILTER_XT_MATCH_POLICY=m
> CONFIG_NETFILTER_XT_MATCH_PHYSDEV=m
> CONFIG_NETFILTER_XT_MATCH_PKTTYPE=m
> CONFIG_NETFILTER_XT_MATCH_QUOTA=m
> CONFIG_NETFILTER_XT_MATCH_RATEEST=m
> CONFIG_NETFILTER_XT_MATCH_REALM=m
> CONFIG_NETFILTER_XT_MATCH_RECENT=m
> CONFIG_NETFILTER_XT_MATCH_SCTP=m
> CONFIG_NETFILTER_XT_MATCH_SOCKET=m
> CONFIG_NETFILTER_XT_MATCH_STATE=m
> CONFIG_NETFILTER_XT_MATCH_STATISTIC=m
> CONFIG_NETFILTER_XT_MATCH_STRING=m
> CONFIG_NETFILTER_XT_MATCH_TCPMSS=m
> CONFIG_NETFILTER_XT_MATCH_TIME=m
> CONFIG_NETFILTER_XT_MATCH_U32=m
> 
> CONFIG_NF_DEFRAG_IPV4=m
> CONFIG_NF_CONNTRACK_IPV4=m
> CONFIG_NF_CONNTRACK_PROC_COMPAT=y
> CONFIG_IP_NF_QUEUE=m
> CONFIG_IP_NF_IPTABLES=m
> CONFIG_IP_NF_MATCH_AH=m
> CONFIG_IP_NF_MATCH_ECN=m
> CONFIG_IP_NF_MATCH_TTL=m
> CONFIG_IP_NF_FILTER=m
> CONFIG_IP_NF_TARGET_REJECT=m
> CONFIG_IP_NF_TARGET_LOG=m
> CONFIG_IP_NF_TARGET_ULOG=m
> CONFIG_NF_NAT=m
> CONFIG_NF_NAT_NEEDED=y
> CONFIG_IP_NF_TARGET_MASQUERADE=m
> CONFIG_IP_NF_TARGET_NETMAP=m
> CONFIG_IP_NF_TARGET_REDIRECT=m
> CONFIG_NF_NAT_SNMP_BASIC=m
> CONFIG_NF_NAT_PROTO_DCCP=m
> CONFIG_NF_NAT_PROTO_GRE=m
> CONFIG_NF_NAT_PROTO_UDPLITE=m
> CONFIG_NF_NAT_PROTO_SCTP=m
> CONFIG_NF_NAT_FTP=m
> CONFIG_NF_NAT_IRC=m
> CONFIG_NF_NAT_TFTP=m
> CONFIG_NF_NAT_AMANDA=m
> CONFIG_NF_NAT_PPTP=m
> CONFIG_NF_NAT_H323=m
> CONFIG_NF_NAT_SIP=m
> CONFIG_IP_NF_MANGLE=m
> CONFIG_IP_NF_TARGET_CLUSTERIP=m
> CONFIG_IP_NF_TARGET_ECN=m
> CONFIG_IP_NF_TARGET_TTL=m
> CONFIG_IP_NF_RAW=m
> CONFIG_IP_NF_ARPTABLES=m
> CONFIG_IP_NF_ARPFILTER=m
> CONFIG_IP_NF_ARP_MANGLE=m
> CONFIG_IP_SCTP=m
> CONFIG_SCTP_HMAC_MD5=y
> CONFIG_STP=y
> CONFIG_BRIDGE=y
> CONFIG_BRIDGE_IGMP_SNOOPING=y
> CONFIG_VLAN_8021Q=y
> CONFIG_LLC=y
> CONFIG_NET_SCHED=y
> 
> CONFIG_NET_SCH_CBQ=m
> CONFIG_NET_SCH_HTB=m
> CONFIG_NET_SCH_HFSC=m
> CONFIG_NET_SCH_PRIO=m
> CONFIG_NET_SCH_RED=m
> CONFIG_NET_SCH_SFB=m
> CONFIG_NET_SCH_SFQ=m
> CONFIG_NET_SCH_TEQL=m
> CONFIG_NET_SCH_TBF=m
> CONFIG_NET_SCH_GRED=m
> CONFIG_NET_SCH_DSMARK=m
> CONFIG_NET_SCH_NETEM=m
> CONFIG_NET_SCH_MQPRIO=m
> CONFIG_NET_SCH_CHOKE=m
> CONFIG_NET_SCH_INGRESS=m
> 
> CONFIG_NET_CLS=y
> CONFIG_NET_CLS_BASIC=m
> CONFIG_NET_CLS_TCINDEX=m
> CONFIG_NET_CLS_ROUTE4=m
> CONFIG_NET_CLS_FW=m
> CONFIG_NET_CLS_U32=m
> CONFIG_NET_CLS_FLOW=m
> CONFIG_NET_EMATCH=y
> CONFIG_NET_EMATCH_STACK=32
> CONFIG_NET_EMATCH_CMP=m
> CONFIG_NET_EMATCH_NBYTE=m
> CONFIG_NET_EMATCH_U32=m
> CONFIG_NET_EMATCH_META=m
> CONFIG_NET_EMATCH_TEXT=m
> CONFIG_NET_CLS_ACT=y
> CONFIG_NET_ACT_POLICE=m
> CONFIG_NET_ACT_GACT=m
> CONFIG_GACT_PROB=y
> CONFIG_NET_ACT_MIRRED=m
> CONFIG_NET_ACT_IPT=m
> CONFIG_NET_ACT_NAT=m
> CONFIG_NET_ACT_PEDIT=m
> CONFIG_NET_ACT_SIMP=m
> CONFIG_NET_ACT_SKBEDIT=m
> CONFIG_NET_ACT_CSUM=m
> CONFIG_NET_SCH_FIFO=y
> CONFIG_RPS=y
> CONFIG_RFS_ACCEL=y
> CONFIG_XPS=y
> 
> 
> 
> CONFIG_UEVENT_HELPER_PATH="/sbin/hotplug"
> CONFIG_DEVTMPFS=y
> CONFIG_STANDALONE=y
> CONFIG_PREVENT_FIRMWARE_BUILD=y
> CONFIG_FW_LOADER=y
> CONFIG_FIRMWARE_IN_KERNEL=y
> CONFIG_EXTRA_FIRMWARE=""
> CONFIG_ARCH_NO_SYSDEV_OPS=y
> CONFIG_PNP=y
> CONFIG_PNP_DEBUG_MESSAGES=y
> 
> CONFIG_PNPACPI=y
> CONFIG_BLK_DEV=y
> CONFIG_BLK_DEV_LOOP=y
> 
> CONFIG_HAVE_IDE=y
> 
> CONFIG_SCSI_MOD=y
> CONFIG_RAID_ATTRS=m
> CONFIG_SCSI=y
> CONFIG_SCSI_DMA=y
> CONFIG_SCSI_PROC_FS=y
> 
> CONFIG_BLK_DEV_SD=y
> CONFIG_BLK_DEV_SR=y
> CONFIG_CHR_DEV_SG=y
> CONFIG_SCSI_WAIT_SCAN=m
> 
> CONFIG_SCSI_SAS_ATTRS=m
> CONFIG_SCSI_LOWLEVEL=y
> CONFIG_MEGARAID_SAS=m
> CONFIG_ATA=y
> CONFIG_ATA_VERBOSE_ERROR=y
> CONFIG_ATA_ACPI=y
> CONFIG_SATA_PMP=y
> 
> CONFIG_SATA_AHCI=m
> CONFIG_SATA_SIL24=m
> CONFIG_ATA_SFF=y
> 
> CONFIG_ATA_BMDMA=y
> 
> CONFIG_SATA_MV=m
> CONFIG_SATA_NV=m
> CONFIG_SATA_PROMISE=m
> CONFIG_SATA_SIL=m
> 
> 
> 
> CONFIG_MD=y
> CONFIG_BLK_DEV_MD=y
> CONFIG_MD_LINEAR=y
> CONFIG_MD_RAID0=y
> CONFIG_MD_RAID1=y
> CONFIG_MD_RAID10=y
> CONFIG_MD_RAID456=y
> CONFIG_BLK_DEV_DM=y
> 
> CONFIG_NETDEVICES=y
> CONFIG_DUMMY=m
> CONFIG_BONDING=m
> CONFIG_TUN=y
> CONFIG_MII=m
> CONFIG_NET_ETHERNET=y
> CONFIG_NET_PCI=y
> CONFIG_FORCEDETH=m
> CONFIG_NETDEV_1000=y
> CONFIG_R8169=m
> 
> 
> 
> CONFIG_PPP=m
> CONFIG_PPP_MULTILINK=y
> CONFIG_PPP_FILTER=y
> CONFIG_PPP_ASYNC=m
> CONFIG_PPP_SYNC_TTY=m
> CONFIG_PPP_DEFLATE=m
> CONFIG_PPP_BSDCOMP=m
> CONFIG_PPP_MPPE=m
> CONFIG_PPPOE=m
> CONFIG_SLHC=m
> CONFIG_NETCONSOLE=m
> CONFIG_NETCONSOLE_DYNAMIC=y
> CONFIG_NETPOLL=y
> CONFIG_NET_POLL_CONTROLLER=y
> 
> CONFIG_INPUT=y
> 
> CONFIG_INPUT_MOUSEDEV=y
> CONFIG_INPUT_MOUSEDEV_SCREEN_X=1024
> CONFIG_INPUT_MOUSEDEV_SCREEN_Y=768
> 
> CONFIG_INPUT_KEYBOARD=y
> CONFIG_KEYBOARD_ATKBD=y
> 
> CONFIG_SERIO=y
> CONFIG_SERIO_I8042=y
> CONFIG_SERIO_LIBPS2=y
> 
> CONFIG_VT=y
> CONFIG_CONSOLE_TRANSLATIONS=y
> CONFIG_VT_CONSOLE=y
> CONFIG_HW_CONSOLE=y
> CONFIG_UNIX98_PTYS=y
> CONFIG_LEGACY_PTYS=y
> CONFIG_LEGACY_PTY_COUNT=256
> 
> CONFIG_SERIAL_8250=y
> CONFIG_SERIAL_8250_CONSOLE=y
> CONFIG_FIX_EARLYCON_MEM=y
> CONFIG_SERIAL_8250_PCI=y
> CONFIG_SERIAL_8250_PNP=y
> CONFIG_SERIAL_8250_NR_UARTS=4
> CONFIG_SERIAL_8250_RUNTIME_UARTS=4
> 
> CONFIG_SERIAL_CORE=y
> CONFIG_SERIAL_CORE_CONSOLE=y
> CONFIG_RTC=y
> CONFIG_HPET=y
> CONFIG_HPET_MMAP=y
> CONFIG_DEVPORT=y
> CONFIG_I2C=y
> CONFIG_I2C_BOARDINFO=y
> CONFIG_I2C_COMPAT=y
> CONFIG_I2C_CHARDEV=y
> CONFIG_I2C_HELPER_AUTO=y
> CONFIG_I2C_ALGOBIT=y
> 
> 
> CONFIG_I2C_AMD756=m
> CONFIG_I2C_PIIX4=m
> CONFIG_I2C_NFORCE2=m
> CONFIG_I2C_VIA=m
> CONFIG_I2C_VIAPRO=m
> 
> CONFIG_I2C_SCMI=m
> 
> 
> 
> 
> 
> CONFIG_ARCH_WANT_OPTIONAL_GPIOLIB=y
> CONFIG_HWMON=y
> CONFIG_HWMON_VID=m
> 
> CONFIG_SENSORS_K8TEMP=m
> CONFIG_SENSORS_K10TEMP=m
> CONFIG_SENSORS_W83781D=m
> CONFIG_SENSORS_W83791D=m
> CONFIG_SENSORS_W83792D=m
> CONFIG_SENSORS_W83793=m
> CONFIG_SENSORS_W83795=m
> CONFIG_SENSORS_W83795_FANCTRL=y
> CONFIG_SENSORS_W83L785TS=m
> CONFIG_SENSORS_W83L786NG=m
> CONFIG_SENSORS_W83627HF=m
> CONFIG_SENSORS_W83627EHF=m
> 
> CONFIG_SENSORS_ATK0110=m
> CONFIG_THERMAL=y
> CONFIG_THERMAL_HWMON=y
> CONFIG_SSB_POSSIBLE=y
> 
> 
> CONFIG_VGA_ARB=y
> CONFIG_VGA_ARB_MAX_GPUS=16
> CONFIG_VGASTATE=y
> CONFIG_FB=y
> CONFIG_FB_DDC=y
> CONFIG_FB_CFB_FILLRECT=y
> CONFIG_FB_CFB_COPYAREA=y
> CONFIG_FB_CFB_IMAGEBLIT=y
> CONFIG_FB_MODE_HELPERS=y
> 
> CONFIG_FB_NVIDIA=y
> CONFIG_FB_NVIDIA_I2C=y
> CONFIG_FB_RADEON=y
> CONFIG_FB_RADEON_I2C=y
> 
> 
> CONFIG_VGA_CONSOLE=y
> CONFIG_DUMMY_CONSOLE=y
> CONFIG_FRAMEBUFFER_CONSOLE=y
> CONFIG_FONT_8x8=y
> CONFIG_FONT_8x16=y
> CONFIG_LOGO=y
> CONFIG_LOGO_LINUX_MONO=y
> CONFIG_LOGO_LINUX_VGA16=y
> CONFIG_LOGO_LINUX_CLUT224=y
> CONFIG_HID_SUPPORT=y
> CONFIG_HID=y
> CONFIG_HIDRAW=y
> 
> CONFIG_USB_HID=m
> CONFIG_USB_HIDDEV=y
> 
> CONFIG_HID_A4TECH=m
> CONFIG_HID_APPLE=m
> CONFIG_HID_BELKIN=m
> CONFIG_HID_CHERRY=m
> CONFIG_HID_CHICONY=m
> CONFIG_HID_CYPRESS=m
> CONFIG_HID_EZKEY=m
> CONFIG_HID_KYE=m
> CONFIG_HID_KENSINGTON=m
> CONFIG_HID_LOGITECH=m
> CONFIG_HID_MICROSOFT=m
> CONFIG_HID_MONTEREY=m
> CONFIG_USB_SUPPORT=y
> CONFIG_USB_ARCH_HAS_HCD=y
> CONFIG_USB_ARCH_HAS_OHCI=y
> CONFIG_USB_ARCH_HAS_EHCI=y
> CONFIG_USB=m
> 
> 
> CONFIG_USB_XHCI_HCD=m
> CONFIG_USB_EHCI_HCD=m
> CONFIG_USB_OHCI_HCD=m
> CONFIG_USB_OHCI_LITTLE_ENDIAN=y
> CONFIG_USB_UHCI_HCD=m
> 
> 
> CONFIG_USB_ACM=m
> 
> 
> CONFIG_USB_STORAGE=m
> CONFIG_USB_LIBUSUAL=y
> 
> 
> CONFIG_USB_SERIAL=m
> CONFIG_USB_SERIAL_GENERIC=y
> CONFIG_USB_SERIAL_FTDI_SIO=m
> CONFIG_USB_SERIAL_IPAQ=m
> CONFIG_USB_SERIAL_PL2303=m
> 
> 
> 
> CONFIG_FIRMWARE_MEMMAP=y
> CONFIG_DMIID=y
> 
> CONFIG_EXT2_FS=y
> CONFIG_EXT2_FS_XATTR=y
> CONFIG_EXT3_FS=y
> CONFIG_EXT3_FS_XATTR=y
> CONFIG_EXT4_FS=y
> CONFIG_EXT4_FS_XATTR=y
> CONFIG_JBD=y
> CONFIG_JBD2=y
> CONFIG_FS_MBCACHE=y
> CONFIG_EXPORTFS=y
> CONFIG_FILE_LOCKING=y
> CONFIG_FSNOTIFY=y
> CONFIG_DNOTIFY=y
> CONFIG_INOTIFY_USER=y
> CONFIG_FANOTIFY=y
> CONFIG_FUSE_FS=m
> CONFIG_CUSE=m
> 
> CONFIG_FSCACHE=m
> CONFIG_FSCACHE_DEBUG=y
> CONFIG_CACHEFILES=m
> 
> CONFIG_ISO9660_FS=m
> CONFIG_JOLIET=y
> CONFIG_ZISOFS=y
> CONFIG_UDF_FS=m
> CONFIG_UDF_NLS=y
> 
> CONFIG_FAT_FS=m
> CONFIG_MSDOS_FS=m
> CONFIG_VFAT_FS=m
> CONFIG_FAT_DEFAULT_CODEPAGE=437
> CONFIG_FAT_DEFAULT_IOCHARSET="iso8859-1"
> CONFIG_NTFS_FS=m
> 
> CONFIG_PROC_FS=y
> CONFIG_PROC_SYSCTL=y
> CONFIG_PROC_PAGE_MONITOR=y
> CONFIG_SYSFS=y
> CONFIG_TMPFS=y
> CONFIG_HUGETLBFS=y
> CONFIG_HUGETLB_PAGE=y
> CONFIG_CONFIGFS_FS=m
> CONFIG_NETWORK_FILESYSTEMS=y
> CONFIG_NFS_FS=y
> CONFIG_NFS_V3=y
> CONFIG_NFSD=y
> CONFIG_NFSD_DEPRECATED=y
> CONFIG_NFSD_V3=y
> CONFIG_LOCKD=y
> CONFIG_LOCKD_V4=y
> CONFIG_NFS_COMMON=y
> CONFIG_SUNRPC=y
> 
> CONFIG_PARTITION_ADVANCED=y
> CONFIG_MSDOS_PARTITION=y
> CONFIG_EFI_PARTITION=y
> CONFIG_NLS=y
> CONFIG_NLS_DEFAULT="iso8859-1"
> CONFIG_NLS_CODEPAGE_437=y
> CONFIG_NLS_ISO8859_1=y
> CONFIG_DLM=m
> 
> CONFIG_TRACE_IRQFLAGS_SUPPORT=y
> CONFIG_PRINTK_TIME=y
> CONFIG_DEFAULT_MESSAGE_LOGLEVEL=4
> CONFIG_ENABLE_WARN_DEPRECATED=y
> CONFIG_ENABLE_MUST_CHECK=y
> CONFIG_FRAME_WARN=2048
> CONFIG_MAGIC_SYSRQ=y
> CONFIG_DEBUG_FS=y
> CONFIG_DEBUG_KERNEL=y
> CONFIG_DETECT_HUNG_TASK=y
> CONFIG_BOOTPARAM_HUNG_TASK_PANIC_VALUE=0
> CONFIG_TIMER_STATS=y
> CONFIG_DEBUG_BUGVERBOSE=y
> CONFIG_DEBUG_MEMORY_INIT=y
> CONFIG_ARCH_WANT_FRAME_POINTERS=y
> CONFIG_SYSCTL_SYSCALL_CHECK=y
> CONFIG_USER_STACKTRACE_SUPPORT=y
> CONFIG_HAVE_FUNCTION_TRACER=y
> CONFIG_HAVE_FUNCTION_GRAPH_TRACER=y
> CONFIG_HAVE_FUNCTION_GRAPH_FP_TEST=y
> CONFIG_HAVE_FUNCTION_TRACE_MCOUNT_TEST=y
> CONFIG_HAVE_DYNAMIC_FTRACE=y
> CONFIG_HAVE_FTRACE_MCOUNT_RECORD=y
> CONFIG_HAVE_SYSCALL_TRACEPOINTS=y
> CONFIG_HAVE_C_RECORDMCOUNT=y
> CONFIG_TRACING_SUPPORT=y
> CONFIG_HAVE_ARCH_KGDB=y
> CONFIG_HAVE_ARCH_KMEMCHECK=y
> CONFIG_X86_VERBOSE_BOOTUP=y
> CONFIG_EARLY_PRINTK=y
> CONFIG_HAVE_MMIOTRACE_SUPPORT=y
> CONFIG_IO_DELAY_TYPE_0X80=0
> CONFIG_IO_DELAY_TYPE_0XED=1
> CONFIG_IO_DELAY_TYPE_UDELAY=2
> CONFIG_IO_DELAY_TYPE_NONE=3
> CONFIG_IO_DELAY_0X80=y
> CONFIG_DEFAULT_IO_DELAY_TYPE=0
> 
> CONFIG_DEFAULT_SECURITY_DAC=y
> CONFIG_DEFAULT_SECURITY=""
> CONFIG_XOR_BLOCKS=y
> CONFIG_ASYNC_CORE=y
> CONFIG_ASYNC_MEMCPY=y
> CONFIG_ASYNC_XOR=y
> CONFIG_ASYNC_PQ=y
> CONFIG_ASYNC_RAID6_RECOV=y
> CONFIG_CRYPTO=y
> 
> CONFIG_CRYPTO_ALGAPI=m
> CONFIG_CRYPTO_ALGAPI2=m
> CONFIG_CRYPTO_AEAD=m
> CONFIG_CRYPTO_AEAD2=m
> CONFIG_CRYPTO_BLKCIPHER=m
> CONFIG_CRYPTO_BLKCIPHER2=m
> CONFIG_CRYPTO_HASH=m
> CONFIG_CRYPTO_HASH2=m
> CONFIG_CRYPTO_RNG2=m
> CONFIG_CRYPTO_PCOMP2=m
> CONFIG_CRYPTO_MANAGER=m
> CONFIG_CRYPTO_MANAGER2=m
> CONFIG_CRYPTO_MANAGER_DISABLE_TESTS=y
> CONFIG_CRYPTO_WORKQUEUE=m
> CONFIG_CRYPTO_AUTHENC=m
> 
> 
> CONFIG_CRYPTO_CBC=m
> CONFIG_CRYPTO_ECB=m
> 
> CONFIG_CRYPTO_HMAC=m
> 
> CONFIG_CRYPTO_CRC32C=m
> CONFIG_CRYPTO_MD5=m
> CONFIG_CRYPTO_SHA1=m
> 
> CONFIG_CRYPTO_ARC4=m
> CONFIG_CRYPTO_DES=m
> 
> CONFIG_CRYPTO_DEFLATE=m
> 
> CONFIG_HAVE_KVM=y
> CONFIG_HAVE_KVM_IRQCHIP=y
> CONFIG_HAVE_KVM_EVENTFD=y
> CONFIG_KVM_APIC_ARCHITECTURE=y
> CONFIG_KVM_MMIO=y
> CONFIG_KVM_ASYNC_PF=y
> CONFIG_VIRTUALIZATION=y
> CONFIG_KVM=m
> CONFIG_KVM_INTEL=m
> CONFIG_KVM_AMD=m
> CONFIG_VHOST_NET=m
> 
> CONFIG_RAID6_PQ=y
> CONFIG_BITREVERSE=y
> CONFIG_GENERIC_FIND_FIRST_BIT=y
> CONFIG_GENERIC_FIND_NEXT_BIT=y
> CONFIG_GENERIC_FIND_LAST_BIT=y
> CONFIG_CRC_CCITT=m
> CONFIG_CRC16=y
> CONFIG_CRC_T10DIF=y
> CONFIG_CRC_ITU_T=y
> CONFIG_CRC32=y
> CONFIG_LIBCRC32C=m
> CONFIG_ZLIB_INFLATE=y
> CONFIG_ZLIB_DEFLATE=m
> CONFIG_LZO_DECOMPRESS=y
> CONFIG_XZ_DEC=y
> CONFIG_XZ_DEC_X86=y
> CONFIG_XZ_DEC_POWERPC=y
> CONFIG_XZ_DEC_IA64=y
> CONFIG_XZ_DEC_ARM=y
> CONFIG_XZ_DEC_ARMTHUMB=y
> CONFIG_XZ_DEC_SPARC=y
> CONFIG_XZ_DEC_BCJ=y
> CONFIG_DECOMPRESS_GZIP=y
> CONFIG_DECOMPRESS_BZIP2=y
> CONFIG_DECOMPRESS_LZMA=y
> CONFIG_DECOMPRESS_XZ=y
> CONFIG_DECOMPRESS_LZO=y
> CONFIG_TEXTSEARCH=y
> CONFIG_TEXTSEARCH_KMP=m
> CONFIG_TEXTSEARCH_BM=m
> CONFIG_TEXTSEARCH_FSM=m
> CONFIG_HAS_IOMEM=y
> CONFIG_HAS_IOPORT=y
> CONFIG_HAS_DMA=y
> CONFIG_CPU_RMAP=y
> CONFIG_NLATTR=y
> 
> 00:00.0 Host bridge: Advanced Micro Devices [AMD] RS880 Host Bridge
> 00:01.0 PCI bridge: ASRock Incorporation Device 9602
> 00:02.0 PCI bridge: Advanced Micro Devices [AMD] RS780 PCI to PCI
> bridge (ext gfx port 0)
> 00:03.0 PCI bridge: Advanced Micro Devices [AMD] RS780 PCI to PCI
> bridge (ext gfx port 1)
> 00:04.0 PCI bridge: Advanced Micro Devices [AMD] RS780 PCI to PCI
> bridge (PCIE port 0)
> 00:09.0 PCI bridge: Advanced Micro Devices [AMD] RS780 PCI to PCI
> bridge (PCIE port 4)
> 00:0a.0 PCI bridge: Advanced Micro Devices [AMD] RS780 PCI to PCI
> bridge (PCIE port 5)
> 00:11.0 SATA controller: ATI Technologies Inc SB700/SB800 SATA
> Controller [AHCI mode] (rev 40)
> 00:12.0 USB Controller: ATI Technologies Inc SB700/SB800 USB OHCI0
> Controller
> 00:12.2 USB Controller: ATI Technologies Inc SB700/SB800 USB EHCI Controller
> 00:13.0 USB Controller: ATI Technologies Inc SB700/SB800 USB OHCI0
> Controller
> 00:13.2 USB Controller: ATI Technologies Inc SB700/SB800 USB EHCI Controller
> 00:14.0 SMBus: ATI Technologies Inc SBx00 SMBus Controller (rev 41)
> 00:14.2 Audio device: ATI Technologies Inc SBx00 Azalia (Intel HDA) (rev 40)
> 00:14.3 ISA bridge: ATI Technologies Inc SB700/SB800 LPC host
> controller (rev 40)
> 00:14.4 PCI bridge: ATI Technologies Inc SBx00 PCI to PCI Bridge (rev 40)
> 00:14.5 USB Controller: ATI Technologies Inc SB700/SB800 USB OHCI2
> Controller
> 00:16.0 USB Controller: ATI Technologies Inc SB700/SB800 USB OHCI0
> Controller
> 00:16.2 USB Controller: ATI Technologies Inc SB700/SB800 USB EHCI Controller
> 00:18.0 Host bridge: Advanced Micro Devices [AMD] Family 10h
> Processor HyperTransport Configuration
> 00:18.1 Host bridge: Advanced Micro Devices [AMD] Family 10h
> Processor Address Map
> 00:18.2 Host bridge: Advanced Micro Devices [AMD] Family 10h
> Processor DRAM Controller
> 00:18.3 Host bridge: Advanced Micro Devices [AMD] Family 10h
> Processor Miscellaneous Control
> 00:18.4 Host bridge: Advanced Micro Devices [AMD] Family 10h
> Processor Link Control
> 01:05.0 VGA compatible controller: ATI Technologies Inc RS880
> [Radeon HD 4250]
> 02:00.0 USB Controller: NEC Corporation uPD720200 USB 3.0 Host
> Controller (rev 03)
> 03:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd.
> RTL8111/8168B PCI Express Gigabit Ethernet controller (rev 06)
> 04:00.0 RAID bus controller: LSI Logic / Symbios Logic MegaRAID SAS
> 9240 (rev 02)
> 05:00.0 RAID bus controller: LSI Logic / Symbios Logic MegaRAID SAS
> 9240 (rev 02)
> 07:00.0 SCSI storage controller: Marvell Technology Group Ltd.
> 88SX7042 PCI-e 4-port SATA-II (rev 02)
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

-- 
Regards/Gruss,
    Boris.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
  2011-05-31  5:47   ` Borislav Petkov
@ 2011-05-31  9:26     ` Brad Campbell
  -1 siblings, 0 replies; 111+ messages in thread
From: Brad Campbell @ 2011-05-31  9:26 UTC (permalink / raw)
  To: Borislav Petkov, linux-kernel, kvm, linux-mm

On 31/05/11 13:47, Borislav Petkov wrote:
> Looks like a KSM issue. Disabling CONFIG_KSM should at least stop your
> machine from oopsing.
>
> Adding linux-mm.
>

I initially thought that, so the second panic was produced with KSM 
disabled from boot.

echo 0 > /sys/kernel/mm/ksm/run

If you still think that compiling ksm out of the kernel will prevent it 
then I'm willing to give it a go.

It's a production server, so I can only really bounce it around after 
about 9PM - GMT+8.

Regards,
Brad

^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
@ 2011-05-31  9:26     ` Brad Campbell
  0 siblings, 0 replies; 111+ messages in thread
From: Brad Campbell @ 2011-05-31  9:26 UTC (permalink / raw)
  To: Borislav Petkov, linux-kernel, kvm, linux-mm

On 31/05/11 13:47, Borislav Petkov wrote:
> Looks like a KSM issue. Disabling CONFIG_KSM should at least stop your
> machine from oopsing.
>
> Adding linux-mm.
>

I initially thought that, so the second panic was produced with KSM 
disabled from boot.

echo 0 > /sys/kernel/mm/ksm/run

If you still think that compiling ksm out of the kernel will prevent it 
then I'm willing to give it a go.

It's a production server, so I can only really bounce it around after 
about 9PM - GMT+8.

Regards,
Brad

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
  2011-05-31  9:26     ` Brad Campbell
@ 2011-05-31 10:38       ` Borislav Petkov
  -1 siblings, 0 replies; 111+ messages in thread
From: Borislav Petkov @ 2011-05-31 10:38 UTC (permalink / raw)
  To: Brad Campbell
  Cc: linux-kernel, kvm, linux-mm, Hugh Dickins, Andrea Arcangeli, Izik Eidus

On Tue, May 31, 2011 at 05:26:10PM +0800, Brad Campbell wrote:
> On 31/05/11 13:47, Borislav Petkov wrote:
> >Looks like a KSM issue. Disabling CONFIG_KSM should at least stop your
> >machine from oopsing.
> >
> >Adding linux-mm.
> >
> 
> I initially thought that, so the second panic was produced with KSM
> disabled from boot.
> 
> echo 0 > /sys/kernel/mm/ksm/run
> 
> If you still think that compiling ksm out of the kernel will prevent
> it then I'm willing to give it a go.

Ok, from looking at the code, when KSM inits, it starts the ksm kernel
thread and it looks like your oops comes from the function that is run
in the kernel thread - ksm_scan_thread.

So even if you disable it from sysfs, it runs at least once.

Let's add some more people to Cc and see what happens :).

-- 
Regards/Gruss,
Boris.

^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
@ 2011-05-31 10:38       ` Borislav Petkov
  0 siblings, 0 replies; 111+ messages in thread
From: Borislav Petkov @ 2011-05-31 10:38 UTC (permalink / raw)
  To: Brad Campbell
  Cc: linux-kernel, kvm, linux-mm, Hugh Dickins, Andrea Arcangeli, Izik Eidus

On Tue, May 31, 2011 at 05:26:10PM +0800, Brad Campbell wrote:
> On 31/05/11 13:47, Borislav Petkov wrote:
> >Looks like a KSM issue. Disabling CONFIG_KSM should at least stop your
> >machine from oopsing.
> >
> >Adding linux-mm.
> >
> 
> I initially thought that, so the second panic was produced with KSM
> disabled from boot.
> 
> echo 0 > /sys/kernel/mm/ksm/run
> 
> If you still think that compiling ksm out of the kernel will prevent
> it then I'm willing to give it a go.

Ok, from looking at the code, when KSM inits, it starts the ksm kernel
thread and it looks like your oops comes from the function that is run
in the kernel thread - ksm_scan_thread.

So even if you disable it from sysfs, it runs at least once.

Let's add some more people to Cc and see what happens :).

-- 
Regards/Gruss,
Boris.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
  2011-05-31 10:38       ` Borislav Petkov
  (?)
@ 2011-05-31 14:24         ` Brad Campbell
  -1 siblings, 0 replies; 111+ messages in thread
From: Brad Campbell @ 2011-05-31 14:24 UTC (permalink / raw)
  To: Borislav Petkov, linux-kernel, kvm, linux-mm, Hugh Dickins,
	Andrea Arcangeli, Izik Eidus

On 31/05/11 18:38, Borislav Petkov wrote:
> On Tue, May 31, 2011 at 05:26:10PM +0800, Brad Campbell wrote:
>> On 31/05/11 13:47, Borislav Petkov wrote:
>>> Looks like a KSM issue. Disabling CONFIG_KSM should at least stop your
>>> machine from oopsing.
>>>
>>> Adding linux-mm.
>>>
>>
>> I initially thought that, so the second panic was produced with KSM
>> disabled from boot.
>>
>> echo 0>  /sys/kernel/mm/ksm/run
>>
>> If you still think that compiling ksm out of the kernel will prevent
>> it then I'm willing to give it a go.
>
> Ok, from looking at the code, when KSM inits, it starts the ksm kernel
> thread and it looks like your oops comes from the function that is run
> in the kernel thread - ksm_scan_thread.
>
> So even if you disable it from sysfs, it runs at least once.
>

Just to confirm, I recompiled 2.6.38.7 without KSM enabled and I've been 
unable to reproduce the bug, so it looks like you were on the money.

I've moved back to 2.6.38.7 as 2.6.39 has a painful SCSI bug that panics 
about 75% of boots, and the reboot cycle required to get luck my way 
into a working kernel is just too much hassle.

It would appear that XP zero's its memory space on bootup, so there 
would be lots of pages to merge with a couple of relatively freshly 
booted XP machines running.

Regards,
Brad.

^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
@ 2011-05-31 14:24         ` Brad Campbell
  0 siblings, 0 replies; 111+ messages in thread
From: Brad Campbell @ 2011-05-31 14:24 UTC (permalink / raw)
  To: Borislav Petkov, linux-kernel, kvm, linux-mm, Hugh Dickins,
	Andrea Arcangeli

On 31/05/11 18:38, Borislav Petkov wrote:
> On Tue, May 31, 2011 at 05:26:10PM +0800, Brad Campbell wrote:
>> On 31/05/11 13:47, Borislav Petkov wrote:
>>> Looks like a KSM issue. Disabling CONFIG_KSM should at least stop your
>>> machine from oopsing.
>>>
>>> Adding linux-mm.
>>>
>>
>> I initially thought that, so the second panic was produced with KSM
>> disabled from boot.
>>
>> echo 0>  /sys/kernel/mm/ksm/run
>>
>> If you still think that compiling ksm out of the kernel will prevent
>> it then I'm willing to give it a go.
>
> Ok, from looking at the code, when KSM inits, it starts the ksm kernel
> thread and it looks like your oops comes from the function that is run
> in the kernel thread - ksm_scan_thread.
>
> So even if you disable it from sysfs, it runs at least once.
>

Just to confirm, I recompiled 2.6.38.7 without KSM enabled and I've been 
unable to reproduce the bug, so it looks like you were on the money.

I've moved back to 2.6.38.7 as 2.6.39 has a painful SCSI bug that panics 
about 75% of boots, and the reboot cycle required to get luck my way 
into a working kernel is just too much hassle.

It would appear that XP zero's its memory space on bootup, so there 
would be lots of pages to merge with a couple of relatively freshly 
booted XP machines running.

Regards,
Brad.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
@ 2011-05-31 14:24         ` Brad Campbell
  0 siblings, 0 replies; 111+ messages in thread
From: Brad Campbell @ 2011-05-31 14:24 UTC (permalink / raw)
  To: Borislav Petkov, linux-kernel, kvm, linux-mm, Hugh Dickins,
	Andrea Arcangeli, Izik Eidus

On 31/05/11 18:38, Borislav Petkov wrote:
> On Tue, May 31, 2011 at 05:26:10PM +0800, Brad Campbell wrote:
>> On 31/05/11 13:47, Borislav Petkov wrote:
>>> Looks like a KSM issue. Disabling CONFIG_KSM should at least stop your
>>> machine from oopsing.
>>>
>>> Adding linux-mm.
>>>
>>
>> I initially thought that, so the second panic was produced with KSM
>> disabled from boot.
>>
>> echo 0>  /sys/kernel/mm/ksm/run
>>
>> If you still think that compiling ksm out of the kernel will prevent
>> it then I'm willing to give it a go.
>
> Ok, from looking at the code, when KSM inits, it starts the ksm kernel
> thread and it looks like your oops comes from the function that is run
> in the kernel thread - ksm_scan_thread.
>
> So even if you disable it from sysfs, it runs at least once.
>

Just to confirm, I recompiled 2.6.38.7 without KSM enabled and I've been 
unable to reproduce the bug, so it looks like you were on the money.

I've moved back to 2.6.38.7 as 2.6.39 has a painful SCSI bug that panics 
about 75% of boots, and the reboot cycle required to get luck my way 
into a working kernel is just too much hassle.

It would appear that XP zero's its memory space on bootup, so there 
would be lots of pages to merge with a couple of relatively freshly 
booted XP machines running.

Regards,
Brad.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
  2011-05-31 14:24         ` Brad Campbell
@ 2011-05-31 22:31           ` Hugh Dickins
  -1 siblings, 0 replies; 111+ messages in thread
From: Hugh Dickins @ 2011-05-31 22:31 UTC (permalink / raw)
  To: Brad Campbell
  Cc: Borislav Petkov, linux-kernel, kvm, linux-mm, Andrea Arcangeli,
	Izik Eidus

On Tue, 31 May 2011, Brad Campbell wrote:
> On 31/05/11 18:38, Borislav Petkov wrote:
> > On Tue, May 31, 2011 at 05:26:10PM +0800, Brad Campbell wrote:
> > > On 31/05/11 13:47, Borislav Petkov wrote:
> > > > Looks like a KSM issue. Disabling CONFIG_KSM should at least stop your
> > > > machine from oopsing.
> > > > 
> > > > Adding linux-mm.
> > > > 
> > > 
> > > I initially thought that, so the second panic was produced with KSM
> > > disabled from boot.
> > > 
> > > echo 0>  /sys/kernel/mm/ksm/run
> > > 
> > > If you still think that compiling ksm out of the kernel will prevent
> > > it then I'm willing to give it a go.
> > 
> > Ok, from looking at the code, when KSM inits, it starts the ksm kernel
> > thread and it looks like your oops comes from the function that is run
> > in the kernel thread - ksm_scan_thread.
> > 
> > So even if you disable it from sysfs, it runs at least once.
> > 
> 
> Just to confirm, I recompiled 2.6.38.7 without KSM enabled and I've been
> unable to reproduce the bug, so it looks like you were on the money.
> 
> I've moved back to 2.6.38.7 as 2.6.39 has a painful SCSI bug that panics
> about 75% of boots, and the reboot cycle required to get luck my way into a
> working kernel is just too much hassle.
> 
> It would appear that XP zero's its memory space on bootup, so there would be
> lots of pages to merge with a couple of relatively freshly booted XP
> machines running.

Thanks for the Cc, Borislav.

Brad, my suspicion is that in each case the top 16 bits of RDX have been
mysteriously corrupted from ffff to 0000, causing the general protection
faults.  I don't understand what that has to do with KSM.

But it's only a suspicion, because I can't make sense of the "Code:"
lines in your traces, they have more than the expected 64 bytes, and
only one of them has a ">" (with no "<") to mark faulting instruction.

I did try compiling the 2.6.39 kernel from your config, but of course
we have different compilers, so although I got close, it wasn't exact.

Would you mind mailing me privately (it's about 73MB) the "objdump -trd"
output for your original vmlinux (with KSM on)?  (Those -trd options are
the ones I'm used to typing, I bet not they're not all relevant.)

Of course, it's only a tiny fraction of that output that I need,
might be better to cut it down to remove_rmap_item_from_tree and
dup_fd and ksm_scan_thread, if you have the time to do so.

Thanks,
Hugh

^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
@ 2011-05-31 22:31           ` Hugh Dickins
  0 siblings, 0 replies; 111+ messages in thread
From: Hugh Dickins @ 2011-05-31 22:31 UTC (permalink / raw)
  To: Brad Campbell
  Cc: Borislav Petkov, linux-kernel, kvm, linux-mm, Andrea Arcangeli,
	Izik Eidus

On Tue, 31 May 2011, Brad Campbell wrote:
> On 31/05/11 18:38, Borislav Petkov wrote:
> > On Tue, May 31, 2011 at 05:26:10PM +0800, Brad Campbell wrote:
> > > On 31/05/11 13:47, Borislav Petkov wrote:
> > > > Looks like a KSM issue. Disabling CONFIG_KSM should at least stop your
> > > > machine from oopsing.
> > > > 
> > > > Adding linux-mm.
> > > > 
> > > 
> > > I initially thought that, so the second panic was produced with KSM
> > > disabled from boot.
> > > 
> > > echo 0>  /sys/kernel/mm/ksm/run
> > > 
> > > If you still think that compiling ksm out of the kernel will prevent
> > > it then I'm willing to give it a go.
> > 
> > Ok, from looking at the code, when KSM inits, it starts the ksm kernel
> > thread and it looks like your oops comes from the function that is run
> > in the kernel thread - ksm_scan_thread.
> > 
> > So even if you disable it from sysfs, it runs at least once.
> > 
> 
> Just to confirm, I recompiled 2.6.38.7 without KSM enabled and I've been
> unable to reproduce the bug, so it looks like you were on the money.
> 
> I've moved back to 2.6.38.7 as 2.6.39 has a painful SCSI bug that panics
> about 75% of boots, and the reboot cycle required to get luck my way into a
> working kernel is just too much hassle.
> 
> It would appear that XP zero's its memory space on bootup, so there would be
> lots of pages to merge with a couple of relatively freshly booted XP
> machines running.

Thanks for the Cc, Borislav.

Brad, my suspicion is that in each case the top 16 bits of RDX have been
mysteriously corrupted from ffff to 0000, causing the general protection
faults.  I don't understand what that has to do with KSM.

But it's only a suspicion, because I can't make sense of the "Code:"
lines in your traces, they have more than the expected 64 bytes, and
only one of them has a ">" (with no "<") to mark faulting instruction.

I did try compiling the 2.6.39 kernel from your config, but of course
we have different compilers, so although I got close, it wasn't exact.

Would you mind mailing me privately (it's about 73MB) the "objdump -trd"
output for your original vmlinux (with KSM on)?  (Those -trd options are
the ones I'm used to typing, I bet not they're not all relevant.)

Of course, it's only a tiny fraction of that output that I need,
might be better to cut it down to remove_rmap_item_from_tree and
dup_fd and ksm_scan_thread, if you have the time to do so.

Thanks,
Hugh

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
  2011-05-31 22:31           ` Hugh Dickins
@ 2011-06-01  0:18             ` Brad Campbell
  -1 siblings, 0 replies; 111+ messages in thread
From: Brad Campbell @ 2011-06-01  0:18 UTC (permalink / raw)
  To: Hugh Dickins
  Cc: Borislav Petkov, linux-kernel, kvm, linux-mm, Andrea Arcangeli,
	Izik Eidus

On 01/06/11 06:31, Hugh Dickins wrote:
>
> Brad, my suspicion is that in each case the top 16 bits of RDX have been
> mysteriously corrupted from ffff to 0000, causing the general protection
> faults.  I don't understand what that has to do with KSM.

No, nor do I. The panic I reproduced with KSM off was in a completely 
unrelated code path. To be honest I would not be surprised if it turns 
out I have dodgy RAM, although it has passed multiple memtests and I've 
tried clocking it down. Just a gut feeling.

> But it's only a suspicion, because I can't make sense of the "Code:"
> lines in your traces, they have more than the expected 64 bytes, and
> only one of them has a ">" (with no"<") to mark faulting instruction.

Yeah, with hindsight I must have removed them when I re-formatted the 
code from the oops. Each byte was one line in the syslog so there was a 
lot of deleting to get it to a postable format.

> I did try compiling the 2.6.39 kernel from your config, but of course
> we have different compilers, so although I got close, it wasn't exact.
>
> Would you mind mailing me privately (it's about 73MB) the "objdump -trd"
> output for your original vmlinux (with KSM on)?  (Those -trd options are
> the ones I'm used to typing, I bet not they're not all relevant.)
>
> Of course, it's only a tiny fraction of that output that I need,
> might be better to cut it down to remove_rmap_item_from_tree and
> dup_fd and ksm_scan_thread, if you have the time to do so.

Ok, so since my initial posting I've figured out how to get a clean oops 
out of netconsole, so tonight (after 9PM GMT+8) I'll reproduce the oops 
a couple of times. What about I upload the oops, plus the vmlinux, plus 
.config and System.map to a server with a fat pipe and give you a link 
to it?

At least I can reproduce it quickly and easily.


^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
@ 2011-06-01  0:18             ` Brad Campbell
  0 siblings, 0 replies; 111+ messages in thread
From: Brad Campbell @ 2011-06-01  0:18 UTC (permalink / raw)
  To: Hugh Dickins
  Cc: Borislav Petkov, linux-kernel, kvm, linux-mm, Andrea Arcangeli,
	Izik Eidus

On 01/06/11 06:31, Hugh Dickins wrote:
>
> Brad, my suspicion is that in each case the top 16 bits of RDX have been
> mysteriously corrupted from ffff to 0000, causing the general protection
> faults.  I don't understand what that has to do with KSM.

No, nor do I. The panic I reproduced with KSM off was in a completely 
unrelated code path. To be honest I would not be surprised if it turns 
out I have dodgy RAM, although it has passed multiple memtests and I've 
tried clocking it down. Just a gut feeling.

> But it's only a suspicion, because I can't make sense of the "Code:"
> lines in your traces, they have more than the expected 64 bytes, and
> only one of them has a ">" (with no"<") to mark faulting instruction.

Yeah, with hindsight I must have removed them when I re-formatted the 
code from the oops. Each byte was one line in the syslog so there was a 
lot of deleting to get it to a postable format.

> I did try compiling the 2.6.39 kernel from your config, but of course
> we have different compilers, so although I got close, it wasn't exact.
>
> Would you mind mailing me privately (it's about 73MB) the "objdump -trd"
> output for your original vmlinux (with KSM on)?  (Those -trd options are
> the ones I'm used to typing, I bet not they're not all relevant.)
>
> Of course, it's only a tiny fraction of that output that I need,
> might be better to cut it down to remove_rmap_item_from_tree and
> dup_fd and ksm_scan_thread, if you have the time to do so.

Ok, so since my initial posting I've figured out how to get a clean oops 
out of netconsole, so tonight (after 9PM GMT+8) I'll reproduce the oops 
a couple of times. What about I upload the oops, plus the vmlinux, plus 
.config and System.map to a server with a fat pipe and give you a link 
to it?

At least I can reproduce it quickly and easily.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
  2011-05-31 22:31           ` Hugh Dickins
@ 2011-06-01  0:37             ` Brad Campbell
  -1 siblings, 0 replies; 111+ messages in thread
From: Brad Campbell @ 2011-06-01  0:37 UTC (permalink / raw)
  To: Hugh Dickins
  Cc: Borislav Petkov, linux-kernel, kvm, linux-mm, Andrea Arcangeli,
	Izik Eidus

On 01/06/11 06:31, Hugh Dickins wrote:
> Brad, my suspicion is that in each case the top 16 bits of RDX have been
> mysteriously corrupted from ffff to 0000, causing the general protection
> faults.  I don't understand what that has to do with KSM.
>
> But it's only a suspicion, because I can't make sense of the "Code:"
> lines in your traces, they have more than the expected 64 bytes, and
> only one of them has a ">" (with no"<") to mark faulting instruction.
>
> I did try compiling the 2.6.39 kernel from your config, but of course
> we have different compilers, so although I got close, it wasn't exact.
>
> Would you mind mailing me privately (it's about 73MB) the "objdump -trd"
> output for your original vmlinux (with KSM on)?  (Those -trd options are
> the ones I'm used to typing, I bet not they're not all relevant.)
>
> Of course, it's only a tiny fraction of that output that I need,
> might be better to cut it down to remove_rmap_item_from_tree and
> dup_fd and ksm_scan_thread, if you have the time to do so.

Would you believe about 20 seconds after I pressed send the kernel oopsed.

http://www.fnarfbargle.com/private/003_kernel_oops/

oops reproduced here, but an un-munged version is in that directory 
alongside the kernel.

[36542.880228] general protection fault: 0000 [#1] SMP
[36542.880271] last sysfs file: 
/sys/devices/pci0000:00/0000:00:18.3/temp1_input
[36542.880290] CPU 4
[36542.880301] Modules linked in: xt_iprange xt_DSCP xt_length 
xt_CLASSIFY sch_sfq xt_CHECKSUM ipt_REJECT ipt_MASQUER
ADE ipt_REDIRECT xt_recent xt_state iptable_filter iptable_nat nf_nat 
nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 x
t_TCPMSS xt_tcpmss xt_tcpudp iptable_mangle ip_tables x_tables pppoe 
pppox ppp_generic slhc cls_u32 sch_htb deflate z
lib_deflate des_generic cbc ecb crypto_blkcipher sha1_generic md5 hmac 
crypto_hash cryptomgr aead crypto_algapi af_ke
y fuse hwmon_vid netconsole configfs vhost_net powernow_k8 mperf kvm_amd 
kvm pl2303 usbserial xhci_hcd k10temp i2c_pi
ix4 ahci usb_storage usb_libusual ohci_hcd ehci_hcd r8169 libahci 
usbcore mii sata_mv megaraid_sas [last unloaded: sc
si_wait_scan]
[36542.880842]
[36542.880858] Pid: 13346, comm: bash Not tainted 2.6.38.7 #29 To Be 
Filled By O.E.M. To Be Filled By O.E.M./880G Ext
reme3
[36542.880911] RIP: 0010:[<ffffffff810cf0de>]  [<ffffffff810cf0de>] 
do_vfs_ioctl+0x5e/0x510
[36542.880948] RSP: 0018:ffff8802d25a1ec8  EFLAGS: 00010206
[36542.880965] RAX: fffffffffffffff7 RBX: 000088040eb12840 RCX: 
00007fff4fe4a4c0
[36542.880984] RDX: 0000000000005413 RSI: 0000000000005413 RDI: 
00000000000000ff
[36542.881002] RBP: 00000000000000ff R08: 00007fff4fe4a400 R09: 
0000000000000000
[36542.881020] R10: 00007fff4fe4a380 R11: 0000000000000246 R12: 
00007fff4fe4a4c0
[36542.881038] R13: 00007fff4fe4a4c0 R14: 0000000000000000 R15: 
0000000000000001
[36542.881058] FS:  00007f65f725b700(0000) GS:ffff8800dbd00000(0000) 
knlGS:0000000000000000
[36542.881081] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[36542.881098] CR2: 0000000001f01008 CR3: 00000002d25c3000 CR4: 
00000000000006e0
[36542.881116] DR0: 00000000000000a0 DR1: 0000000000000000 DR2: 
0000000000000003
[36542.881133] DR3: 00000000000000b0 DR6: 00000000ffff0ff0 DR7: 
0000000000000400
[36542.881152] Process bash (pid: 13346, threadinfo ffff8802d25a0000, 
task ffff88041df88000)
[36542.881172] Stack:
[36542.881183]  0000000000000000 ffff88041df88218 0000000000100000 
0000000000000001
[36542.881225]  0000000000000002 00007fff4fe4a2c0 00007fff4fe4a220 
0000000000000002
[36542.881268]  0000000000000000 ffffffff81046d6a 000088040eb12840 
00000000000000ff
[36542.881312] Call Trace:
[36542.881333]  [<ffffffff81046d6a>] ? sys_rt_sigaction+0x8a/0xc0
[36542.881351]  [<ffffffff810cf5d9>] ? sys_ioctl+0x49/0x80
[36542.881373]  [<ffffffff810023fb>] ? system_call_fastpath+0x16/0x1b
[36542.881389] Code: 76 7b 81 fa 77 58 04 c0 0f 84 77 01 00 00 0f 1f 80 
00 00 00 00 0f 87 a2 00 00 00 81 fa 60 54 00 00 0f 1f 40 00 0f 84 ba 01 
00 00 <48> 8b 43 18 48 8b 50 30 0f b7 02 25 00 f0 00 00 3d 00 80 00 00
[36542.881793] RIP  [<ffffffff810cf0de>] do_vfs_ioctl+0x5e/0x510
[36542.881818]  RSP <ffff8802d25a1ec8>
[36542.882082] ---[ end trace 1b8d730cd479e388 ]---
[36542.882126] Kernel panic - not syncing: Fatal exception
[36542.882175] Pid: 13346, comm: bash Tainted: G      D     2.6.38.7 #29
[36542.882222] Call Trace:
[36542.882269]  [<ffffffff813c7f42>] ? panic+0x92/0x18a
[36542.882318]  [<ffffffff81039a41>] ? kmsg_dump+0x41/0xf0
[36542.882366]  [<ffffffff810062bd>] ? oops_end+0x8d/0xa0
[36542.882414]  [<ffffffff813caeef>] ? general_protection+0x1f/0x30
[36542.882463]  [<ffffffff810cf0de>] ? do_vfs_ioctl+0x5e/0x510
[36542.882511]  [<ffffffff81046d6a>] ? sys_rt_sigaction+0x8a/0xc0
[36542.882560]  [<ffffffff810cf5d9>] ? sys_ioctl+0x49/0x80
[36542.882608]  [<ffffffff810023fb>] ? system_call_fastpath+0x16/0x1b
[36542.882688] Rebooting in 60 seconds..[   33.104725] fuse init (API 
version 7.16)

^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
@ 2011-06-01  0:37             ` Brad Campbell
  0 siblings, 0 replies; 111+ messages in thread
From: Brad Campbell @ 2011-06-01  0:37 UTC (permalink / raw)
  To: Hugh Dickins
  Cc: Borislav Petkov, linux-kernel, kvm, linux-mm, Andrea Arcangeli,
	Izik Eidus

On 01/06/11 06:31, Hugh Dickins wrote:
> Brad, my suspicion is that in each case the top 16 bits of RDX have been
> mysteriously corrupted from ffff to 0000, causing the general protection
> faults.  I don't understand what that has to do with KSM.
>
> But it's only a suspicion, because I can't make sense of the "Code:"
> lines in your traces, they have more than the expected 64 bytes, and
> only one of them has a ">" (with no"<") to mark faulting instruction.
>
> I did try compiling the 2.6.39 kernel from your config, but of course
> we have different compilers, so although I got close, it wasn't exact.
>
> Would you mind mailing me privately (it's about 73MB) the "objdump -trd"
> output for your original vmlinux (with KSM on)?  (Those -trd options are
> the ones I'm used to typing, I bet not they're not all relevant.)
>
> Of course, it's only a tiny fraction of that output that I need,
> might be better to cut it down to remove_rmap_item_from_tree and
> dup_fd and ksm_scan_thread, if you have the time to do so.

Would you believe about 20 seconds after I pressed send the kernel oopsed.

http://www.fnarfbargle.com/private/003_kernel_oops/

oops reproduced here, but an un-munged version is in that directory 
alongside the kernel.

[36542.880228] general protection fault: 0000 [#1] SMP
[36542.880271] last sysfs file: 
/sys/devices/pci0000:00/0000:00:18.3/temp1_input
[36542.880290] CPU 4
[36542.880301] Modules linked in: xt_iprange xt_DSCP xt_length 
xt_CLASSIFY sch_sfq xt_CHECKSUM ipt_REJECT ipt_MASQUER
ADE ipt_REDIRECT xt_recent xt_state iptable_filter iptable_nat nf_nat 
nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 x
t_TCPMSS xt_tcpmss xt_tcpudp iptable_mangle ip_tables x_tables pppoe 
pppox ppp_generic slhc cls_u32 sch_htb deflate z
lib_deflate des_generic cbc ecb crypto_blkcipher sha1_generic md5 hmac 
crypto_hash cryptomgr aead crypto_algapi af_ke
y fuse hwmon_vid netconsole configfs vhost_net powernow_k8 mperf kvm_amd 
kvm pl2303 usbserial xhci_hcd k10temp i2c_pi
ix4 ahci usb_storage usb_libusual ohci_hcd ehci_hcd r8169 libahci 
usbcore mii sata_mv megaraid_sas [last unloaded: sc
si_wait_scan]
[36542.880842]
[36542.880858] Pid: 13346, comm: bash Not tainted 2.6.38.7 #29 To Be 
Filled By O.E.M. To Be Filled By O.E.M./880G Ext
reme3
[36542.880911] RIP: 0010:[<ffffffff810cf0de>]  [<ffffffff810cf0de>] 
do_vfs_ioctl+0x5e/0x510
[36542.880948] RSP: 0018:ffff8802d25a1ec8  EFLAGS: 00010206
[36542.880965] RAX: fffffffffffffff7 RBX: 000088040eb12840 RCX: 
00007fff4fe4a4c0
[36542.880984] RDX: 0000000000005413 RSI: 0000000000005413 RDI: 
00000000000000ff
[36542.881002] RBP: 00000000000000ff R08: 00007fff4fe4a400 R09: 
0000000000000000
[36542.881020] R10: 00007fff4fe4a380 R11: 0000000000000246 R12: 
00007fff4fe4a4c0
[36542.881038] R13: 00007fff4fe4a4c0 R14: 0000000000000000 R15: 
0000000000000001
[36542.881058] FS:  00007f65f725b700(0000) GS:ffff8800dbd00000(0000) 
knlGS:0000000000000000
[36542.881081] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[36542.881098] CR2: 0000000001f01008 CR3: 00000002d25c3000 CR4: 
00000000000006e0
[36542.881116] DR0: 00000000000000a0 DR1: 0000000000000000 DR2: 
0000000000000003
[36542.881133] DR3: 00000000000000b0 DR6: 00000000ffff0ff0 DR7: 
0000000000000400
[36542.881152] Process bash (pid: 13346, threadinfo ffff8802d25a0000, 
task ffff88041df88000)
[36542.881172] Stack:
[36542.881183]  0000000000000000 ffff88041df88218 0000000000100000 
0000000000000001
[36542.881225]  0000000000000002 00007fff4fe4a2c0 00007fff4fe4a220 
0000000000000002
[36542.881268]  0000000000000000 ffffffff81046d6a 000088040eb12840 
00000000000000ff
[36542.881312] Call Trace:
[36542.881333]  [<ffffffff81046d6a>] ? sys_rt_sigaction+0x8a/0xc0
[36542.881351]  [<ffffffff810cf5d9>] ? sys_ioctl+0x49/0x80
[36542.881373]  [<ffffffff810023fb>] ? system_call_fastpath+0x16/0x1b
[36542.881389] Code: 76 7b 81 fa 77 58 04 c0 0f 84 77 01 00 00 0f 1f 80 
00 00 00 00 0f 87 a2 00 00 00 81 fa 60 54 00 00 0f 1f 40 00 0f 84 ba 01 
00 00 <48> 8b 43 18 48 8b 50 30 0f b7 02 25 00 f0 00 00 3d 00 80 00 00
[36542.881793] RIP  [<ffffffff810cf0de>] do_vfs_ioctl+0x5e/0x510
[36542.881818]  RSP <ffff8802d25a1ec8>
[36542.882082] ---[ end trace 1b8d730cd479e388 ]---
[36542.882126] Kernel panic - not syncing: Fatal exception
[36542.882175] Pid: 13346, comm: bash Tainted: G      D     2.6.38.7 #29
[36542.882222] Call Trace:
[36542.882269]  [<ffffffff813c7f42>] ? panic+0x92/0x18a
[36542.882318]  [<ffffffff81039a41>] ? kmsg_dump+0x41/0xf0
[36542.882366]  [<ffffffff810062bd>] ? oops_end+0x8d/0xa0
[36542.882414]  [<ffffffff813caeef>] ? general_protection+0x1f/0x30
[36542.882463]  [<ffffffff810cf0de>] ? do_vfs_ioctl+0x5e/0x510
[36542.882511]  [<ffffffff81046d6a>] ? sys_rt_sigaction+0x8a/0xc0
[36542.882560]  [<ffffffff810cf5d9>] ? sys_ioctl+0x49/0x80
[36542.882608]  [<ffffffff810023fb>] ? system_call_fastpath+0x16/0x1b
[36542.882688] Rebooting in 60 seconds..[   33.104725] fuse init (API 
version 7.16)

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
  2011-06-01  0:37             ` Brad Campbell
@ 2011-06-01  1:15               ` Andrea Arcangeli
  -1 siblings, 0 replies; 111+ messages in thread
From: Andrea Arcangeli @ 2011-06-01  1:15 UTC (permalink / raw)
  To: Brad Campbell
  Cc: Hugh Dickins, Borislav Petkov, linux-kernel, kvm, linux-mm, Izik Eidus

Hello,

On Wed, Jun 01, 2011 at 08:37:25AM +0800, Brad Campbell wrote:
> On 01/06/11 06:31, Hugh Dickins wrote:
> > Brad, my suspicion is that in each case the top 16 bits of RDX have been
> > mysteriously corrupted from ffff to 0000, causing the general protection
> > faults.  I don't understand what that has to do with KSM.
> >
> > But it's only a suspicion, because I can't make sense of the "Code:"
> > lines in your traces, they have more than the expected 64 bytes, and
> > only one of them has a ">" (with no"<") to mark faulting instruction.
> >
> > I did try compiling the 2.6.39 kernel from your config, but of course
> > we have different compilers, so although I got close, it wasn't exact.
> >
> > Would you mind mailing me privately (it's about 73MB) the "objdump -trd"
> > output for your original vmlinux (with KSM on)?  (Those -trd options are
> > the ones I'm used to typing, I bet not they're not all relevant.)
> >
> > Of course, it's only a tiny fraction of that output that I need,
> > might be better to cut it down to remove_rmap_item_from_tree and
> > dup_fd and ksm_scan_thread, if you have the time to do so.
> 
> Would you believe about 20 seconds after I pressed send the kernel oopsed.
> 
> http://www.fnarfbargle.com/private/003_kernel_oops/
> 
> oops reproduced here, but an un-munged version is in that directory 
> alongside the kernel.
> 
> [36542.880228] general protection fault: 0000 [#1] SMP

Reminds me of another oops that was reported on the kvm list for
2.6.38.1 with message id 4D8C6110.6090204. There the top 16 bits of
rsi were flipped and it was a general protection too because of
hitting on the not mappable virtual range.

http://www.virtall.com/files/temp/kvm.txt
http://www.virtall.com/files/temp/config-2.6.38.1
http://virtall.com/files/temp/mmu-objdump.txt

That oops happened in kvm_unmap_rmapp though, but it looked memory
corruption (Avi suggested use after free) but it was a production
system so we couldn't debug it further.

I recommend next thing to reproduce again with 2.6.39 or
3.0.0-rc1. Let's fix your scsi trouble if needed but it's better you
test with 2.6.39.

We'd need chmod +r vmlinux on private/003_kernel_oops/

Thanks,
Andrea

^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
@ 2011-06-01  1:15               ` Andrea Arcangeli
  0 siblings, 0 replies; 111+ messages in thread
From: Andrea Arcangeli @ 2011-06-01  1:15 UTC (permalink / raw)
  To: Brad Campbell
  Cc: Hugh Dickins, Borislav Petkov, linux-kernel, kvm, linux-mm, Izik Eidus

Hello,

On Wed, Jun 01, 2011 at 08:37:25AM +0800, Brad Campbell wrote:
> On 01/06/11 06:31, Hugh Dickins wrote:
> > Brad, my suspicion is that in each case the top 16 bits of RDX have been
> > mysteriously corrupted from ffff to 0000, causing the general protection
> > faults.  I don't understand what that has to do with KSM.
> >
> > But it's only a suspicion, because I can't make sense of the "Code:"
> > lines in your traces, they have more than the expected 64 bytes, and
> > only one of them has a ">" (with no"<") to mark faulting instruction.
> >
> > I did try compiling the 2.6.39 kernel from your config, but of course
> > we have different compilers, so although I got close, it wasn't exact.
> >
> > Would you mind mailing me privately (it's about 73MB) the "objdump -trd"
> > output for your original vmlinux (with KSM on)?  (Those -trd options are
> > the ones I'm used to typing, I bet not they're not all relevant.)
> >
> > Of course, it's only a tiny fraction of that output that I need,
> > might be better to cut it down to remove_rmap_item_from_tree and
> > dup_fd and ksm_scan_thread, if you have the time to do so.
> 
> Would you believe about 20 seconds after I pressed send the kernel oopsed.
> 
> http://www.fnarfbargle.com/private/003_kernel_oops/
> 
> oops reproduced here, but an un-munged version is in that directory 
> alongside the kernel.
> 
> [36542.880228] general protection fault: 0000 [#1] SMP

Reminds me of another oops that was reported on the kvm list for
2.6.38.1 with message id 4D8C6110.6090204. There the top 16 bits of
rsi were flipped and it was a general protection too because of
hitting on the not mappable virtual range.

http://www.virtall.com/files/temp/kvm.txt
http://www.virtall.com/files/temp/config-2.6.38.1
http://virtall.com/files/temp/mmu-objdump.txt

That oops happened in kvm_unmap_rmapp though, but it looked memory
corruption (Avi suggested use after free) but it was a production
system so we couldn't debug it further.

I recommend next thing to reproduce again with 2.6.39 or
3.0.0-rc1. Let's fix your scsi trouble if needed but it's better you
test with 2.6.39.

We'd need chmod +r vmlinux on private/003_kernel_oops/

Thanks,
Andrea

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
  2011-06-01  1:15               ` Andrea Arcangeli
@ 2011-06-01  2:03                 ` Brad Campbell
  -1 siblings, 0 replies; 111+ messages in thread
From: Brad Campbell @ 2011-06-01  2:03 UTC (permalink / raw)
  To: Andrea Arcangeli
  Cc: Hugh Dickins, Borislav Petkov, linux-kernel, kvm, linux-mm, Izik Eidus

On 01/06/11 09:15, Andrea Arcangeli wrote:
> Hello,
>
> On Wed, Jun 01, 2011 at 08:37:25AM +0800, Brad Campbell wrote:
>> On 01/06/11 06:31, Hugh Dickins wrote:
>>> Brad, my suspicion is that in each case the top 16 bits of RDX have been
>>> mysteriously corrupted from ffff to 0000, causing the general protection
>>> faults.  I don't understand what that has to do with KSM.
>>>
>>> But it's only a suspicion, because I can't make sense of the "Code:"
>>> lines in your traces, they have more than the expected 64 bytes, and
>>> only one of them has a ">" (with no"<") to mark faulting instruction.
>>>
>>> I did try compiling the 2.6.39 kernel from your config, but of course
>>> we have different compilers, so although I got close, it wasn't exact.
>>>
>>> Would you mind mailing me privately (it's about 73MB) the "objdump -trd"
>>> output for your original vmlinux (with KSM on)?  (Those -trd options are
>>> the ones I'm used to typing, I bet not they're not all relevant.)
>>>
>>> Of course, it's only a tiny fraction of that output that I need,
>>> might be better to cut it down to remove_rmap_item_from_tree and
>>> dup_fd and ksm_scan_thread, if you have the time to do so.
>>
>> Would you believe about 20 seconds after I pressed send the kernel oopsed.
>>
>> http://www.fnarfbargle.com/private/003_kernel_oops/
>>
>> oops reproduced here, but an un-munged version is in that directory
>> alongside the kernel.
>>
>> [36542.880228] general protection fault: 0000 [#1] SMP
>
> Reminds me of another oops that was reported on the kvm list for
> 2.6.38.1 with message id 4D8C6110.6090204. There the top 16 bits of
> rsi were flipped and it was a general protection too because of
> hitting on the not mappable virtual range.
>
> http://www.virtall.com/files/temp/kvm.txt
> http://www.virtall.com/files/temp/config-2.6.38.1
> http://virtall.com/files/temp/mmu-objdump.txt
>
> That oops happened in kvm_unmap_rmapp though, but it looked memory
> corruption (Avi suggested use after free) but it was a production
> system so we couldn't debug it further.
>
> I recommend next thing to reproduce again with 2.6.39 or
> 3.0.0-rc1. Let's fix your scsi trouble if needed but it's better you
> test with 2.6.39.
>
> We'd need chmod +r vmlinux on private/003_kernel_oops/

Ok, here we go then.

http://www.fnarfbargle.com/private/004_kernel_oops/

The permissions are right this time.
2.6.39 + KSM

[  694.227866] general protection fault: 0000 [#1] SMP
[  694.228001] last sysfs file: /sys/devices/platform/w83627ehf.656/cpu0_vid
[  694.228050] CPU 3
[  694.228091] Modules linked in: xt_iprange xt_DSCP xt_length 
xt_CLASSIFY sch_sfq xt_CHECKSUM ipt_REJECT ipt_MASQUERADE ipt_REDIRECT 
xt_recent xt_state iptable_filter iptable_nat nf_nat nf_conntrack_ipv4 
nf_conntrack nf_defrag_ipv4 xt_TCPMSS xt_tcpmss xt_tcpudp iptable_mangle 
ip_tables x_tables pppoe pppox ppp_generic slhc cls_u32 sch_htb deflate 
zlib_deflate des_generic cbc ecb crypto_blkcipher sha1_generic md5 hmac 
crypto_hash cryptomgr aead crypto_algapi af_key fuse w83627ehf hwmon_vid 
netconsole configfs vhost_net powernow_k8 mperf kvm_amd kvm pl2303 
usbserial i2c_piix4 k10temp xhci_hcd usb_storage usb_libusual ohci_hcd 
r8169 ehci_hcd ahci usbcore sata_mv mii libahci megaraid_sas [last 
unloaded: scsi_wait_scan]
[  694.230897]
[  694.230944] Pid: 11841, comm: keepalive Not tainted 2.6.39 #3 To Be 
Filled By O.E.M. To Be Filled By O.E.M./880G Extreme3
[  694.231111] RIP: 0010:[<ffffffff810db878>]  [<ffffffff810db878>] 
dup_fd+0x168/0x300
[  694.231210] RSP: 0018:ffff8802f524fdd0  EFLAGS: 00010206
[  694.231258] RAX: 00000000000007f8 RBX: ffff8802f5721b80 RCX: 
bfffffffffffffff
[  694.231308] RDX: 00008802f51cacc0 RSI: 00000000000000ff RDI: 
0000000000000800
[  694.231358] RBP: ffff8803bf419800 R08: ffff88030167f6c0 R09: 
0000000000000003
[  694.231407] R10: 0000000000000001 R11: 4000000000000000 R12: 
0000000000000100
[  694.231457] R13: ffff880417aa9800 R14: ffff88030167f440 R15: 
ffff8803bd8c1600
[  694.231507] FS:  00007f02cfc32700(0000) GS:ffff88041fcc0000(0000) 
knlGS:0000000000000000
[  694.231560] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[  694.231609] CR2: 00007f02cf5d4810 CR3: 00000002f52c3000 CR4: 
00000000000006e0
[  694.231657] DR0: 0000000000000045 DR1: 0000000000000000 DR2: 
0000000000000000
[  694.231707] DR3: 0000000000000005 DR6: 00000000ffff0ff0 DR7: 
0000000000000400
[  694.231757] Process keepalive (pid: 11841, threadinfo 
ffff8802f524e000, task ffff8802f5143690)
[  694.231809] Stack:
[  694.231852]  ffff8802f5143690 0000000000000020 ffff8802f56badc0 
ffff8802f5721b90
[  694.232050]  ffff880417aa54e0 0000000001200011 ffff880417aa54e0 
0000000000000000
[  694.232248]  00007f02cfc329d0 ffff8802f5143690 0000000000000000 
ffffffff81037645
[  694.232448] Call Trace:
[  694.232499]  [<ffffffff81037645>] ? copy_process+0xa75/0xfd0
[  694.232549]  [<ffffffff81037c0d>] ? do_fork+0x6d/0x2b0
[  694.232599]  [<ffffffff810457a9>] ? sigprocmask+0x69/0x100
[  694.232651]  [<ffffffff813d0ca3>] ? stub_clone+0x13/0x20
[  694.232699]  [<ffffffff813d0a3b>] ? system_call_fastpath+0x16/0x1b
[  694.232745] Code: 4c 89 c2 e8 6b e5 0f 00 45 85 e4 74 78 41 8d 44 24 
ff 31 f6 41 ba 01 00 00 00 48 8d 3c c5 08 00 00 00 31 c0 eb 1a 0f 1f 44 
00 00 <f0> 48 ff 42 30 48 89 54 05 00 48 83 c0 08 ff c6 48 39 f8 74 3b
[  694.235190] RIP  [<ffffffff810db878>] dup_fd+0x168/0x300
[  694.235282]  RSP <ffff8802f524fdd0>
[  694.235379] ---[ end trace 949fad05591fcdb3 ]---
[  694.235428] Kernel panic - not syncing: Fatal exception
[  694.235478] Pid: 11841, comm: keepalive Tainted: G      D     2.6.39 #3
[  694.235525] Call Trace:
[  694.235573]  [<ffffffff813cd6f5>] ? panic+0x92/0x18a
[  694.235624]  [<ffffffff81038b61>] ? kmsg_dump+0x41/0xf0
[  694.235676]  [<ffffffff810050ad>] ? oops_end+0x8d/0xa0
[  694.235726]  [<ffffffff813d05ef>] ? general_protection+0x1f/0x30
[  694.235778]  [<ffffffff810db878>] ? dup_fd+0x168/0x300
[  694.235827]  [<ffffffff81037645>] ? copy_process+0xa75/0xfd0
[  694.235877]  [<ffffffff81037c0d>] ? do_fork+0x6d/0x2b0
[  694.235926]  [<ffffffff810457a9>] ? sigprocmask+0x69/0x100
[  694.235978]  [<ffffffff813d0ca3>] ? stub_clone+0x13/0x20
[  694.236028]  [<ffffffff813d0a3b>] ? system_call_fastpath+0x16/0x1b
[  694.236083] Rebooting in 60 seconds..

^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
@ 2011-06-01  2:03                 ` Brad Campbell
  0 siblings, 0 replies; 111+ messages in thread
From: Brad Campbell @ 2011-06-01  2:03 UTC (permalink / raw)
  To: Andrea Arcangeli
  Cc: Hugh Dickins, Borislav Petkov, linux-kernel, kvm, linux-mm, Izik Eidus

On 01/06/11 09:15, Andrea Arcangeli wrote:
> Hello,
>
> On Wed, Jun 01, 2011 at 08:37:25AM +0800, Brad Campbell wrote:
>> On 01/06/11 06:31, Hugh Dickins wrote:
>>> Brad, my suspicion is that in each case the top 16 bits of RDX have been
>>> mysteriously corrupted from ffff to 0000, causing the general protection
>>> faults.  I don't understand what that has to do with KSM.
>>>
>>> But it's only a suspicion, because I can't make sense of the "Code:"
>>> lines in your traces, they have more than the expected 64 bytes, and
>>> only one of them has a ">" (with no"<") to mark faulting instruction.
>>>
>>> I did try compiling the 2.6.39 kernel from your config, but of course
>>> we have different compilers, so although I got close, it wasn't exact.
>>>
>>> Would you mind mailing me privately (it's about 73MB) the "objdump -trd"
>>> output for your original vmlinux (with KSM on)?  (Those -trd options are
>>> the ones I'm used to typing, I bet not they're not all relevant.)
>>>
>>> Of course, it's only a tiny fraction of that output that I need,
>>> might be better to cut it down to remove_rmap_item_from_tree and
>>> dup_fd and ksm_scan_thread, if you have the time to do so.
>>
>> Would you believe about 20 seconds after I pressed send the kernel oopsed.
>>
>> http://www.fnarfbargle.com/private/003_kernel_oops/
>>
>> oops reproduced here, but an un-munged version is in that directory
>> alongside the kernel.
>>
>> [36542.880228] general protection fault: 0000 [#1] SMP
>
> Reminds me of another oops that was reported on the kvm list for
> 2.6.38.1 with message id 4D8C6110.6090204. There the top 16 bits of
> rsi were flipped and it was a general protection too because of
> hitting on the not mappable virtual range.
>
> http://www.virtall.com/files/temp/kvm.txt
> http://www.virtall.com/files/temp/config-2.6.38.1
> http://virtall.com/files/temp/mmu-objdump.txt
>
> That oops happened in kvm_unmap_rmapp though, but it looked memory
> corruption (Avi suggested use after free) but it was a production
> system so we couldn't debug it further.
>
> I recommend next thing to reproduce again with 2.6.39 or
> 3.0.0-rc1. Let's fix your scsi trouble if needed but it's better you
> test with 2.6.39.
>
> We'd need chmod +r vmlinux on private/003_kernel_oops/

Ok, here we go then.

http://www.fnarfbargle.com/private/004_kernel_oops/

The permissions are right this time.
2.6.39 + KSM

[  694.227866] general protection fault: 0000 [#1] SMP
[  694.228001] last sysfs file: /sys/devices/platform/w83627ehf.656/cpu0_vid
[  694.228050] CPU 3
[  694.228091] Modules linked in: xt_iprange xt_DSCP xt_length 
xt_CLASSIFY sch_sfq xt_CHECKSUM ipt_REJECT ipt_MASQUERADE ipt_REDIRECT 
xt_recent xt_state iptable_filter iptable_nat nf_nat nf_conntrack_ipv4 
nf_conntrack nf_defrag_ipv4 xt_TCPMSS xt_tcpmss xt_tcpudp iptable_mangle 
ip_tables x_tables pppoe pppox ppp_generic slhc cls_u32 sch_htb deflate 
zlib_deflate des_generic cbc ecb crypto_blkcipher sha1_generic md5 hmac 
crypto_hash cryptomgr aead crypto_algapi af_key fuse w83627ehf hwmon_vid 
netconsole configfs vhost_net powernow_k8 mperf kvm_amd kvm pl2303 
usbserial i2c_piix4 k10temp xhci_hcd usb_storage usb_libusual ohci_hcd 
r8169 ehci_hcd ahci usbcore sata_mv mii libahci megaraid_sas [last 
unloaded: scsi_wait_scan]
[  694.230897]
[  694.230944] Pid: 11841, comm: keepalive Not tainted 2.6.39 #3 To Be 
Filled By O.E.M. To Be Filled By O.E.M./880G Extreme3
[  694.231111] RIP: 0010:[<ffffffff810db878>]  [<ffffffff810db878>] 
dup_fd+0x168/0x300
[  694.231210] RSP: 0018:ffff8802f524fdd0  EFLAGS: 00010206
[  694.231258] RAX: 00000000000007f8 RBX: ffff8802f5721b80 RCX: 
bfffffffffffffff
[  694.231308] RDX: 00008802f51cacc0 RSI: 00000000000000ff RDI: 
0000000000000800
[  694.231358] RBP: ffff8803bf419800 R08: ffff88030167f6c0 R09: 
0000000000000003
[  694.231407] R10: 0000000000000001 R11: 4000000000000000 R12: 
0000000000000100
[  694.231457] R13: ffff880417aa9800 R14: ffff88030167f440 R15: 
ffff8803bd8c1600
[  694.231507] FS:  00007f02cfc32700(0000) GS:ffff88041fcc0000(0000) 
knlGS:0000000000000000
[  694.231560] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[  694.231609] CR2: 00007f02cf5d4810 CR3: 00000002f52c3000 CR4: 
00000000000006e0
[  694.231657] DR0: 0000000000000045 DR1: 0000000000000000 DR2: 
0000000000000000
[  694.231707] DR3: 0000000000000005 DR6: 00000000ffff0ff0 DR7: 
0000000000000400
[  694.231757] Process keepalive (pid: 11841, threadinfo 
ffff8802f524e000, task ffff8802f5143690)
[  694.231809] Stack:
[  694.231852]  ffff8802f5143690 0000000000000020 ffff8802f56badc0 
ffff8802f5721b90
[  694.232050]  ffff880417aa54e0 0000000001200011 ffff880417aa54e0 
0000000000000000
[  694.232248]  00007f02cfc329d0 ffff8802f5143690 0000000000000000 
ffffffff81037645
[  694.232448] Call Trace:
[  694.232499]  [<ffffffff81037645>] ? copy_process+0xa75/0xfd0
[  694.232549]  [<ffffffff81037c0d>] ? do_fork+0x6d/0x2b0
[  694.232599]  [<ffffffff810457a9>] ? sigprocmask+0x69/0x100
[  694.232651]  [<ffffffff813d0ca3>] ? stub_clone+0x13/0x20
[  694.232699]  [<ffffffff813d0a3b>] ? system_call_fastpath+0x16/0x1b
[  694.232745] Code: 4c 89 c2 e8 6b e5 0f 00 45 85 e4 74 78 41 8d 44 24 
ff 31 f6 41 ba 01 00 00 00 48 8d 3c c5 08 00 00 00 31 c0 eb 1a 0f 1f 44 
00 00 <f0> 48 ff 42 30 48 89 54 05 00 48 83 c0 08 ff c6 48 39 f8 74 3b
[  694.235190] RIP  [<ffffffff810db878>] dup_fd+0x168/0x300
[  694.235282]  RSP <ffff8802f524fdd0>
[  694.235379] ---[ end trace 949fad05591fcdb3 ]---
[  694.235428] Kernel panic - not syncing: Fatal exception
[  694.235478] Pid: 11841, comm: keepalive Tainted: G      D     2.6.39 #3
[  694.235525] Call Trace:
[  694.235573]  [<ffffffff813cd6f5>] ? panic+0x92/0x18a
[  694.235624]  [<ffffffff81038b61>] ? kmsg_dump+0x41/0xf0
[  694.235676]  [<ffffffff810050ad>] ? oops_end+0x8d/0xa0
[  694.235726]  [<ffffffff813d05ef>] ? general_protection+0x1f/0x30
[  694.235778]  [<ffffffff810db878>] ? dup_fd+0x168/0x300
[  694.235827]  [<ffffffff81037645>] ? copy_process+0xa75/0xfd0
[  694.235877]  [<ffffffff81037c0d>] ? do_fork+0x6d/0x2b0
[  694.235926]  [<ffffffff810457a9>] ? sigprocmask+0x69/0x100
[  694.235978]  [<ffffffff813d0ca3>] ? stub_clone+0x13/0x20
[  694.236028]  [<ffffffff813d0a3b>] ? system_call_fastpath+0x16/0x1b
[  694.236083] Rebooting in 60 seconds..

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
  2011-06-01  1:15               ` Andrea Arcangeli
@ 2011-06-01  4:52                 ` Hugh Dickins
  -1 siblings, 0 replies; 111+ messages in thread
From: Hugh Dickins @ 2011-06-01  4:52 UTC (permalink / raw)
  To: Andrea Arcangeli
  Cc: Brad Campbell, Borislav Petkov, linux-kernel, kvm, linux-mm

On Wed, 1 Jun 2011, Andrea Arcangeli wrote:
> On Wed, Jun 01, 2011 at 08:37:25AM +0800, Brad Campbell wrote:
> > On 01/06/11 06:31, Hugh Dickins wrote:
> > > Brad, my suspicion is that in each case the top 16 bits of RDX have been
> > > mysteriously corrupted from ffff to 0000, causing the general protection
> > > faults.  I don't understand what that has to do with KSM.
> > >
> > > But it's only a suspicion, because I can't make sense of the "Code:"
> > > lines in your traces, they have more than the expected 64 bytes, and
> > > only one of them has a ">" (with no"<") to mark faulting instruction.
> > >
> > > I did try compiling the 2.6.39 kernel from your config, but of course
> > > we have different compilers, so although I got close, it wasn't exact.
> > >
> > > Would you mind mailing me privately (it's about 73MB) the "objdump -trd"
> > > output for your original vmlinux (with KSM on)?  (Those -trd options are
> > > the ones I'm used to typing, I bet not they're not all relevant.)
> > >
> > > Of course, it's only a tiny fraction of that output that I need,
> > > might be better to cut it down to remove_rmap_item_from_tree and
> > > dup_fd and ksm_scan_thread, if you have the time to do so.
> > 
> > Would you believe about 20 seconds after I pressed send the kernel oopsed.
> > 
> > http://www.fnarfbargle.com/private/003_kernel_oops/
> > 
> > oops reproduced here, but an un-munged version is in that directory 
> > alongside the kernel.
> > 
> > [36542.880228] general protection fault: 0000 [#1] SMP
> 
> Reminds me of another oops that was reported on the kvm list for
> 2.6.38.1 with message id 4D8C6110.6090204. There the top 16 bits of
> rsi were flipped and it was a general protection too because of
> hitting on the not mappable virtual range.
> 
> http://www.virtall.com/files/temp/kvm.txt
> http://www.virtall.com/files/temp/config-2.6.38.1
> http://virtall.com/files/temp/mmu-objdump.txt
> 
> That oops happened in kvm_unmap_rmapp though, but it looked memory
> corruption (Avi suggested use after free) but it was a production
> system so we couldn't debug it further.
> 
> I recommend next thing to reproduce again with 2.6.39 or
> 3.0.0-rc1. Let's fix your scsi trouble if needed but it's better you
> test with 2.6.39.

Brad, thanks for this and the other further crash, with vmlinux etc:
very helpful info.

Andrea, I'm pretty sure you're right to connect Brad's report with
the one above.

In four out of five of Brad's reports (cannot tell in the fifth),
the bad pointer (with top 16 bits 0000 instead of ffff) had been
loaded from SLUB memory at an address offset 0x7f8 (1 case) or
0xff8 (3 cases) i.e. it's the short at 0x7fe or 0xffe that has
been zeroed.

No reason to suspect KSM's rmap_item code, or file table handling:
they just seem to be the victims of corruption from elsewhere.

I notice %rax and %rsi, the corrupted pointer in your kvm.txt
case, is itself a ...7f8 address; and %r13 an ...ff8 address.
I've not even glanced at the code, but I wonder if that implies
that KVM is close to the origin of the corruption.

I doubt I'll be able to spend more time on this, hope you can
take over.

I guess Brad could try SLUB debugging, boot with slub_debug=P
for poisoning perhaps; though it might upset alignments and
drive the problem underground.  Or see if the same happens
with SLAB instead of SLUB.

But I rather hope that you or someone will understand the 7fe clue.

Thanks,
Hugh

^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
@ 2011-06-01  4:52                 ` Hugh Dickins
  0 siblings, 0 replies; 111+ messages in thread
From: Hugh Dickins @ 2011-06-01  4:52 UTC (permalink / raw)
  To: Andrea Arcangeli
  Cc: Brad Campbell, Borislav Petkov, linux-kernel, kvm, linux-mm

On Wed, 1 Jun 2011, Andrea Arcangeli wrote:
> On Wed, Jun 01, 2011 at 08:37:25AM +0800, Brad Campbell wrote:
> > On 01/06/11 06:31, Hugh Dickins wrote:
> > > Brad, my suspicion is that in each case the top 16 bits of RDX have been
> > > mysteriously corrupted from ffff to 0000, causing the general protection
> > > faults.  I don't understand what that has to do with KSM.
> > >
> > > But it's only a suspicion, because I can't make sense of the "Code:"
> > > lines in your traces, they have more than the expected 64 bytes, and
> > > only one of them has a ">" (with no"<") to mark faulting instruction.
> > >
> > > I did try compiling the 2.6.39 kernel from your config, but of course
> > > we have different compilers, so although I got close, it wasn't exact.
> > >
> > > Would you mind mailing me privately (it's about 73MB) the "objdump -trd"
> > > output for your original vmlinux (with KSM on)?  (Those -trd options are
> > > the ones I'm used to typing, I bet not they're not all relevant.)
> > >
> > > Of course, it's only a tiny fraction of that output that I need,
> > > might be better to cut it down to remove_rmap_item_from_tree and
> > > dup_fd and ksm_scan_thread, if you have the time to do so.
> > 
> > Would you believe about 20 seconds after I pressed send the kernel oopsed.
> > 
> > http://www.fnarfbargle.com/private/003_kernel_oops/
> > 
> > oops reproduced here, but an un-munged version is in that directory 
> > alongside the kernel.
> > 
> > [36542.880228] general protection fault: 0000 [#1] SMP
> 
> Reminds me of another oops that was reported on the kvm list for
> 2.6.38.1 with message id 4D8C6110.6090204. There the top 16 bits of
> rsi were flipped and it was a general protection too because of
> hitting on the not mappable virtual range.
> 
> http://www.virtall.com/files/temp/kvm.txt
> http://www.virtall.com/files/temp/config-2.6.38.1
> http://virtall.com/files/temp/mmu-objdump.txt
> 
> That oops happened in kvm_unmap_rmapp though, but it looked memory
> corruption (Avi suggested use after free) but it was a production
> system so we couldn't debug it further.
> 
> I recommend next thing to reproduce again with 2.6.39 or
> 3.0.0-rc1. Let's fix your scsi trouble if needed but it's better you
> test with 2.6.39.

Brad, thanks for this and the other further crash, with vmlinux etc:
very helpful info.

Andrea, I'm pretty sure you're right to connect Brad's report with
the one above.

In four out of five of Brad's reports (cannot tell in the fifth),
the bad pointer (with top 16 bits 0000 instead of ffff) had been
loaded from SLUB memory at an address offset 0x7f8 (1 case) or
0xff8 (3 cases) i.e. it's the short at 0x7fe or 0xffe that has
been zeroed.

No reason to suspect KSM's rmap_item code, or file table handling:
they just seem to be the victims of corruption from elsewhere.

I notice %rax and %rsi, the corrupted pointer in your kvm.txt
case, is itself a ...7f8 address; and %r13 an ...ff8 address.
I've not even glanced at the code, but I wonder if that implies
that KVM is close to the origin of the corruption.

I doubt I'll be able to spend more time on this, hope you can
take over.

I guess Brad could try SLUB debugging, boot with slub_debug=P
for poisoning perhaps; though it might upset alignments and
drive the problem underground.  Or see if the same happens
with SLAB instead of SLUB.

But I rather hope that you or someone will understand the 7fe clue.

Thanks,
Hugh

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
  2011-06-01  4:52                 ` Hugh Dickins
@ 2011-06-01  6:31                   ` Brad Campbell
  -1 siblings, 0 replies; 111+ messages in thread
From: Brad Campbell @ 2011-06-01  6:31 UTC (permalink / raw)
  To: Hugh Dickins
  Cc: Andrea Arcangeli, Borislav Petkov, linux-kernel, kvm, linux-mm

On 01/06/11 12:52, Hugh Dickins wrote:

>
> I guess Brad could try SLUB debugging, boot with slub_debug=P
> for poisoning perhaps; though it might upset alignments and
> drive the problem underground.  Or see if the same happens
> with SLAB instead of SLUB.

Not much use I'm afraid.
This is all I get in the log

[ 3161.300073] 
=============================================================================
[ 3161.300147] BUG kmalloc-512: Freechain corrupt

The qemu process is then frozen, unkillable but reported in state "R"

13881 ?        R      3:27 /usr/bin/qemu -S -M pc-0.13 -enable-kvm -m 
1024 -smp 2,sockets=2,cores=1,threads=1 -nam

The machine then progressively dies until it's frozen solid with no 
further error messages.

I stupidly forgot to do an alt-sysrq-t prior to doing an alt-sysrq-b, 
but at least it responded to that.

On the bright side I can reproduce it at will.

^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
@ 2011-06-01  6:31                   ` Brad Campbell
  0 siblings, 0 replies; 111+ messages in thread
From: Brad Campbell @ 2011-06-01  6:31 UTC (permalink / raw)
  To: Hugh Dickins
  Cc: Andrea Arcangeli, Borislav Petkov, linux-kernel, kvm, linux-mm

On 01/06/11 12:52, Hugh Dickins wrote:

>
> I guess Brad could try SLUB debugging, boot with slub_debug=P
> for poisoning perhaps; though it might upset alignments and
> drive the problem underground.  Or see if the same happens
> with SLAB instead of SLUB.

Not much use I'm afraid.
This is all I get in the log

[ 3161.300073] 
=============================================================================
[ 3161.300147] BUG kmalloc-512: Freechain corrupt

The qemu process is then frozen, unkillable but reported in state "R"

13881 ?        R      3:27 /usr/bin/qemu -S -M pc-0.13 -enable-kvm -m 
1024 -smp 2,sockets=2,cores=1,threads=1 -nam

The machine then progressively dies until it's frozen solid with no 
further error messages.

I stupidly forgot to do an alt-sysrq-t prior to doing an alt-sysrq-b, 
but at least it responded to that.

On the bright side I can reproduce it at will.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
  2011-06-01  6:31                   ` Brad Campbell
@ 2011-06-01  6:56                     ` Avi Kivity
  -1 siblings, 0 replies; 111+ messages in thread
From: Avi Kivity @ 2011-06-01  6:56 UTC (permalink / raw)
  To: Brad Campbell
  Cc: Hugh Dickins, Andrea Arcangeli, Borislav Petkov, linux-kernel,
	kvm, linux-mm

On 06/01/2011 09:31 AM, Brad Campbell wrote:
> On 01/06/11 12:52, Hugh Dickins wrote:
>
>>
>> I guess Brad could try SLUB debugging, boot with slub_debug=P
>> for poisoning perhaps; though it might upset alignments and
>> drive the problem underground.  Or see if the same happens
>> with SLAB instead of SLUB.
>
> Not much use I'm afraid.
> This is all I get in the log
>
> [ 3161.300073] 
> =============================================================================
> [ 3161.300147] BUG kmalloc-512: Freechain corrupt
>
> The qemu process is then frozen, unkillable but reported in state "R"
>
> 13881 ?        R      3:27 /usr/bin/qemu -S -M pc-0.13 -enable-kvm -m 
> 1024 -smp 2,sockets=2,cores=1,threads=1 -nam
>
> The machine then progressively dies until it's frozen solid with no 
> further error messages.
>
> I stupidly forgot to do an alt-sysrq-t prior to doing an alt-sysrq-b, 
> but at least it responded to that.
>
> On the bright side I can reproduce it at will.

Please try slub_debug=FZPU; that should point the finger (hopefully at 
somebody else).

-- 
I have a truly marvellous patch that fixes the bug which this
signature is too narrow to contain.


^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
@ 2011-06-01  6:56                     ` Avi Kivity
  0 siblings, 0 replies; 111+ messages in thread
From: Avi Kivity @ 2011-06-01  6:56 UTC (permalink / raw)
  To: Brad Campbell
  Cc: Hugh Dickins, Andrea Arcangeli, Borislav Petkov, linux-kernel,
	kvm, linux-mm

On 06/01/2011 09:31 AM, Brad Campbell wrote:
> On 01/06/11 12:52, Hugh Dickins wrote:
>
>>
>> I guess Brad could try SLUB debugging, boot with slub_debug=P
>> for poisoning perhaps; though it might upset alignments and
>> drive the problem underground.  Or see if the same happens
>> with SLAB instead of SLUB.
>
> Not much use I'm afraid.
> This is all I get in the log
>
> [ 3161.300073] 
> =============================================================================
> [ 3161.300147] BUG kmalloc-512: Freechain corrupt
>
> The qemu process is then frozen, unkillable but reported in state "R"
>
> 13881 ?        R      3:27 /usr/bin/qemu -S -M pc-0.13 -enable-kvm -m 
> 1024 -smp 2,sockets=2,cores=1,threads=1 -nam
>
> The machine then progressively dies until it's frozen solid with no 
> further error messages.
>
> I stupidly forgot to do an alt-sysrq-t prior to doing an alt-sysrq-b, 
> but at least it responded to that.
>
> On the bright side I can reproduce it at will.

Please try slub_debug=FZPU; that should point the finger (hopefully at 
somebody else).

-- 
I have a truly marvellous patch that fixes the bug which this
signature is too narrow to contain.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
  2011-06-01  6:56                     ` Avi Kivity
  (?)
@ 2011-06-01  9:29                       ` Brad Campbell
  -1 siblings, 0 replies; 111+ messages in thread
From: Brad Campbell @ 2011-06-01  9:29 UTC (permalink / raw)
  To: Avi Kivity
  Cc: Hugh Dickins, Andrea Arcangeli, Borislav Petkov, linux-kernel,
	kvm, linux-mm

On 01/06/11 14:56, Avi Kivity wrote:
> On 06/01/2011 09:31 AM, Brad Campbell wrote:
>> On 01/06/11 12:52, Hugh Dickins wrote:
>>
>>>
>>> I guess Brad could try SLUB debugging, boot with slub_debug=P
>>> for poisoning perhaps; though it might upset alignments and
>>> drive the problem underground. Or see if the same happens
>>> with SLAB instead of SLUB.
>>
>> Not much use I'm afraid.
>> This is all I get in the log
>>
>> [ 3161.300073]
>> =============================================================================
>>
>> [ 3161.300147] BUG kmalloc-512: Freechain corrupt
>>
>> The qemu process is then frozen, unkillable but reported in state "R"
>>
>> 13881 ? R 3:27 /usr/bin/qemu -S -M pc-0.13 -enable-kvm -m 1024 -smp
>> 2,sockets=2,cores=1,threads=1 -nam
>>
>> The machine then progressively dies until it's frozen solid with no
>> further error messages.
>>
>> I stupidly forgot to do an alt-sysrq-t prior to doing an alt-sysrq-b,
>> but at least it responded to that.
>>
>> On the bright side I can reproduce it at will.
>
> Please try slub_debug=FZPU; that should point the finger (hopefully at
> somebody else).
>

Well the first attempt locked the machine solid. No network, no console..

I saw 
"=========================================================================="

on the console.. nothing after that. Would not respond to sysrq-t or any 
other sysrq combination other than -b, which rebooted the box.


No output on netconsole at all, I had to walk to the other building to 
look at the monitor and reboot it.

The second attempt jammed netconsole again, but I managed to get this 
from an ssh session I already had established. The machine died a slow 
and horrible death, but remained interactive enough for me to reboot it with

echo b > /proc/sysrq-trigger

Nothing else worked.



[  376.269051] 
=============================================================================
[  413.755328] 
=============================================================================
[  413.755337] BUG kmalloc-1024: Object padding overwritten
[  413.755341] 
-----------------------------------------------------------------------------
[  413.755344]
[  413.755349] INFO: 0xffff880417179566-0xffff880417179567. First byte 
0x0 instead of 0x5a
[  413.755363] INFO: Allocated in tcp_send_ack+0x26/0x120 age=5320 cpu=5 
pid=0
[  413.755374] INFO: Freed in __kfree_skb+0x11/0x90 age=5320 cpu=5 pid=0
[  413.755380] INFO: Slab 0xffffea000e50d240 objects=29 used=5 
fp=0xffff880417179120 flags=0x80000000000040c1
[  413.755386] INFO: Object 0xffff880417179120 @offset=4384 
fp=0xffff8804171799b0
[  413.755389]
[  413.755392] Bytes b4 0xffff880417179110:  d9 2b 00 00 01 00 00 00 5a 
5a 5a 5a 5a 5a 5a 5a Ù+......ZZZZZZZZ
[  413.755408]   Object 0xffff880417179120:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755423]   Object 0xffff880417179130:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755438]   Object 0xffff880417179140:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755452]   Object 0xffff880417179150:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755466]   Object 0xffff880417179160:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755480]   Object 0xffff880417179170:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755494]   Object 0xffff880417179180:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755508]   Object 0xffff880417179190:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755522]   Object 0xffff8804171791a0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755536]   Object 0xffff8804171791b0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755551]   Object 0xffff8804171791c0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755565]   Object 0xffff8804171791d0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755579]   Object 0xffff8804171791e0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755593]   Object 0xffff8804171791f0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755607]   Object 0xffff880417179200:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755621]   Object 0xffff880417179210:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755635]   Object 0xffff880417179220:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755650]   Object 0xffff880417179230:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755664]   Object 0xffff880417179240:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755678]   Object 0xffff880417179250:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755692]   Object 0xffff880417179260:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755706]   Object 0xffff880417179270:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755720]   Object 0xffff880417179280:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755734]   Object 0xffff880417179290:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755749]   Object 0xffff8804171792a0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755763]   Object 0xffff8804171792b0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755777]   Object 0xffff8804171792c0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755791]   Object 0xffff8804171792d0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755805]   Object 0xffff8804171792e0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755819]   Object 0xffff8804171792f0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755834]   Object 0xffff880417179300:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755848]   Object 0xffff880417179310:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755862]   Object 0xffff880417179320:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755876]   Object 0xffff880417179330:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755890]   Object 0xffff880417179340:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755904]   Object 0xffff880417179350:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755919]   Object 0xffff880417179360:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755933]   Object 0xffff880417179370:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755947]   Object 0xffff880417179380:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755961]   Object 0xffff880417179390:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755975]   Object 0xffff8804171793a0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755989]   Object 0xffff8804171793b0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.756004]   Object 0xffff8804171793c0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.756018]   Object 0xffff8804171793d0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.756032]   Object 0xffff8804171793e0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.756046]   Object 0xffff8804171793f0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.756060]   Object 0xffff880417179400:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.756074]   Object 0xffff880417179410:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.756089]   Object 0xffff880417179420:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.756103]   Object 0xffff880417179430:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.756117]   Object 0xffff880417179440:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.756131]   Object 0xffff880417179450:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.756145]   Object 0xffff880417179460:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.756160]   Object 0xffff880417179470:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.756174]   Object 0xffff880417179480:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.756188]   Object 0xffff880417179490:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.756202]   Object 0xffff8804171794a0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.756216]   Object 0xffff8804171794b0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.756230]   Object 0xffff8804171794c0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.756245]   Object 0xffff8804171794d0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.756259]   Object 0xffff8804171794e0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.756273]   Object 0xffff8804171794f0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.756287]   Object 0xffff880417179500:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.756301]   Object 0xffff880417179510:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b a5 kkkkkkkkkkkkkkk¥
[  413.756316]  Redzone 0xffff880417179520:  bb bb bb bb bb bb bb bb 
                      »»»»»»»»
[  413.756329]  Padding 0xffff880417179560:  5a 5a 5a 5a 5a 5a 00 00 
                      ZZZZZZ..
[  413.756345] Pid: 5247, comm: qemu Not tainted 2.6.39 #2
[  413.756349] Call Trace:
[  413.756353]  <IRQ>  [<ffffffff810b7ccd>] ? 
check_bytes_and_report+0x10d/0x150
[  413.756372]  [<ffffffff81318f1c>] ? pskb_expand_head+0x15c/0x250
[  413.756379]  [<ffffffff810b7db9>] ? check_object+0xa9/0x260
[  413.756387]  [<ffffffff81318f1c>] ? pskb_expand_head+0x15c/0x250
[  413.756393]  [<ffffffff810b86f4>] ? alloc_debug_processing+0x104/0x190
[  413.756400]  [<ffffffff810b9ac2>] ? T.912+0x272/0x2d0
[  413.756409]  [<ffffffff810ba59d>] ? __kmalloc+0x10d/0x160
[  413.756416]  [<ffffffff81318f1c>] ? pskb_expand_head+0x15c/0x250
[  413.756424]  [<ffffffff813a6c45>] ? nf_bridge_copy_header+0x145/0x160
[  413.756431]  [<ffffffff8139f78d>] ? br_dev_queue_push_xmit+0x6d/0x80
[  413.756439]  [<ffffffff813a55a0>] ? br_nf_post_routing+0x2a0/0x2f0
[  413.756447]  [<ffffffff81346bc4>] ? nf_iterate+0x84/0xb0
[  413.756453]  [<ffffffff8139f720>] ? br_flood_deliver+0x20/0x20
[  413.756459]  [<ffffffff81346c64>] ? nf_hook_slow+0x74/0x120
[  413.756465]  [<ffffffff8139f720>] ? br_flood_deliver+0x20/0x20
[  413.756472]  [<ffffffff8139f7da>] ? br_forward_finish+0x3a/0x60
[  413.756479]  [<ffffffff813a5758>] ? br_nf_forward_finish+0x168/0x170
[  413.756487]  [<ffffffff813a5c90>] ? br_nf_forward_ip+0x360/0x3a0
[  413.756492]  [<ffffffff81346bc4>] ? nf_iterate+0x84/0xb0
[  413.756498]  [<ffffffff8139f7a0>] ? br_dev_queue_push_xmit+0x80/0x80
[  413.756504]  [<ffffffff81346c64>] ? nf_hook_slow+0x74/0x120
[  413.756510]  [<ffffffff8139f7a0>] ? br_dev_queue_push_xmit+0x80/0x80
[  413.756516]  [<ffffffff8139f800>] ? br_forward_finish+0x60/0x60
[  413.756522]  [<ffffffff8139f800>] ? br_forward_finish+0x60/0x60
[  413.756528]  [<ffffffff8139f875>] ? __br_forward+0x75/0xc0
[  413.756534]  [<ffffffff8139f426>] ? deliver_clone+0x36/0x60
[  413.756540]  [<ffffffff8139f69d>] ? br_flood+0xbd/0x100
[  413.756546]  [<ffffffff813a05b0>] ? br_handle_local_finish+0x40/0x40
[  413.756552]  [<ffffffff813a080e>] ? br_handle_frame_finish+0x25e/0x280
[  413.756560]  [<ffffffff813a60f0>] ? br_nf_pre_routing_finish+0x1a0/0x330
[  413.756568]  [<ffffffff813a6958>] ? br_nf_pre_routing+0x6d8/0x800
[  413.756577]  [<ffffffff8102d46a>] ? enqueue_task+0x3a/0x90
[  413.756582]  [<ffffffff81346bc4>] ? nf_iterate+0x84/0xb0
[  413.756589]  [<ffffffff813a05b0>] ? br_handle_local_finish+0x40/0x40
[  413.756594]  [<ffffffff81346c64>] ? nf_hook_slow+0x74/0x120
[  413.756600]  [<ffffffff813a05b0>] ? br_handle_local_finish+0x40/0x40
[  413.756607]  [<ffffffff810339b0>] ? try_to_wake_up+0x2c0/0x2c0
[  413.756613]  [<ffffffff813a09d9>] ? br_handle_frame+0x1a9/0x280
[  413.756620]  [<ffffffff813a0830>] ? br_handle_frame_finish+0x280/0x280
[  413.756627]  [<ffffffff81320ef7>] ? __netif_receive_skb+0x157/0x5c0
[  413.756634]  [<ffffffff81321443>] ? process_backlog+0xe3/0x1d0
[  413.756641]  [<ffffffff81321da5>] ? net_rx_action+0xc5/0x1d0
[  413.756650]  [<ffffffff8103df11>] ? __do_softirq+0x91/0x120
[  413.756657]  [<ffffffff813d838c>] ? call_softirq+0x1c/0x30
[  413.756660]  <EOI>  [<ffffffff81003cbd>] ? do_softirq+0x4d/0x80
[  413.756673]  [<ffffffff81321ece>] ? netif_rx_ni+0x1e/0x30
[  413.756681]  [<ffffffff812b3ae2>] ? tun_chr_aio_write+0x332/0x4e0
[  413.756688]  [<ffffffff812b37b0>] ? tun_sendmsg+0x4d0/0x4d0
[  413.756697]  [<ffffffff810c24e9>] ? do_sync_readv_writev+0xa9/0xf0
[  413.756704]  [<ffffffff81063f9c>] ? do_futex+0x13c/0xa70
[  413.756711]  [<ffffffff811d6730>] ? timerqueue_add+0x60/0xb0
[  413.756719]  [<ffffffff81056ab7>] ? __hrtimer_start_range_ns+0x1e7/0x410
[  413.756726]  [<ffffffff810c231b>] ? rw_copy_check_uvector+0x7b/0x140
[  413.756734]  [<ffffffff810c2bcf>] ? do_readv_writev+0xdf/0x210
[  413.756742]  [<ffffffff810c2e7e>] ? sys_writev+0x4e/0xc0
[  413.756750]  [<ffffffff813d753b>] ? system_call_fastpath+0x16/0x1b
[  413.756756] FIX kmalloc-1024: Restoring 
0xffff880417179566-0xffff880417179567=0x5a
[  413.756760]
[  556.640033] 
=============================================================================
[  556.640041] BUG kmalloc-512: Object padding overwritten
[  556.640045] 
-----------------------------------------------------------------------------
[  556.640048]
[  556.640053] INFO: 0xffff880403bf091e-0xffff880403bf091f. First byte 
0x0 instead of 0x5a
[  556.640069] INFO: Allocated in sock_alloc_send_pskb+0x1d0/0x320 
age=21401 cpu=2 pid=5630
[  556.640080] INFO: Freed in __kfree_skb+0x11/0x90 age=21386 cpu=2 pid=3753
[  556.640086] INFO: Slab 0xffffea000e0d1c80 objects=28 used=10 
fp=0xffff880403bf06d8 flags=0x80000000000040c1
[  556.640092] INFO: Object 0xffff880403bf06d8 @offset=1752 
fp=0xffff880403bf1488
[  556.640096]
[  556.640099] Bytes b4 0xffff880403bf06c8:  90 c4 ff ff 00 00 00 00 5a 
5a 5a 5a 5a 5a 5a 5a .Äÿÿ....ZZZZZZZZ
[  556.640115]   Object 0xffff880403bf06d8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  556.640130]   Object 0xffff880403bf06e8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  556.640144]   Object 0xffff880403bf06f8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  556.640158]   Object 0xffff880403bf0708:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  556.640172]   Object 0xffff880403bf0718:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  556.640187]   Object 0xffff880403bf0728:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  556.640201]   Object 0xffff880403bf0738:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  556.640215]   Object 0xffff880403bf0748:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  556.640229]   Object 0xffff880403bf0758:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  556.640243]   Object 0xffff880403bf0768:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  556.640257]   Object 0xffff880403bf0778:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  556.640271]   Object 0xffff880403bf0788:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  556.640286]   Object 0xffff880403bf0798:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  556.640300]   Object 0xffff880403bf07a8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  556.640314]   Object 0xffff880403bf07b8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  556.640328]   Object 0xffff880403bf07c8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  556.640343]   Object 0xffff880403bf07d8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  556.640357]   Object 0xffff880403bf07e8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  556.640371]   Object 0xffff880403bf07f8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  556.640385]   Object 0xffff880403bf0808:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  556.640399]   Object 0xffff880403bf0818:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  556.640413]   Object 0xffff880403bf0828:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  556.640428]   Object 0xffff880403bf0838:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  556.640442]   Object 0xffff880403bf0848:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  556.640456]   Object 0xffff880403bf0858:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  556.640471]   Object 0xffff880403bf0868:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  556.640485]   Object 0xffff880403bf0878:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  556.640499]   Object 0xffff880403bf0888:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  556.640513]   Object 0xffff880403bf0898:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  556.640527]   Object 0xffff880403bf08a8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  556.640542]   Object 0xffff880403bf08b8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  556.640556]   Object 0xffff880403bf08c8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b a5 kkkkkkkkkkkkkkk¥
[  556.640570]  Redzone 0xffff880403bf08d8:  bb bb bb bb bb bb bb bb 
                      »»»»»»»»
[  556.640583]  Padding 0xffff880403bf0918:  5a 5a 5a 5a 5a 5a 00 00 
                      ZZZZZZ..
[  556.640599] Pid: 4809, comm: qemu Not tainted 2.6.39 #2
[  556.640603] Call Trace:
[  556.640607]  <IRQ>  [<ffffffff810b7ccd>] ? 
check_bytes_and_report+0x10d/0x150
[  556.640626]  [<ffffffff8131aae7>] ? __netdev_alloc_skb+0x17/0x40
[  556.640632]  [<ffffffff810b7db9>] ? check_object+0xa9/0x260
[  556.640640]  [<ffffffff8131aae7>] ? __netdev_alloc_skb+0x17/0x40
[  556.640647]  [<ffffffff810b86f4>] ? alloc_debug_processing+0x104/0x190
[  556.640654]  [<ffffffff810b9ac2>] ? T.912+0x272/0x2d0
[  556.640661]  [<ffffffff810bb70d>] ? __kmalloc_track_caller+0x10d/0x160
[  556.640668]  [<ffffffff813198c2>] ? __alloc_skb+0x72/0x160
[  556.640676]  [<ffffffff8131aae7>] ? __netdev_alloc_skb+0x17/0x40
[  556.640684]  [<ffffffff813a7b93>] ? 
br_ip4_multicast_alloc_query+0x23/0x1c0
[  556.640690]  [<ffffffff813a7e76>] ? br_multicast_send_query+0x76/0x130
[  556.640698]  [<ffffffff8104dba8>] ? wq_worker_waking_up+0x8/0x30
[  556.640706]  [<ffffffff8102ca81>] ? sched_slice+0x51/0x80
[  556.640711]  [<ffffffff813a7fb5>] ? 
br_multicast_port_query_expired+0x45/0x70
[  556.640719]  [<ffffffff81043f77>] ? run_timer_softirq+0x137/0x270
[  556.640725]  [<ffffffff81033fe9>] ? scheduler_tick+0x289/0x2d0
[  556.640731]  [<ffffffff813a7f70>] ? br_multicast_query_expired+0x40/0x40
[  556.640740]  [<ffffffff8103df11>] ? __do_softirq+0x91/0x120
[  556.640747]  [<ffffffff813d838c>] ? call_softirq+0x1c/0x30
[  556.640755]  [<ffffffff81003cbd>] ? do_softirq+0x4d/0x80
[  556.640762]  [<ffffffff8103ddbe>] ? irq_exit+0x8e/0xb0
[  556.640770]  [<ffffffff81019d1a>] ? smp_apic_timer_interrupt+0x6a/0xa0
[  556.640776]  [<ffffffff813d7e53>] ? apic_timer_interrupt+0x13/0x20
[  556.640780]  <EOI>  [<ffffffff8102965a>] ? 
flush_tlb_others_ipi+0x11a/0x130
[  556.640791]  [<ffffffff81029647>] ? flush_tlb_others_ipi+0x107/0x130
[  556.640799]  [<ffffffff810aadab>] ? ptep_clear_flush+0xb/0x10
[  556.640806]  [<ffffffff8109a40b>] ? do_wp_page+0x34b/0x7a0
[  556.640836]  [<ffffffffa00a188a>] ? kvm_read_guest_page+0x5a/0x70 [kvm]
[  556.640863]  [<ffffffffa00b7db4>] ? 
paging64_walk_addr_generic+0x264/0x4c0 [kvm]
[  556.640871]  [<ffffffff8109acad>] ? handle_pte_fault+0x44d/0x990
[  556.640878]  [<ffffffff8109b4e8>] ? follow_page+0x268/0x440
[  556.640886]  [<ffffffff8109bd4d>] ? __get_user_pages+0x12d/0x530
[  556.640892]  [<ffffffff81028dbb>] ? gup_pud_range+0x12b/0x1b0
[  556.640914]  [<ffffffffa00a2d2e>] ? get_user_page_nowait+0x2e/0x40 [kvm]
[  556.640935]  [<ffffffffa00a2e7b>] ? hva_to_pfn+0x13b/0x440 [kvm]
[  556.640956]  [<ffffffffa00a31e9>] ? __gfn_to_pfn+0x39/0xf0 [kvm]
[  556.640979]  [<ffffffffa00b825d>] ? try_async_pf+0x4d/0x190 [kvm]
[  556.641002]  [<ffffffffa00b92be>] ? tdp_page_fault+0x10e/0x200 [kvm]
[  556.641026]  [<ffffffffa00b968c>] ? kvm_mmu_page_fault+0x1c/0x80 [kvm]
[  556.641050]  [<ffffffffa00b0a7a>] ? 
kvm_arch_vcpu_ioctl_run+0x3fa/0xcf0 [kvm]
[  556.641058]  [<ffffffff8106239f>] ? futex_wake+0x10f/0x120
[  556.641065]  [<ffffffff81063f7b>] ? do_futex+0x11b/0xa70
[  556.641086]  [<ffffffffa00a517f>] ? kvm_vcpu_ioctl+0x4df/0x5e0 [kvm]
[  556.641097]  [<ffffffff81046671>] ? __dequeue_signal+0xe1/0x170
[  556.641104]  [<ffffffff8104821c>] ? do_send_sig_info+0x6c/0x90
[  556.641112]  [<ffffffff81046aac>] ? dequeue_signal+0x3c/0x170
[  556.641119]  [<ffffffff8104551f>] ? copy_siginfo_to_user+0xef/0x1d0
[  556.641125]  [<ffffffff810d26db>] ? do_vfs_ioctl+0x9b/0x4f0
[  556.641132]  [<ffffffff8106494a>] ? sys_futex+0x7a/0x180
[  556.641137]  [<ffffffff810d2b79>] ? sys_ioctl+0x49/0x80
[  556.641145]  [<ffffffff813d753b>] ? system_call_fastpath+0x16/0x1b
[  556.641152] FIX kmalloc-512: Restoring 
0xffff880403bf091e-0xffff880403bf091f=0x5a
[  556.641155]
[  602.610062] INFO: task ksmd:552 blocked for more than 120 seconds.
[  602.610068] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" 
disables this message.
[  602.610073] ksmd            D 0000000000000000     0   552      2 
0x00000000
[  602.610083]  ffff88041db68ca0 0000000000000046 000212d01fc11d00 
ffff880400000000
[  602.610090]  ffffffff81593020 ffff88041c1a9fd8 0000000000004000 
ffff88041c1a8010
[  602.610097]  ffff88041c1a9fd8 ffff88041db68ca0 0000000000000000 
0000000200000002
[  602.610104] Call Trace:
[  602.610121]  [<ffffffff813d5595>] ? schedule_timeout+0x1c5/0x230
[  602.610132]  [<ffffffff8102f69c>] ? enqueue_task_fair+0x14c/0x190
[  602.610141]  [<ffffffff8102db27>] ? task_rq_lock+0x47/0x90
[  602.610148]  [<ffffffff813d4562>] ? wait_for_common+0xd2/0x180
[  602.610154]  [<ffffffff810339b0>] ? try_to_wake_up+0x2c0/0x2c0
[  602.610162]  [<ffffffff8104cb94>] ? flush_work+0x24/0x30
[  602.610167]  [<ffffffff8104be10>] ? do_work_for_cpu+0x20/0x20
[  602.610174]  [<ffffffff8104e0ab>] ? schedule_on_each_cpu+0xab/0xe0
[  602.610181]  [<ffffffff810b5c05>] ? ksm_scan_thread+0x7f5/0xc20
[  602.610189]  [<ffffffff81052a20>] ? wake_up_bit+0x40/0x40
[  602.610194]  [<ffffffff810b5410>] ? 
try_to_merge_with_ksm_page+0x570/0x570
[  602.610200]  [<ffffffff810b5410>] ? 
try_to_merge_with_ksm_page+0x570/0x570
[  602.610207]  [<ffffffff810525b6>] ? kthread+0x96/0xa0
[  602.610214]  [<ffffffff813d8294>] ? kernel_thread_helper+0x4/0x10
[  602.610221]  [<ffffffff81052520>] ? kthread_worker_fn+0x120/0x120
[  602.610227]  [<ffffffff813d8290>] ? gs_change+0xb/0xb
[  602.610233] INFO: task fsnotify_mark:662 blocked for more than 120 
seconds.
[  602.610237] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" 
disables this message.
[  602.610241] fsnotify_mark   D 0000000000000000     0   662      2 
0x00000000
[  602.610248]  ffff88041c31d860 0000000000000046 0000000000000000 
0000000000000000
[  602.610255]  ffffffff81593020 ffff88041c365fd8 0000000000004000 
ffff88041c364010
[  602.610262]  ffff88041c365fd8 ffff88041c31d860 0000000000000000 
0000000000000000
[  602.610268] Call Trace:
[  602.610275]  [<ffffffff81035521>] ? load_balance+0x91/0x5e0
[  602.610282]  [<ffffffff813d5595>] ? schedule_timeout+0x1c5/0x230
[  602.610289]  [<ffffffff8102f323>] ? pick_next_task_fair+0x103/0x190
[  602.610295]  [<ffffffff813d4a6d>] ? schedule+0x28d/0x910
[  602.610302]  [<ffffffff813d4562>] ? wait_for_common+0xd2/0x180
[  602.610307]  [<ffffffff810339b0>] ? try_to_wake_up+0x2c0/0x2c0
[  602.610314]  [<ffffffff81074420>] ? synchronize_rcu_bh+0x50/0x50
[  602.610320]  [<ffffffff8107446a>] ? synchronize_sched+0x4a/0x50
[  602.610326]  [<ffffffff8104f940>] ? find_ge_pid+0x40/0x40
[  602.610333]  [<ffffffff8105742b>] ? __synchronize_srcu+0x5b/0xc0
[  602.610342]  [<ffffffff810f5f63>] ? fsnotify_mark_destroy+0x83/0x150
[  602.610348]  [<ffffffff81052a20>] ? wake_up_bit+0x40/0x40
[  602.610356]  [<ffffffff810f5ee0>] ? 
fsnotify_set_mark_ignored_mask_locked+0x20/0x20
[  602.610364]  [<ffffffff810f5ee0>] ? 
fsnotify_set_mark_ignored_mask_locked+0x20/0x20
[  602.610371]  [<ffffffff810525b6>] ? kthread+0x96/0xa0
[  602.610377]  [<ffffffff813d8294>] ? kernel_thread_helper+0x4/0x10
[  602.610385]  [<ffffffff81052520>] ? kthread_worker_fn+0x120/0x120
[  602.610390]  [<ffffffff813d8290>] ? gs_change+0xb/0xb
[  644.689408] 
=============================================================================
[  644.689416] BUG kmalloc-512: Object padding overwritten
[  644.689420] 
-----------------------------------------------------------------------------
[  644.689423]
[  644.689428] INFO: 0xffff88041411f476-0xffff88041411f477. First byte 
0x0 instead of 0x5a
[  644.689444] INFO: Allocated in load_elf_binary+0xa6c/0x1c00 age=18648 
cpu=3 pid=5915
[  644.689454] INFO: Freed in load_elf_binary+0xab8/0x1c00 age=18648 
cpu=3 pid=5915
[  644.689461] INFO: Slab 0xffffea000e463e20 objects=28 used=10 
fp=0xffff88041411f230 flags=0x80000000000040c1
[  644.689467] INFO: Object 0xffff88041411f230 @offset=12848 
fp=0xffff88041411eda0
[  644.689470]
[  644.689474] Bytes b4 0xffff88041411f220:  22 6f 00 00 01 00 00 00 5a 
5a 5a 5a 5a 5a 5a 5a "o......ZZZZZZZZ
[  644.689490]   Object 0xffff88041411f230:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  644.689505]   Object 0xffff88041411f240:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  644.689519]   Object 0xffff88041411f250:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  644.689533]   Object 0xffff88041411f260:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  644.689547]   Object 0xffff88041411f270:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  644.689562]   Object 0xffff88041411f280:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  644.689576]   Object 0xffff88041411f290:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  644.689590]   Object 0xffff88041411f2a0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  644.689604]   Object 0xffff88041411f2b0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  644.689618]   Object 0xffff88041411f2c0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  644.689632]   Object 0xffff88041411f2d0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  644.689647]   Object 0xffff88041411f2e0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  644.689661]   Object 0xffff88041411f2f0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  644.689675]   Object 0xffff88041411f300:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  644.689689]   Object 0xffff88041411f310:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  644.689703]   Object 0xffff88041411f320:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  644.689717]   Object 0xffff88041411f330:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  644.689732]   Object 0xffff88041411f340:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  644.689746]   Object 0xffff88041411f350:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  644.689760]   Object 0xffff88041411f360:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  644.689774]   Object 0xffff88041411f370:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  644.689788]   Object 0xffff88041411f380:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  644.689802]   Object 0xffff88041411f390:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  644.689817]   Object 0xffff88041411f3a0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  644.689831]   Object 0xffff88041411f3b0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  644.689845]   Object 0xffff88041411f3c0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  644.689859]   Object 0xffff88041411f3d0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  644.689873]   Object 0xffff88041411f3e0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  644.689888]   Object 0xffff88041411f3f0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  644.689902]   Object 0xffff88041411f400:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  644.689916]   Object 0xffff88041411f410:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  644.689930]   Object 0xffff88041411f420:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b a5 kkkkkkkkkkkkkkk¥
[  644.689945]  Redzone 0xffff88041411f430:  bb bb bb bb bb bb bb bb 
                      »»»»»»»»
[  644.689957]  Padding 0xffff88041411f470:  5a 5a 5a 5a 5a 5a 00 00 
                      ZZZZZZ..
[  644.689974] Pid: 6597, comm: awk Not tainted 2.6.39 #2
[  644.689978] Call Trace:
[  644.689989]  [<ffffffff810b7ccd>] ? check_bytes_and_report+0x10d/0x150
[  644.689998]  [<ffffffff8110773c>] ? load_elf_binary+0xa6c/0x1c00
[  644.690005]  [<ffffffff810b7db9>] ? check_object+0xa9/0x260
[  644.690012]  [<ffffffff8110773c>] ? load_elf_binary+0xa6c/0x1c00
[  644.690019]  [<ffffffff810b86f4>] ? alloc_debug_processing+0x104/0x190
[  644.690026]  [<ffffffff810b9ac2>] ? T.912+0x272/0x2d0
[  644.690032]  [<ffffffff810ba59d>] ? __kmalloc+0x10d/0x160
[  644.690039]  [<ffffffff8110773c>] ? load_elf_binary+0xa6c/0x1c00
[  644.690047]  [<ffffffff8109bd9d>] ? __get_user_pages+0x17d/0x530
[  644.690055]  [<ffffffff810c9556>] ? get_arg_page+0x56/0x100
[  644.690062]  [<ffffffff810c8070>] ? search_binary_handler+0x90/0x240
[  644.690069]  [<ffffffff810c9e9f>] ? do_execve+0x22f/0x2f0
[  644.690076]  [<ffffffff810094a6>] ? sys_execve+0x36/0x60
[  644.690085]  [<ffffffff813d78dc>] ? stub_execve+0x6c/0xc0
[  644.690092] FIX kmalloc-512: Restoring 
0xffff88041411f476-0xffff88041411f477=0x5a
[  644.690095]
[  704.979481] 
=============================================================================
[  704.979488] BUG kmalloc-512: Object padding overwritten
[  704.979492] 
-----------------------------------------------------------------------------
[  704.979496]
[  704.979501] INFO: 0xffff8804168c1fee-0xffff8804168c1fef. First byte 
0x0 instead of 0x5a
[  704.979517] INFO: Allocated in load_elf_binary+0xa6c/0x1c00 age=24184 
cpu=0 pid=5999
[  704.979527] INFO: Freed in load_elf_binary+0xab8/0x1c00 age=24184 
cpu=0 pid=5999
[  704.979534] INFO: Slab 0xffffea000e4eea00 objects=28 used=10 
fp=0xffff8804168c1da8 flags=0x80000000000040c1
[  704.979540] INFO: Object 0xffff8804168c1da8 @offset=7592 
fp=0xffff8804168c2910
[  704.979543]
[  704.979547] Bytes b4 0xffff8804168c1d98:  93 98 ff ff 00 00 00 00 5a 
5a 5a 5a 5a 5a 5a 5a ..ÿÿ....ZZZZZZZZ
[  704.979563]   Object 0xffff8804168c1da8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  704.979578]   Object 0xffff8804168c1db8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  704.979592]   Object 0xffff8804168c1dc8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  704.979606]   Object 0xffff8804168c1dd8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  704.979620]   Object 0xffff8804168c1de8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  704.979634]   Object 0xffff8804168c1df8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  704.979648]   Object 0xffff8804168c1e08:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  704.979662]   Object 0xffff8804168c1e18:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  704.979676]   Object 0xffff8804168c1e28:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  704.979690]   Object 0xffff8804168c1e38:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  704.979704]   Object 0xffff8804168c1e48:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  704.979719]   Object 0xffff8804168c1e58:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  704.979733]   Object 0xffff8804168c1e68:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  704.979747]   Object 0xffff8804168c1e78:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  704.979761]   Object 0xffff8804168c1e88:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  704.979775]   Object 0xffff8804168c1e98:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  704.979789]   Object 0xffff8804168c1ea8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  704.979803]   Object 0xffff8804168c1eb8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  704.979817]   Object 0xffff8804168c1ec8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  704.979831]   Object 0xffff8804168c1ed8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  704.979845]   Object 0xffff8804168c1ee8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  704.979859]   Object 0xffff8804168c1ef8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  704.979873]   Object 0xffff8804168c1f08:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  704.979888]   Object 0xffff8804168c1f18:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  704.979902]   Object 0xffff8804168c1f28:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  704.979916]   Object 0xffff8804168c1f38:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  704.979930]   Object 0xffff8804168c1f48:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  704.979944]   Object 0xffff8804168c1f58:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  704.979958]   Object 0xffff8804168c1f68:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  704.979972]   Object 0xffff8804168c1f78:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  704.979986]   Object 0xffff8804168c1f88:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  704.980000]   Object 0xffff8804168c1f98:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b a5 kkkkkkkkkkkkkkk¥
[  704.980015]  Redzone 0xffff8804168c1fa8:  bb bb bb bb bb bb bb bb 
                      »»»»»»»»
[  704.980028]  Padding 0xffff8804168c1fe8:  5a 5a 5a 5a 5a 5a 00 00 
                      ZZZZZZ..
[  704.980044] Pid: 6812, comm: get-rrdtool-dat Not tainted 2.6.39 #2
[  704.980048] Call Trace:
[  704.980060]  [<ffffffff810b7ccd>] ? check_bytes_and_report+0x10d/0x150
[  704.980069]  [<ffffffff81106e49>] ? load_elf_binary+0x179/0x1c00
[  704.980075]  [<ffffffff810b7db9>] ? check_object+0xa9/0x260
[  704.980083]  [<ffffffff81106e49>] ? load_elf_binary+0x179/0x1c00
[  704.980090]  [<ffffffff810b86f4>] ? alloc_debug_processing+0x104/0x190
[  704.980096]  [<ffffffff810b9ac2>] ? T.912+0x272/0x2d0
[  704.980102]  [<ffffffff810ba59d>] ? __kmalloc+0x10d/0x160
[  704.980109]  [<ffffffff81106e49>] ? load_elf_binary+0x179/0x1c00
[  704.980117]  [<ffffffff8109b782>] ? __pte_alloc+0x42/0x130
[  704.980123]  [<ffffffff8109b5a5>] ? follow_page+0x325/0x440
[  704.980130]  [<ffffffff8109b5a5>] ? follow_page+0x325/0x440
[  704.980137]  [<ffffffff8109bd9d>] ? __get_user_pages+0x17d/0x530
[  704.980145]  [<ffffffff810c9556>] ? get_arg_page+0x56/0x100
[  704.980152]  [<ffffffff810c8070>] ? search_binary_handler+0x90/0x240
[  704.980158]  [<ffffffff810c9e9f>] ? do_execve+0x22f/0x2f0
[  704.980166]  [<ffffffff810094a6>] ? sys_execve+0x36/0x60
[  704.980175]  [<ffffffff813d78dc>] ? stub_execve+0x6c/0xc0
[  704.980182] FIX kmalloc-512: Restoring 
0xffff8804168c1fee-0xffff8804168c1fef=0x5a
[  704.980185]
[  722.610058] INFO: task ksmd:552 blocked for more than 120 seconds.
[  722.610063] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" 
disables this message.
[  722.610069] ksmd            D 0000000000000000     0   552      2 
0x00000000
[  722.610078]  ffff88041db68ca0 0000000000000046 000212d01fc11d00 
ffff880400000000
[  722.610085]  ffffffff81593020 ffff88041c1a9fd8 0000000000004000 
ffff88041c1a8010
[  722.610092]  ffff88041c1a9fd8 ffff88041db68ca0 0000000000000000 
0000000200000002
[  722.610099] Call Trace:
[  722.610115]  [<ffffffff813d5595>] ? schedule_timeout+0x1c5/0x230
[  722.610127]  [<ffffffff8102f69c>] ? enqueue_task_fair+0x14c/0x190
[  722.610135]  [<ffffffff8102db27>] ? task_rq_lock+0x47/0x90
[  722.610142]  [<ffffffff813d4562>] ? wait_for_common+0xd2/0x180
[  722.610148]  [<ffffffff810339b0>] ? try_to_wake_up+0x2c0/0x2c0
[  722.610156]  [<ffffffff8104cb94>] ? flush_work+0x24/0x30
[  722.610162]  [<ffffffff8104be10>] ? do_work_for_cpu+0x20/0x20
[  722.610168]  [<ffffffff8104e0ab>] ? schedule_on_each_cpu+0xab/0xe0
[  722.610175]  [<ffffffff810b5c05>] ? ksm_scan_thread+0x7f5/0xc20
[  722.610182]  [<ffffffff81052a20>] ? wake_up_bit+0x40/0x40
[  722.610188]  [<ffffffff810b5410>] ? 
try_to_merge_with_ksm_page+0x570/0x570
[  722.610194]  [<ffffffff810b5410>] ? 
try_to_merge_with_ksm_page+0x570/0x570
[  722.610201]  [<ffffffff810525b6>] ? kthread+0x96/0xa0
[  722.610208]  [<ffffffff813d8294>] ? kernel_thread_helper+0x4/0x10
[  722.610215]  [<ffffffff81052520>] ? kthread_worker_fn+0x120/0x120
[  722.610221]  [<ffffffff813d8290>] ? gs_change+0xb/0xb
[  722.610226] INFO: task fsnotify_mark:662 blocked for more than 120 
seconds.
[  722.610230] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" 
disables this message.
[  722.610235] fsnotify_mark   D 0000000000000000     0   662      2 
0x00000000
[  722.610241]  ffff88041c31d860 0000000000000046 0000000000000000 
0000000000000000
[  722.610248]  ffffffff81593020 ffff88041c365fd8 0000000000004000 
ffff88041c364010
[  722.610255]  ffff88041c365fd8 ffff88041c31d860 0000000000000000 
0000000000000000
[  722.610261] Call Trace:
[  722.610268]  [<ffffffff81035521>] ? load_balance+0x91/0x5e0
[  722.610275]  [<ffffffff813d5595>] ? schedule_timeout+0x1c5/0x230
[  722.610282]  [<ffffffff8102f323>] ? pick_next_task_fair+0x103/0x190
[  722.610289]  [<ffffffff813d4a6d>] ? schedule+0x28d/0x910
[  722.610295]  [<ffffffff813d4562>] ? wait_for_common+0xd2/0x180
[  722.610301]  [<ffffffff810339b0>] ? try_to_wake_up+0x2c0/0x2c0
[  722.610308]  [<ffffffff81074420>] ? synchronize_rcu_bh+0x50/0x50
[  722.610313]  [<ffffffff8107446a>] ? synchronize_sched+0x4a/0x50
[  722.610319]  [<ffffffff8104f940>] ? find_ge_pid+0x40/0x40
[  722.610326]  [<ffffffff8105742b>] ? __synchronize_srcu+0x5b/0xc0
[  722.610335]  [<ffffffff810f5f63>] ? fsnotify_mark_destroy+0x83/0x150
[  722.610342]  [<ffffffff81052a20>] ? wake_up_bit+0x40/0x40
[  722.610350]  [<ffffffff810f5ee0>] ? 
fsnotify_set_mark_ignored_mask_locked+0x20/0x20
[  722.610358]  [<ffffffff810f5ee0>] ? 
fsnotify_set_mark_ignored_mask_locked+0x20/0x20
[  722.610365]  [<ffffffff810525b6>] ? kthread+0x96/0xa0
[  722.610371]  [<ffffffff813d8294>] ? kernel_thread_helper+0x4/0x10
[  722.610378]  [<ffffffff81052520>] ? kthread_worker_fn+0x120/0x120
[  722.610384]  [<ffffffff813d8290>] ? gs_change+0xb/0xb
[  722.610392] INFO: task jbd2/md0-8:2528 blocked for more than 120 seconds.
[  722.610396] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" 
disables this message.
[  722.610400] jbd2/md0-8      D 0000000000000005     0  2528      2 
0x00000000
[  722.610407]  ffff88041d91c570 0000000000000046 ffff88041baeea28 
ffffea0000000000
[  722.610413]  ffff88041d91a5e0 ffff880419f9bfd8 0000000000004000 
ffff880419f9a010
[  722.610420]  ffff880419f9bfd8 ffff88041d91c570 ffff88041baeea28 
ffffffff810b8495
[  722.610426] Call Trace:
[  722.610432]  [<ffffffff810b8495>] ? init_object+0x85/0xa0
[  722.610438]  [<ffffffff810b8916>] ? free_debug_processing+0x196/0x250
[  722.610447]  [<ffffffff8105adae>] ? ktime_get_ts+0x6e/0xf0
[  722.610455]  [<ffffffff810810f0>] ? __lock_page+0x70/0x70
[  722.610461]  [<ffffffff813d5174>] ? io_schedule+0x84/0xd0
[  722.610469]  [<ffffffff811d4403>] ? 
radix_tree_gang_lookup_tag_slot+0x93/0xf0
[  722.610476]  [<ffffffff810810f9>] ? sleep_on_page+0x9/0x10
[  722.610482]  [<ffffffff813d57df>] ? __wait_on_bit+0x4f/0x80
[  722.610489]  [<ffffffff810812eb>] ? wait_on_page_bit+0x6b/0x80
[  722.610496]  [<ffffffff81052a50>] ? autoremove_wake_function+0x30/0x30
[  722.610504]  [<ffffffff8108a458>] ? pagevec_lookup_tag+0x18/0x20
[  722.610509]  [<ffffffff81081f2a>] ? filemap_fdatawait_range+0xfa/0x180
[  722.610518]  [<ffffffff811be09f>] ? submit_bio+0x6f/0xf0
[  722.610526]  [<ffffffff81176276>] ? 
jbd2_journal_commit_transaction+0x796/0x1270
[  722.610536]  [<ffffffff81179ed1>] ? kjournald2+0xb1/0x1e0
[  722.610542]  [<ffffffff81052a20>] ? wake_up_bit+0x40/0x40
[  722.610549]  [<ffffffff81179e20>] ? commit_timeout+0x10/0x10
[  722.610556]  [<ffffffff81179e20>] ? commit_timeout+0x10/0x10
[  722.610563]  [<ffffffff810525b6>] ? kthread+0x96/0xa0
[  722.610569]  [<ffffffff813d8294>] ? kernel_thread_helper+0x4/0x10
[  722.610576]  [<ffffffff81052520>] ? kthread_worker_fn+0x120/0x120
[  722.610582]  [<ffffffff813d8290>] ? gs_change+0xb/0xb
[  722.610591] INFO: task nfsd:4326 blocked for more than 120 seconds.
[  722.610594] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" 
disables this message.
[  722.610598] nfsd            D 0000000000000000     0  4326      2 
0x00000000
[  722.610605]  ffff88041b604bc0 0000000000000046 0000000000000016 
ffffffff00000000
[  722.610611]  ffff88041d8fa5e0 ffff88041c527fd8 0000000000004000 
ffff88041c526010
[  722.610617]  ffff88041c527fd8 ffff88041b604bc0 000000000000009c 
ffff880000000000
[  722.610623] Call Trace:
[  722.610630]  [<ffffffff8101e845>] ? amd_flush_garts+0x105/0x140
[  722.610637]  [<ffffffff8101fa80>] ? gart_map_sg+0x480/0x480
[  722.610643]  [<ffffffff8101f5d3>] ? flush_gart+0x23/0x50
[  722.610650]  [<ffffffff81080f98>] ? find_get_page+0x18/0x90
[  722.610657]  [<ffffffff81174b95>] ? do_get_write_access+0x265/0x4a0
[  722.610665]  [<ffffffff81052a50>] ? autoremove_wake_function+0x30/0x30
[  722.610672]  [<ffffffff81174ef9>] ? 
jbd2_journal_get_write_access+0x29/0x50
[  722.610680]  [<ffffffff8115f122>] ? 
__ext4_journal_get_write_access+0x32/0x80
[  722.610689]  [<ffffffff81143908>] ? ext4_reserve_inode_write+0x78/0xa0
[  722.610696]  [<ffffffff81143970>] ? ext4_mark_inode_dirty+0x40/0x1e0
[  722.610703]  [<ffffffff81156c0b>] ? ext4_journal_start_sb+0x6b/0x160
[  722.610711]  [<ffffffff81322645>] ? dev_hard_start_xmit+0x305/0x5f0
[  722.610719]  [<ffffffff81352fd0>] ? ip_finish_output2+0x290/0x290
[  722.610726]  [<ffffffff81143c65>] ? ext4_dirty_inode+0x35/0x70
[  722.610733]  [<ffffffff810e4a08>] ? __mark_inode_dirty+0x38/0x210
[  722.610741]  [<ffffffff810d9317>] ? file_update_time+0xf7/0x180
[  722.610747]  [<ffffffff81082458>] ? __generic_file_aio_write+0x1f8/0x430
[  722.610755]  [<ffffffff81373ef9>] ? udp_sendmsg+0x3c9/0x7e0
[  722.610763]  [<ffffffff81314260>] ? sock_alloc_send_pskb+0x1d0/0x320
[  722.610769]  [<ffffffff81082703>] ? generic_file_aio_write+0x73/0xf0
[  722.610776]  [<ffffffff8113f74e>] ? ext4_file_write+0x6e/0x2b0
[  722.610783]  [<ffffffff810da8ac>] ? iget_locked+0x4c/0x140
[  722.610789]  [<ffffffff8119a590>] ? fh_compose+0x4c0/0x4c0
[  722.610795]  [<ffffffff8113f6e0>] ? ext4_llseek+0x110/0x110
[  722.610803]  [<ffffffff810c24e9>] ? do_sync_readv_writev+0xa9/0xf0
[  722.610811]  [<ffffffff810c231b>] ? rw_copy_check_uvector+0x7b/0x140
[  722.610819]  [<ffffffff810c2bcf>] ? do_readv_writev+0xdf/0x210
[  722.610825]  [<ffffffff810b8735>] ? alloc_debug_processing+0x145/0x190
[  722.610831]  [<ffffffff810b9bc5>] ? kmem_cache_alloc+0xa5/0xb0
[  722.610838]  [<ffffffff8113f463>] ? ext4_file_open+0x63/0x180
[  722.610844]  [<ffffffff8119c21d>] ? nfsd_vfs_write+0xed/0x3a0
[  722.610851]  [<ffffffff810c1127>] ? __dentry_open+0x1f7/0x2b0
[  722.610857]  [<ffffffff8119c892>] ? nfsd_open+0xf2/0x1b0
[  722.610862]  [<ffffffff8119cd34>] ? nfsd_write+0xf4/0x110
[  722.610868]  [<ffffffff81199930>] ? nfsd_proc_write+0xb0/0x120
[  722.610876]  [<ffffffff811971c5>] ? nfsd_dispatch+0xf5/0x230
[  722.610882]  [<ffffffff813b456f>] ? svc_process+0x4af/0x820
[  722.610887]  [<ffffffff810339b0>] ? try_to_wake_up+0x2c0/0x2c0
[  722.610894]  [<ffffffff811977a0>] ? nfsd_svc+0x1b0/0x1b0
[  722.610901]  [<ffffffff8119784d>] ? nfsd+0xad/0x150
[  722.610907]  [<ffffffff810525b6>] ? kthread+0x96/0xa0
[  722.610913]  [<ffffffff813d8294>] ? kernel_thread_helper+0x4/0x10
[  722.610921]  [<ffffffff81052520>] ? kthread_worker_fn+0x120/0x120
[  722.610926]  [<ffffffff813d8290>] ? gs_change+0xb/0xb
[  765.241265] 
=============================================================================
[  765.241267] BUG kmalloc-512: Object padding overwritten
[  765.241268] 
-----------------------------------------------------------------------------
[  765.241269]
[  765.241271] INFO: 0xffff8804168c1486-0xffff8804168c1487. First byte 
0x0 instead of 0x5a
[  765.241277] INFO: Allocated in bio_kmalloc+0x2d/0x70 age=73010 cpu=5 
pid=2495
[  765.241281] INFO: Freed in r10buf_pool_free+0x71/0x90 age=72999 cpu=5 
pid=1728
[  765.241283] INFO: Slab 0xffffea000e4eea00 objects=28 used=13 
fp=0xffff8804168c1240 flags=0x80000000000040c1
[  765.241284] INFO: Object 0xffff8804168c1240 @offset=4672 
fp=0xffff8804168c0ff8
[  765.241285]
[  765.241286] Bytes b4 0xffff8804168c1230:  95 98 ff ff 00 00 00 00 5a 
5a 5a 5a 5a 5a 5a 5a ..ÿÿ....ZZZZZZZZ
[  765.241291]   Object 0xffff8804168c1240:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  765.241294]   Object 0xffff8804168c1250:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  765.241298]   Object 0xffff8804168c1260:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  765.241301]   Object 0xffff8804168c1270:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  765.241305]   Object 0xffff8804168c1280:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  765.241308]   Object 0xffff8804168c1290:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  765.241312]   Object 0xffff8804168c12a0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  765.241315]   Object 0xffff8804168c12b0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  765.241319]   Object 0xffff8804168c12c0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  765.241322]   Object 0xffff8804168c12d0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  765.241326]   Object 0xffff8804168c12e0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  765.241329]   Object 0xffff8804168c12f0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  765.241332]   Object 0xffff8804168c1300:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  765.241336]   Object 0xffff8804168c1310:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  765.241339]   Object 0xffff8804168c1320:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  765.241343]   Object 0xffff8804168c1330:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  765.241346]   Object 0xffff8804168c1340:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  765.241350]   Object 0xffff8804168c1350:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  765.241353]   Object 0xffff8804168c1360:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  765.241357]   Object 0xffff8804168c1370:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  765.241360]   Object 0xffff8804168c1380:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  765.241364]   Object 0xffff8804168c1390:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  765.241367]   Object 0xffff8804168c13a0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  765.241371]   Object 0xffff8804168c13b0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  765.241374]   Object 0xffff8804168c13c0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  765.241378]   Object 0xffff8804168c13d0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  765.241381]   Object 0xffff8804168c13e0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  765.241385]   Object 0xffff8804168c13f0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  765.241388]   Object 0xffff8804168c1400:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  765.241392]   Object 0xffff8804168c1410:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  765.241395]   Object 0xffff8804168c1420:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  765.241399]   Object 0xffff8804168c1430:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b a5 kkkkkkkkkkkkkkk¥
[  765.241402]  Redzone 0xffff8804168c1440:  bb bb bb bb bb bb bb bb 
                      »»»»»»»»
[  765.241406]  Padding 0xffff8804168c1480:  5a 5a 5a 5a 5a 5a 00 00 
                      ZZZZZZ..
[  765.241410] Pid: 7147, comm: apcaccess Not tainted 2.6.39 #2
[  765.241411] Call Trace:
[  765.241416]  [<ffffffff810b7ccd>] ? check_bytes_and_report+0x10d/0x150
[  765.241419]  [<ffffffff8110773c>] ? load_elf_binary+0xa6c/0x1c00
[  765.241421]  [<ffffffff810b7db9>] ? check_object+0xa9/0x260
[  765.241423]  [<ffffffff8110773c>] ? load_elf_binary+0xa6c/0x1c00
[  765.241424]  [<ffffffff810b86f4>] ? alloc_debug_processing+0x104/0x190
[  765.241426]  [<ffffffff810b9ac2>] ? T.912+0x272/0x2d0
[  765.241428]  [<ffffffff810ba59d>] ? __kmalloc+0x10d/0x160
[  765.241429]  [<ffffffff8110773c>] ? load_elf_binary+0xa6c/0x1c00
[  765.241432]  [<ffffffff8109bd9d>] ? __get_user_pages+0x17d/0x530
[  765.241434]  [<ffffffff810c9556>] ? get_arg_page+0x56/0x100
[  765.241437]  [<ffffffff811da4ad>] ? strnlen_user+0x2d/0x80
[  765.241439]  [<ffffffff810c8070>] ? search_binary_handler+0x90/0x240
[  765.241440]  [<ffffffff810c9e9f>] ? do_execve+0x22f/0x2f0
[  765.241443]  [<ffffffff810094a6>] ? sys_execve+0x36/0x60
[  765.241446]  [<ffffffff813d78dc>] ? stub_execve+0x6c/0xc0
[  765.241448] FIX kmalloc-512: Restoring 
0xffff8804168c1486-0xffff8804168c1487=0x5a
[  765.241449]
[  789.536436] 
=============================================================================
[  789.536444] BUG kmalloc-2048: Object padding overwritten
[  789.536448] 
-----------------------------------------------------------------------------
[  789.536451]
[  789.536456] INFO: 0xffff8802c8b1a11e-0xffff8802c8b1a11f. First byte 
0x0 instead of 0x5a
[  789.536471] INFO: Allocated in sk_stream_alloc_skb+0x3a/0x110 age=63 
cpu=3 pid=5169
[  789.536482] INFO: Freed in __kfree_skb+0x11/0x90 age=63 cpu=5 pid=0
[  789.536488] INFO: Slab 0xffffea0009be6d40 objects=15 used=4 
fp=0xffff8802c8b198d8 flags=0x80000000000040c1
[  789.536494] INFO: Object 0xffff8802c8b198d8 @offset=6360 
fp=0xffff8802c8b1a968
[  789.536497]
[  789.536501] Bytes b4 0xffff8802c8b198c8:  cc be 00 00 01 00 00 00 5a 
5a 5a 5a 5a 5a 5a 5a ÌŸ......ZZZZZZZZ
[  789.536517]   Object 0xffff8802c8b198d8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.536532]   Object 0xffff8802c8b198e8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.536546]   Object 0xffff8802c8b198f8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.536560]   Object 0xffff8802c8b19908:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.536575]   Object 0xffff8802c8b19918:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.536589]   Object 0xffff8802c8b19928:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.536603]   Object 0xffff8802c8b19938:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.536617]   Object 0xffff8802c8b19948:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.536631]   Object 0xffff8802c8b19958:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.536645]   Object 0xffff8802c8b19968:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.536660]   Object 0xffff8802c8b19978:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.536674]   Object 0xffff8802c8b19988:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.536688]   Object 0xffff8802c8b19998:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.536702]   Object 0xffff8802c8b199a8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.536716]   Object 0xffff8802c8b199b8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.536730]   Object 0xffff8802c8b199c8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.536745]   Object 0xffff8802c8b199d8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.536759]   Object 0xffff8802c8b199e8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.536773]   Object 0xffff8802c8b199f8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.536787]   Object 0xffff8802c8b19a08:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.536801]   Object 0xffff8802c8b19a18:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.536816]   Object 0xffff8802c8b19a28:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.536830]   Object 0xffff8802c8b19a38:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.536844]   Object 0xffff8802c8b19a48:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.536858]   Object 0xffff8802c8b19a58:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.536872]   Object 0xffff8802c8b19a68:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.536887]   Object 0xffff8802c8b19a78:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.536901]   Object 0xffff8802c8b19a88:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.536915]   Object 0xffff8802c8b19a98:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.536929]   Object 0xffff8802c8b19aa8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.536943]   Object 0xffff8802c8b19ab8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.536958]   Object 0xffff8802c8b19ac8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.536972]   Object 0xffff8802c8b19ad8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.536986]   Object 0xffff8802c8b19ae8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537000]   Object 0xffff8802c8b19af8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537014]   Object 0xffff8802c8b19b08:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537029]   Object 0xffff8802c8b19b18:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537043]   Object 0xffff8802c8b19b28:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537057]   Object 0xffff8802c8b19b38:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537071]   Object 0xffff8802c8b19b48:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537085]   Object 0xffff8802c8b19b58:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537100]   Object 0xffff8802c8b19b68:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537114]   Object 0xffff8802c8b19b78:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537128]   Object 0xffff8802c8b19b88:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537143]   Object 0xffff8802c8b19b98:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537158]   Object 0xffff8802c8b19ba8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537172]   Object 0xffff8802c8b19bb8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537187]   Object 0xffff8802c8b19bc8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537201]   Object 0xffff8802c8b19bd8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537215]   Object 0xffff8802c8b19be8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537229]   Object 0xffff8802c8b19bf8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537244]   Object 0xffff8802c8b19c08:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537258]   Object 0xffff8802c8b19c18:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537272]   Object 0xffff8802c8b19c28:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537286]   Object 0xffff8802c8b19c38:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537301]   Object 0xffff8802c8b19c48:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537315]   Object 0xffff8802c8b19c58:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537329]   Object 0xffff8802c8b19c68:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537343]   Object 0xffff8802c8b19c78:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537357]   Object 0xffff8802c8b19c88:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537372]   Object 0xffff8802c8b19c98:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537386]   Object 0xffff8802c8b19ca8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537400]   Object 0xffff8802c8b19cb8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537414]   Object 0xffff8802c8b19cc8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537428]   Object 0xffff8802c8b19cd8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537443]   Object 0xffff8802c8b19ce8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537457]   Object 0xffff8802c8b19cf8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537471]   Object 0xffff8802c8b19d08:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537485]   Object 0xffff8802c8b19d18:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537500]   Object 0xffff8802c8b19d28:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537514]   Object 0xffff8802c8b19d38:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537528]   Object 0xffff8802c8b19d48:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537542]   Object 0xffff8802c8b19d58:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537557]   Object 0xffff8802c8b19d68:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537571]   Object 0xffff8802c8b19d78:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537585]   Object 0xffff8802c8b19d88:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537599]   Object 0xffff8802c8b19d98:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537613]   Object 0xffff8802c8b19da8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537628]   Object 0xffff8802c8b19db8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537642]   Object 0xffff8802c8b19dc8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537656]   Object 0xffff8802c8b19dd8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537670]   Object 0xffff8802c8b19de8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537685]   Object 0xffff8802c8b19df8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537699]   Object 0xffff8802c8b19e08:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537713]   Object 0xffff8802c8b19e18:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537727]   Object 0xffff8802c8b19e28:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537742]   Object 0xffff8802c8b19e38:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537756]   Object 0xffff8802c8b19e48:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537770]   Object 0xffff8802c8b19e58:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537784]   Object 0xffff8802c8b19e68:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537799]   Object 0xffff8802c8b19e78:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537813]   Object 0xffff8802c8b19e88:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537827]   Object 0xffff8802c8b19e98:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537841]   Object 0xffff8802c8b19ea8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537856]   Object 0xffff8802c8b19eb8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537870]   Object 0xffff8802c8b19ec8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537884]   Object 0xffff8802c8b19ed8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537898]   Object 0xffff8802c8b19ee8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537912]   Object 0xffff8802c8b19ef8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537927]   Object 0xffff8802c8b19f08:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537941]   Object 0xffff8802c8b19f18:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537955]   Object 0xffff8802c8b19f28:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537969]   Object 0xffff8802c8b19f38:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537984]   Object 0xffff8802c8b19f48:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537998]   Object 0xffff8802c8b19f58:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.538012]   Object 0xffff8802c8b19f68:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.538026]   Object 0xffff8802c8b19f78:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.538041]   Object 0xffff8802c8b19f88:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.538055]   Object 0xffff8802c8b19f98:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.538069]   Object 0xffff8802c8b19fa8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.538083]   Object 0xffff8802c8b19fb8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.538097]   Object 0xffff8802c8b19fc8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.538112]   Object 0xffff8802c8b19fd8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.538126]   Object 0xffff8802c8b19fe8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.538140]   Object 0xffff8802c8b19ff8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.538154]   Object 0xffff8802c8b1a008:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.538169]   Object 0xffff8802c8b1a018:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.538183]   Object 0xffff8802c8b1a028:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.538197]   Object 0xffff8802c8b1a038:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.538211]   Object 0xffff8802c8b1a048:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.538226]   Object 0xffff8802c8b1a058:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.538240]   Object 0xffff8802c8b1a068:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.538254]   Object 0xffff8802c8b1a078:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.538268]   Object 0xffff8802c8b1a088:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.538283]   Object 0xffff8802c8b1a098:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.538297]   Object 0xffff8802c8b1a0a8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.538311]   Object 0xffff8802c8b1a0b8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.538325]   Object 0xffff8802c8b1a0c8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b a5 kkkkkkkkkkkkkkk¥
[  789.538340]  Redzone 0xffff8802c8b1a0d8:  bb bb bb bb bb bb bb bb 
                      »»»»»»»»
[  789.538353]  Padding 0xffff8802c8b1a118:  5a 5a 5a 5a 5a 5a 00 00 
                      ZZZZZZ..
[  789.538369] Pid: 5247, comm: qemu Not tainted 2.6.39 #2
[  789.538373] Call Trace:
[  789.538385]  [<ffffffff810b7ccd>] ? check_bytes_and_report+0x10d/0x150
[  789.538394]  [<ffffffff81314260>] ? sock_alloc_send_pskb+0x1d0/0x320
[  789.538400]  [<ffffffff810b7db9>] ? check_object+0xa9/0x260
[  789.538407]  [<ffffffff81314260>] ? sock_alloc_send_pskb+0x1d0/0x320
[  789.538414]  [<ffffffff810b86f4>] ? alloc_debug_processing+0x104/0x190
[  789.538420]  [<ffffffff810b9ac2>] ? T.912+0x272/0x2d0
[  789.538427]  [<ffffffff810bb70d>] ? __kmalloc_track_caller+0x10d/0x160
[  789.538434]  [<ffffffff813198c2>] ? __alloc_skb+0x72/0x160
[  789.538441]  [<ffffffff81314260>] ? sock_alloc_send_pskb+0x1d0/0x320
[  789.538450]  [<ffffffff8105af7b>] ? getnstimeofday+0x4b/0xd0
[  789.538457]  [<ffffffff81320d32>] ? netif_rx+0xb2/0x120
[  789.538465]  [<ffffffff812b393e>] ? tun_chr_aio_write+0x18e/0x4e0
[  789.538472]  [<ffffffff812b37b0>] ? tun_sendmsg+0x4d0/0x4d0
[  789.538481]  [<ffffffff810c24e9>] ? do_sync_readv_writev+0xa9/0xf0
[  789.538489]  [<ffffffff810c231b>] ? rw_copy_check_uvector+0x7b/0x140
[  789.538496]  [<ffffffff810c2bcf>] ? do_readv_writev+0xdf/0x210
[  789.538504]  [<ffffffff810c2e7e>] ? sys_writev+0x4e/0xc0
[  789.538513]  [<ffffffff813d753b>] ? system_call_fastpath+0x16/0x1b
[  789.538519] FIX kmalloc-2048: Restoring 
0xffff8802c8b1a11e-0xffff8802c8b1a11f=0x5a
[  789.538522]
[  825.601291] 
=============================================================================
[  825.601298] BUG kmalloc-512: Object padding overwritten
[  825.601302] 
-----------------------------------------------------------------------------
[  825.601306]
[  825.601311] INFO: 0xffff88041411efe6-0xffff88041411efe7. First byte 
0x0 instead of 0x5a
[  825.601327] INFO: Allocated in sock_alloc_send_pskb+0x1d0/0x320 
age=36742 cpu=3 pid=4846
[  825.601338] INFO: Freed in __kfree_skb+0x11/0x90 age=36742 cpu=3 pid=3781
[  825.601344] INFO: Slab 0xffffea000e463e20 objects=28 used=11 
fp=0xffff88041411eda0 flags=0x80000000000040c1
[  825.601350] INFO: Object 0xffff88041411eda0 @offset=11680 
fp=0xffff88041411cdb0
[  825.601353]
[  825.601357] Bytes b4 0xffff88041411ed90:  e2 96 ff ff 00 00 00 00 5a 
5a 5a 5a 5a 5a 5a 5a â.ÿÿ....ZZZZZZZZ
[  825.601373]   Object 0xffff88041411eda0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  825.601388]   Object 0xffff88041411edb0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  825.601402]   Object 0xffff88041411edc0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  825.601416]   Object 0xffff88041411edd0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  825.601430]   Object 0xffff88041411ede0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  825.601444]   Object 0xffff88041411edf0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  825.601459]   Object 0xffff88041411ee00:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  825.601473]   Object 0xffff88041411ee10:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  825.601487]   Object 0xffff88041411ee20:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  825.601501]   Object 0xffff88041411ee30:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  825.601515]   Object 0xffff88041411ee40:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  825.601529]   Object 0xffff88041411ee50:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  825.601543]   Object 0xffff88041411ee60:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  825.601557]   Object 0xffff88041411ee70:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  825.601571]   Object 0xffff88041411ee80:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  825.601586]   Object 0xffff88041411ee90:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  825.601600]   Object 0xffff88041411eea0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  825.601614]   Object 0xffff88041411eeb0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  825.601628]   Object 0xffff88041411eec0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  825.601642]   Object 0xffff88041411eed0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  825.601656]   Object 0xffff88041411eee0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  825.601670]   Object 0xffff88041411eef0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  825.601684]   Object 0xffff88041411ef00:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  825.601699]   Object 0xffff88041411ef10:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  825.601713]   Object 0xffff88041411ef20:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  825.601727]   Object 0xffff88041411ef30:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  825.601741]   Object 0xffff88041411ef40:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  825.601755]   Object 0xffff88041411ef50:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  825.601769]   Object 0xffff88041411ef60:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  825.601783]   Object 0xffff88041411ef70:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  825.601797]   Object 0xffff88041411ef80:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  825.601812]   Object 0xffff88041411ef90:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b a5 kkkkkkkkkkkkkkk¥
[  825.601826]  Redzone 0xffff88041411efa0:  bb bb bb bb bb bb bb bb 
                      »»»»»»»»
[  825.601839]  Padding 0xffff88041411efe0:  5a 5a 5a 5a 5a 5a 00 00 
                      ZZZZZZ..
[  825.601855] Pid: 7353, comm: apcaccess Not tainted 2.6.39 #2
[  825.601859] Call Trace:
[  825.601871]  [<ffffffff810b7ccd>] ? check_bytes_and_report+0x10d/0x150
[  825.601881]  [<ffffffff8110773c>] ? load_elf_binary+0xa6c/0x1c00
[  825.601887]  [<ffffffff810b7db9>] ? check_object+0xa9/0x260
[  825.601895]  [<ffffffff8110773c>] ? load_elf_binary+0xa6c/0x1c00
[  825.601902]  [<ffffffff810b86f4>] ? alloc_debug_processing+0x104/0x190
[  825.601908]  [<ffffffff810b9ac2>] ? T.912+0x272/0x2d0
[  825.601914]  [<ffffffff810ba59d>] ? __kmalloc+0x10d/0x160
[  825.601922]  [<ffffffff8110773c>] ? load_elf_binary+0xa6c/0x1c00
[  825.601930]  [<ffffffff8109bd9d>] ? __get_user_pages+0x17d/0x530
[  825.601938]  [<ffffffff810c9556>] ? get_arg_page+0x56/0x100
[  825.601945]  [<ffffffff810c8070>] ? search_binary_handler+0x90/0x240
[  825.601951]  [<ffffffff810c9e9f>] ? do_execve+0x22f/0x2f0
[  825.601959]  [<ffffffff810094a6>] ? sys_execve+0x36/0x60
[  825.601968]  [<ffffffff813d78dc>] ? stub_execve+0x6c/0xc0
[  825.601975] FIX kmalloc-512: Restoring 
0xffff88041411efe6-0xffff88041411efe7=0x5a
[  825.601978]
[  842.610057] INFO: task ksmd:552 blocked for more than 120 seconds.
[  842.610063] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" 
disables this message.
[  842.610069] ksmd            D 0000000000000000     0   552      2 
0x00000000
[  842.610078]  ffff88041db68ca0 0000000000000046 000212d01fc11d00 
ffff880400000000
[  842.610086]  ffffffff81593020 ffff88041c1a9fd8 0000000000004000 
ffff88041c1a8010
[  842.610093]  ffff88041c1a9fd8 ffff88041db68ca0 0000000000000000 
0000000200000002
[  842.610100] Call Trace:
[  842.610116]  [<ffffffff813d5595>] ? schedule_timeout+0x1c5/0x230
[  842.610128]  [<ffffffff8102f69c>] ? enqueue_task_fair+0x14c/0x190
[  842.610137]  [<ffffffff8102db27>] ? task_rq_lock+0x47/0x90
[  842.610143]  [<ffffffff813d4562>] ? wait_for_common+0xd2/0x180
[  842.610150]  [<ffffffff810339b0>] ? try_to_wake_up+0x2c0/0x2c0
[  842.610157]  [<ffffffff8104cb94>] ? flush_work+0x24/0x30
[  842.610163]  [<ffffffff8104be10>] ? do_work_for_cpu+0x20/0x20
[  842.610170]  [<ffffffff8104e0ab>] ? schedule_on_each_cpu+0xab/0xe0
[  842.610177]  [<ffffffff810b5c05>] ? ksm_scan_thread+0x7f5/0xc20
[  842.610184]  [<ffffffff81052a20>] ? wake_up_bit+0x40/0x40
[  842.610190]  [<ffffffff810b5410>] ? 
try_to_merge_with_ksm_page+0x570/0x570
[  842.610196]  [<ffffffff810b5410>] ? 
try_to_merge_with_ksm_page+0x570/0x570
[  842.610203]  [<ffffffff810525b6>] ? kthread+0x96/0xa0
[  842.610210]  [<ffffffff813d8294>] ? kernel_thread_helper+0x4/0x10
[  842.610218]  [<ffffffff81052520>] ? kthread_worker_fn+0x120/0x120
[  842.610223]  [<ffffffff813d8290>] ? gs_change+0xb/0xb
[  842.610229] INFO: task fsnotify_mark:662 blocked for more than 120 
seconds.
[  842.610233] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" 
disables this message.
[  842.610237] fsnotify_mark   D 0000000000000000     0   662      2 
0x00000000
[  842.610244]  ffff88041c31d860 0000000000000046 0000000000000000 
0000000000000000
[  842.610251]  ffffffff81593020 ffff88041c365fd8 0000000000004000 
ffff88041c364010
[  842.610257]  ffff88041c365fd8 ffff88041c31d860 0000000000000000 
0000000000000000
[  842.610263] Call Trace:
[  842.610270]  [<ffffffff81035521>] ? load_balance+0x91/0x5e0
[  842.610278]  [<ffffffff813d5595>] ? schedule_timeout+0x1c5/0x230
[  842.610285]  [<ffffffff8102f323>] ? pick_next_task_fair+0x103/0x190
[  842.610291]  [<ffffffff813d4a6d>] ? schedule+0x28d/0x910
[  842.610297]  [<ffffffff813d4562>] ? wait_for_common+0xd2/0x180
[  842.610303]  [<ffffffff810339b0>] ? try_to_wake_up+0x2c0/0x2c0
[  842.610310]  [<ffffffff81074420>] ? synchronize_rcu_bh+0x50/0x50
[  842.610316]  [<ffffffff8107446a>] ? synchronize_sched+0x4a/0x50
[  842.610322]  [<ffffffff8104f940>] ? find_ge_pid+0x40/0x40
[  842.610329]  [<ffffffff8105742b>] ? __synchronize_srcu+0x5b/0xc0
[  842.610338]  [<ffffffff810f5f63>] ? fsnotify_mark_destroy+0x83/0x150
[  842.610344]  [<ffffffff81052a20>] ? wake_up_bit+0x40/0x40
[  842.610352]  [<ffffffff810f5ee0>] ? 
fsnotify_set_mark_ignored_mask_locked+0x20/0x20
[  842.610360]  [<ffffffff810f5ee0>] ? 
fsnotify_set_mark_ignored_mask_locked+0x20/0x20
[  842.610367]  [<ffffffff810525b6>] ? kthread+0x96/0xa0
[  842.610373]  [<ffffffff813d8294>] ? kernel_thread_helper+0x4/0x10
[  842.610381]  [<ffffffff81052520>] ? kthread_worker_fn+0x120/0x120
[  842.610386]  [<ffffffff813d8290>] ? gs_change+0xb/0xb
[  842.610395] INFO: task jbd2/md0-8:2528 blocked for more than 120 seconds.
[  842.610399] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" 
disables this message.
[  842.610403] jbd2/md0-8      D 0000000000000005     0  2528      2 
0x00000000
[  842.610410]  ffff88041d91c570 0000000000000046 ffff88041baeea28 
ffffea0000000000
[  842.610417]  ffff88041d91a5e0 ffff880419f9bfd8 0000000000004000 
ffff880419f9a010
[  842.610423]  ffff880419f9bfd8 ffff88041d91c570 ffff88041baeea28 
ffffffff810b8495
[  842.610429] Call Trace:
[  842.610435]  [<ffffffff810b8495>] ? init_object+0x85/0xa0
[  842.610442]  [<ffffffff810b8916>] ? free_debug_processing+0x196/0x250
[  842.610450]  [<ffffffff8105adae>] ? ktime_get_ts+0x6e/0xf0
[  842.610458]  [<ffffffff810810f0>] ? __lock_page+0x70/0x70
[  842.610464]  [<ffffffff813d5174>] ? io_schedule+0x84/0xd0
[  842.610473]  [<ffffffff811d4403>] ? 
radix_tree_gang_lookup_tag_slot+0x93/0xf0
[  842.610480]  [<ffffffff810810f9>] ? sleep_on_page+0x9/0x10
[  842.610486]  [<ffffffff813d57df>] ? __wait_on_bit+0x4f/0x80
[  842.610492]  [<ffffffff810812eb>] ? wait_on_page_bit+0x6b/0x80
[  842.610499]  [<ffffffff81052a50>] ? autoremove_wake_function+0x30/0x30
[  842.610507]  [<ffffffff8108a458>] ? pagevec_lookup_tag+0x18/0x20
[  842.610512]  [<ffffffff81081f2a>] ? filemap_fdatawait_range+0xfa/0x180
[  842.610521]  [<ffffffff811be09f>] ? submit_bio+0x6f/0xf0
[  842.610530]  [<ffffffff81176276>] ? 
jbd2_journal_commit_transaction+0x796/0x1270
[  842.610539]  [<ffffffff81179ed1>] ? kjournald2+0xb1/0x1e0
[  842.610546]  [<ffffffff81052a20>] ? wake_up_bit+0x40/0x40
[  842.610553]  [<ffffffff81179e20>] ? commit_timeout+0x10/0x10
[  842.610560]  [<ffffffff81179e20>] ? commit_timeout+0x10/0x10
[  842.610566]  [<ffffffff810525b6>] ? kthread+0x96/0xa0
[  842.610572]  [<ffffffff813d8294>] ? kernel_thread_helper+0x4/0x10
[  842.610580]  [<ffffffff81052520>] ? kthread_worker_fn+0x120/0x120
[  842.610585]  [<ffffffff813d8290>] ? gs_change+0xb/0xb
[  842.610594] INFO: task nfsd:4326 blocked for more than 120 seconds.
[  842.610598] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" 
disables this message.
[  842.610602] nfsd            D 0000000000000000     0  4326      2 
0x00000000
[  842.610608]  ffff88041b604bc0 0000000000000046 0000000000000016 
ffffffff00000000
[  842.610614]  ffff88041d8fa5e0 ffff88041c527fd8 0000000000004000 
ffff88041c526010
[  842.610621]  ffff88041c527fd8 ffff88041b604bc0 000000000000009c 
ffff880000000000
[  842.610626] Call Trace:
[  842.610634]  [<ffffffff8101e845>] ? amd_flush_garts+0x105/0x140
[  842.610640]  [<ffffffff8101fa80>] ? gart_map_sg+0x480/0x480
[  842.610646]  [<ffffffff8101f5d3>] ? flush_gart+0x23/0x50
[  842.610653]  [<ffffffff81080f98>] ? find_get_page+0x18/0x90
[  842.610660]  [<ffffffff81174b95>] ? do_get_write_access+0x265/0x4a0
[  842.610668]  [<ffffffff81052a50>] ? autoremove_wake_function+0x30/0x30
[  842.610675]  [<ffffffff81174ef9>] ? 
jbd2_journal_get_write_access+0x29/0x50
[  842.610684]  [<ffffffff8115f122>] ? 
__ext4_journal_get_write_access+0x32/0x80
[  842.610692]  [<ffffffff81143908>] ? ext4_reserve_inode_write+0x78/0xa0
[  842.610700]  [<ffffffff81143970>] ? ext4_mark_inode_dirty+0x40/0x1e0
[  842.610706]  [<ffffffff81156c0b>] ? ext4_journal_start_sb+0x6b/0x160
[  842.610715]  [<ffffffff81322645>] ? dev_hard_start_xmit+0x305/0x5f0
[  842.610723]  [<ffffffff81352fd0>] ? ip_finish_output2+0x290/0x290
[  842.610730]  [<ffffffff81143c65>] ? ext4_dirty_inode+0x35/0x70
[  842.610738]  [<ffffffff810e4a08>] ? __mark_inode_dirty+0x38/0x210
[  842.610745]  [<ffffffff810d9317>] ? file_update_time+0xf7/0x180
[  842.610751]  [<ffffffff81082458>] ? __generic_file_aio_write+0x1f8/0x430
[  842.610760]  [<ffffffff81373ef9>] ? udp_sendmsg+0x3c9/0x7e0
[  842.610767]  [<ffffffff81314260>] ? sock_alloc_send_pskb+0x1d0/0x320
[  842.610774]  [<ffffffff81082703>] ? generic_file_aio_write+0x73/0xf0
[  842.610781]  [<ffffffff8113f74e>] ? ext4_file_write+0x6e/0x2b0
[  842.610788]  [<ffffffff810da8ac>] ? iget_locked+0x4c/0x140
[  842.610794]  [<ffffffff8119a590>] ? fh_compose+0x4c0/0x4c0
[  842.610800]  [<ffffffff8113f6e0>] ? ext4_llseek+0x110/0x110
[  842.610808]  [<ffffffff810c24e9>] ? do_sync_readv_writev+0xa9/0xf0
[  842.610816]  [<ffffffff810c231b>] ? rw_copy_check_uvector+0x7b/0x140
[  842.610824]  [<ffffffff810c2bcf>] ? do_readv_writev+0xdf/0x210
[  842.610830]  [<ffffffff810b8735>] ? alloc_debug_processing+0x145/0x190
[  842.610836]  [<ffffffff810b9bc5>] ? kmem_cache_alloc+0xa5/0xb0
[  842.610843]  [<ffffffff8113f463>] ? ext4_file_open+0x63/0x180
[  842.610849]  [<ffffffff8119c21d>] ? nfsd_vfs_write+0xed/0x3a0
[  842.610856]  [<ffffffff810c1127>] ? __dentry_open+0x1f7/0x2b0
[  842.610862]  [<ffffffff8119c892>] ? nfsd_open+0xf2/0x1b0
[  842.610867]  [<ffffffff8119cd34>] ? nfsd_write+0xf4/0x110
[  842.610873]  [<ffffffff81199930>] ? nfsd_proc_write+0xb0/0x120
[  842.610880]  [<ffffffff811971c5>] ? nfsd_dispatch+0xf5/0x230
[  842.610886]  [<ffffffff813b456f>] ? svc_process+0x4af/0x820
[  842.610892]  [<ffffffff810339b0>] ? try_to_wake_up+0x2c0/0x2c0
[  842.610899]  [<ffffffff811977a0>] ? nfsd_svc+0x1b0/0x1b0
[  842.610906]  [<ffffffff8119784d>] ? nfsd+0xad/0x150
[  842.610912]  [<ffffffff810525b6>] ? kthread+0x96/0xa0
[  842.610918]  [<ffffffff813d8294>] ? kernel_thread_helper+0x4/0x10
[  842.610926]  [<ffffffff81052520>] ? kthread_worker_fn+0x120/0x120
[  842.610931]  [<ffffffff813d8290>] ? gs_change+0xb/0xb
[  854.241450] 
=============================================================================
[  854.241459] BUG kmalloc-1024: Object padding overwritten
[  854.241463] 
-----------------------------------------------------------------------------
[  854.241466]
[  854.241471] INFO: 0xffff8802c86059e6-0xffff8802c86059e7. First byte 
0x0 instead of 0x5a
[  854.241488] INFO: Allocated in __blockdev_direct_IO+0x16c/0xa90 
age=518 cpu=5 pid=7094
[  854.241500] INFO: Freed in __kfree_skb+0x11/0x90 age=519 cpu=5 pid=5247
[  854.241507] INFO: Slab 0xffffea0009bd5000 objects=29 used=22 
fp=0xffff8802c8606f50 flags=0x80000000000040c1
[  854.241513] INFO: Object 0xffff8802c86055a0 @offset=21920 
fp=0xffff8802c86066c0
[  854.241516]
[  854.241520] Bytes b4 0xffff8802c8605590:  00 00 00 00 00 00 00 00 5a 
5a 5a 5a 5a 5a 5a 5a ........ZZZZZZZZ
[  854.241536]   Object 0xffff8802c86055a0:  00 00 00 00 00 00 00 00 e0 
61 29 fa 03 88 ff ff ........àa)ú..ÿÿ
[  854.241551]   Object 0xffff8802c86055b0:  00 00 00 00 00 00 00 00 00 
00 41 ce 04 00 00 00 ..........AÎ....
[  854.241565]   Object 0xffff8802c86055c0:  03 00 00 00 0c 00 00 00 00 
00 00 00 01 00 00 00 ................
[  854.241579]   Object 0xffff8802c86055d0:  00 00 00 00 00 00 00 00 00 
60 00 00 00 00 00 00 .........`......
[  854.241593]   Object 0xffff8802c86055e0:  b7 2c 0a 00 00 00 00 00 00 
00 00 00 00 00 00 00 ·,..............
[  854.241607]   Object 0xffff8802c86055f0:  b7 2c 0a 00 00 00 00 00 00 
00 00 00 00 00 00 00 ·,..............
[  854.241621]   Object 0xffff8802c8605600:  01 00 00 00 00 00 00 00 f0 
77 14 81 ff ff ff ff ........ðw..ÿÿÿÿ
[  854.241635]   Object 0xffff8802c8605610:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[  854.241649]   Object 0xffff8802c8605620:  00 00 00 00 00 00 00 00 b7 
54 3b 02 00 00 00 00 ........·T;.....
[  854.241663]   Object 0xffff8802c8605630:  b7 54 3b 02 00 00 00 00 20 
00 00 00 00 00 00 00 ·T;.............
[  854.241677]   Object 0xffff8802c8605640:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[  854.241691]   Object 0xffff8802c8605650:  b6 54 3b 02 00 00 00 00 00 
10 00 00 00 00 00 00 ¶T;.............
[  854.241705]   Object 0xffff8802c8605660:  00 00 00 00 00 00 00 00 00 
80 8e 1a 04 88 ff ff ..............ÿÿ
[  854.241719]   Object 0xffff8802c8605670:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[  854.241733]   Object 0xffff8802c8605680:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[  854.241747]   Object 0xffff8802c8605690:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[  854.241761]   Object 0xffff8802c86056a0:  00 00 00 00 00 00 00 00 00 
00 00 00 00 10 00 00 ................
[  854.241775]   Object 0xffff8802c86056b0:  b6 54 3b 02 00 00 00 00 00 
60 cb a2 00 00 00 00 ¶T;......`Ë¢....
[  854.241789]   Object 0xffff8802c86056c0:  06 06 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[  854.241803]   Object 0xffff8802c86056d0:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[  854.241817]   Object 0xffff8802c86056e0:  68 fd 1c c9 02 88 ff ff 00 
00 00 00 00 00 00 00 hý.É..ÿÿ........
[  854.241831]   Object 0xffff8802c86056f0:  00 60 00 00 00 00 00 00 01 
00 00 00 01 00 00 00 .`..............
[  854.241845]   Object 0xffff8802c8605700:  00 80 c9 dd 39 7f 00 00 01 
00 00 00 01 00 00 00 ..ÉÝ9...........
[  854.241860]   Object 0xffff8802c8605710:  00 00 00 00 00 00 00 00 08 
81 bb 0c 00 ea ff ff ..........»..êÿÿ
[  854.241874]   Object 0xffff8802c8605720:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.241888]   Object 0xffff8802c8605730:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.241902]   Object 0xffff8802c8605740:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.241916]   Object 0xffff8802c8605750:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.241930]   Object 0xffff8802c8605760:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.241944]   Object 0xffff8802c8605770:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.241959]   Object 0xffff8802c8605780:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.241973]   Object 0xffff8802c8605790:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.241987]   Object 0xffff8802c86057a0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.242001]   Object 0xffff8802c86057b0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.242015]   Object 0xffff8802c86057c0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.242029]   Object 0xffff8802c86057d0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.242043]   Object 0xffff8802c86057e0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.242058]   Object 0xffff8802c86057f0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.242072]   Object 0xffff8802c8605800:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.242086]   Object 0xffff8802c8605810:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.242100]   Object 0xffff8802c8605820:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.242114]   Object 0xffff8802c8605830:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.242128]   Object 0xffff8802c8605840:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.242142]   Object 0xffff8802c8605850:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.242157]   Object 0xffff8802c8605860:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.242171]   Object 0xffff8802c8605870:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.242185]   Object 0xffff8802c8605880:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.242199]   Object 0xffff8802c8605890:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.242213]   Object 0xffff8802c86058a0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.242227]   Object 0xffff8802c86058b0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.242241]   Object 0xffff8802c86058c0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.242256]   Object 0xffff8802c86058d0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.242270]   Object 0xffff8802c86058e0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.242284]   Object 0xffff8802c86058f0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.242298]   Object 0xffff8802c8605900:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.242312]   Object 0xffff8802c8605910:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.242326]   Object 0xffff8802c8605920:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.242340]   Object 0xffff8802c8605930:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.242355]   Object 0xffff8802c8605940:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.242369]   Object 0xffff8802c8605950:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.242383]   Object 0xffff8802c8605960:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.242397]   Object 0xffff8802c8605970:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.242411]   Object 0xffff8802c8605980:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.242425]   Object 0xffff8802c8605990:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b a5 kkkkkkkkkkkkkkk¥
[  854.242440]  Redzone 0xffff8802c86059a0:  cc cc cc cc cc cc cc cc 
                      ÌÌÌÌÌÌÌÌ
[  854.242453]  Padding 0xffff8802c86059e0:  5a 5a 5a 5a 5a 5a 00 00 
                      ZZZZZZ..
[  854.242470] Pid: 7094, comm: qemu Not tainted 2.6.39 #2
[  854.242474] Call Trace:
[  854.242485]  [<ffffffff810b7ccd>] ? check_bytes_and_report+0x10d/0x150
[  854.242492]  [<ffffffff810b7db9>] ? check_object+0xa9/0x260
[  854.242500]  [<ffffffff810f3126>] ? __blockdev_direct_IO+0xa16/0xa90
[  854.242507]  [<ffffffff810b88b3>] ? free_debug_processing+0x133/0x250
[  854.242513]  [<ffffffff810b8afb>] ? __slab_free+0x12b/0x140
[  854.242520]  [<ffffffff810f3126>] ? __blockdev_direct_IO+0xa16/0xa90
[  854.242530]  [<ffffffff81144a17>] ? ext4_ind_direct_IO+0xf7/0x410
[  854.242536]  [<ffffffff811477f0>] ? noalloc_get_block_write+0x30/0x30
[  854.242543]  [<ffffffff81082029>] ? __filemap_fdatawrite_range+0x49/0x50
[  854.242550]  [<ffffffff810830c3>] ? generic_file_aio_read+0x653/0x6d0
[  854.242558]  [<ffffffff810567a2>] ? hrtimer_cancel+0x12/0x20
[  854.242565]  [<ffffffff81062137>] ? futex_wait+0x197/0x240
[  854.242572]  [<ffffffff81082a70>] ? delete_from_page_cache+0x70/0x70
[  854.242580]  [<ffffffff810c24e9>] ? do_sync_readv_writev+0xa9/0xf0
[  854.242587]  [<ffffffff8106239f>] ? futex_wake+0x10f/0x120
[  854.242594]  [<ffffffff81063f7b>] ? do_futex+0x11b/0xa70
[  854.242601]  [<ffffffff810471cc>] ? T.680+0x13c/0x280
[  854.242608]  [<ffffffff810c231b>] ? rw_copy_check_uvector+0x7b/0x140
[  854.242616]  [<ffffffff810c2bcf>] ? do_readv_writev+0xdf/0x210
[  854.242624]  [<ffffffff810c3003>] ? sys_preadv+0xc3/0xd0
[  854.242633]  [<ffffffff813d753b>] ? system_call_fastpath+0x16/0x1b
[  854.242640] FIX kmalloc-1024: Restoring 
0xffff8802c86059e6-0xffff8802c86059e7=0x5a
[  854.242643]
[  860.203918] 
=============================================================================
[  860.203927] BUG kmalloc-1024: Object padding overwritten
[  860.203931] 
-----------------------------------------------------------------------------
[  860.203934]
[  860.203939] INFO: 0xffff88041ad15156-0xffff88041ad15157. First byte 
0x0 instead of 0x5a
[  860.203955] INFO: Allocated in __blockdev_direct_IO+0x16c/0xa90 
age=1609 cpu=3 pid=7094
[  860.203965] INFO: Freed in __blockdev_direct_IO+0xa16/0xa90 age=1609 
cpu=3 pid=7094
[  860.203972] INFO: Slab 0xffffea000e5ddb80 objects=29 used=25 
fp=0xffff88041ad14d10 flags=0x80000000000040c1
[  860.203977] INFO: Object 0xffff88041ad14d10 @offset=19728 
fp=0xffff88041ad10000
[  860.203981]
[  860.203984] Bytes b4 0xffff88041ad14d00:  62 28 00 00 01 00 00 00 5a 
5a 5a 5a 5a 5a 5a 5a b(......ZZZZZZZZ
[  860.204000]   Object 0xffff88041ad14d10:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204015]   Object 0xffff88041ad14d20:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204030]   Object 0xffff88041ad14d30:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204044]   Object 0xffff88041ad14d40:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204058]   Object 0xffff88041ad14d50:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204073]   Object 0xffff88041ad14d60:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204087]   Object 0xffff88041ad14d70:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204101]   Object 0xffff88041ad14d80:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204116]   Object 0xffff88041ad14d90:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204130]   Object 0xffff88041ad14da0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204144]   Object 0xffff88041ad14db0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204159]   Object 0xffff88041ad14dc0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204173]   Object 0xffff88041ad14dd0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204187]   Object 0xffff88041ad14de0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204202]   Object 0xffff88041ad14df0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204216]   Object 0xffff88041ad14e00:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204230]   Object 0xffff88041ad14e10:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204245]   Object 0xffff88041ad14e20:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204259]   Object 0xffff88041ad14e30:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204273]   Object 0xffff88041ad14e40:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204288]   Object 0xffff88041ad14e50:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204302]   Object 0xffff88041ad14e60:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204316]   Object 0xffff88041ad14e70:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204331]   Object 0xffff88041ad14e80:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204345]   Object 0xffff88041ad14e90:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204359]   Object 0xffff88041ad14ea0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204374]   Object 0xffff88041ad14eb0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204388]   Object 0xffff88041ad14ec0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204402]   Object 0xffff88041ad14ed0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204417]   Object 0xffff88041ad14ee0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204431]   Object 0xffff88041ad14ef0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204445]   Object 0xffff88041ad14f00:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204460]   Object 0xffff88041ad14f10:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204474]   Object 0xffff88041ad14f20:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204488]   Object 0xffff88041ad14f30:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204503]   Object 0xffff88041ad14f40:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204517]   Object 0xffff88041ad14f50:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204531]   Object 0xffff88041ad14f60:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204546]   Object 0xffff88041ad14f70:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204560]   Object 0xffff88041ad14f80:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204574]   Object 0xffff88041ad14f90:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204589]   Object 0xffff88041ad14fa0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204603]   Object 0xffff88041ad14fb0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204618]   Object 0xffff88041ad14fc0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204632]   Object 0xffff88041ad14fd0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204646]   Object 0xffff88041ad14fe0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204661]   Object 0xffff88041ad14ff0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204675]   Object 0xffff88041ad15000:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204689]   Object 0xffff88041ad15010:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204704]   Object 0xffff88041ad15020:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204718]   Object 0xffff88041ad15030:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204732]   Object 0xffff88041ad15040:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204747]   Object 0xffff88041ad15050:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204761]   Object 0xffff88041ad15060:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204775]   Object 0xffff88041ad15070:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204790]   Object 0xffff88041ad15080:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204804]   Object 0xffff88041ad15090:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204818]   Object 0xffff88041ad150a0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204833]   Object 0xffff88041ad150b0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204847]   Object 0xffff88041ad150c0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204861]   Object 0xffff88041ad150d0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204876]   Object 0xffff88041ad150e0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204890]   Object 0xffff88041ad150f0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204904]   Object 0xffff88041ad15100:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b a5 kkkkkkkkkkkkkkk¥
[  860.204919]  Redzone 0xffff88041ad15110:  bb bb bb bb bb bb bb bb 
                      »»»»»»»»
[  860.204932]  Padding 0xffff88041ad15150:  5a 5a 5a 5a 5a 5a 00 00 
                      ZZZZZZ..
[  860.204949] Pid: 7100, comm: qemu Not tainted 2.6.39 #2
[  860.204953] Call Trace:
[  860.204964]  [<ffffffff810b7ccd>] ? check_bytes_and_report+0x10d/0x150
[  860.204973]  [<ffffffff810f287c>] ? __blockdev_direct_IO+0x16c/0xa90
[  860.204979]  [<ffffffff810b7db9>] ? check_object+0xa9/0x260
[  860.204986]  [<ffffffff810f287c>] ? __blockdev_direct_IO+0x16c/0xa90
[  860.204993]  [<ffffffff810b86f4>] ? alloc_debug_processing+0x104/0x190
[  860.204999]  [<ffffffff810b9ac2>] ? T.912+0x272/0x2d0
[  860.205006]  [<ffffffff810b9bc5>] ? kmem_cache_alloc+0xa5/0xb0
[  860.205013]  [<ffffffff810f287c>] ? __blockdev_direct_IO+0x16c/0xa90
[  860.205022]  [<ffffffff8108a475>] ? pagevec_lookup+0x15/0x20
[  860.205029]  [<ffffffff8108b5b7>] ? 
invalidate_inode_pages2_range+0x87/0x300
[  860.205039]  [<ffffffff81144e41>] ? ext4_direct_IO+0x111/0x1f0
[  860.205045]  [<ffffffff811477a0>] ? _ext4_get_block+0x160/0x160
[  860.205052]  [<ffffffff81144f20>] ? ext4_direct_IO+0x1f0/0x1f0
[  860.205058]  [<ffffffff81082182>] ? generic_file_direct_write+0xd2/0x1b0
[  860.205064]  [<ffffffff8108252b>] ? __generic_file_aio_write+0x2cb/0x430
[  860.205071]  [<ffffffff81082703>] ? generic_file_aio_write+0x73/0xf0
[  860.205079]  [<ffffffff8113f74e>] ? ext4_file_write+0x6e/0x2b0
[  860.205087]  [<ffffffff810c25ef>] ? do_sync_write+0xbf/0x100
[  860.205095]  [<ffffffff810471cc>] ? T.680+0x13c/0x280
[  860.205103]  [<ffffffff810485a8>] ? kill_pid_info+0x38/0x60
[  860.205110]  [<ffffffff81048854>] ? sys_kill+0x94/0x1b0
[  860.205117]  [<ffffffff810c3198>] ? vfs_write+0xc8/0x190
[  860.205124]  [<ffffffff810c32fb>] ? sys_pwrite64+0x9b/0xb0
[  860.205134]  [<ffffffff813d753b>] ? system_call_fastpath+0x16/0x1b
[  860.205140] FIX kmalloc-1024: Restoring 
0xffff88041ad15156-0xffff88041ad15157=0x5a
[  860.205143]

^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
@ 2011-06-01  9:29                       ` Brad Campbell
  0 siblings, 0 replies; 111+ messages in thread
From: Brad Campbell @ 2011-06-01  9:29 UTC (permalink / raw)
  To: Avi Kivity
  Cc: Hugh Dickins, Andrea Arcangeli, Borislav Petkov, linux-kernel,
	kvm, linux-mm

On 01/06/11 14:56, Avi Kivity wrote:
> On 06/01/2011 09:31 AM, Brad Campbell wrote:
>> On 01/06/11 12:52, Hugh Dickins wrote:
>>
>>>
>>> I guess Brad could try SLUB debugging, boot with slub_debug=P
>>> for poisoning perhaps; though it might upset alignments and
>>> drive the problem underground. Or see if the same happens
>>> with SLAB instead of SLUB.
>>
>> Not much use I'm afraid.
>> This is all I get in the log
>>
>> [ 3161.300073]
>> =============================================================================
>>
>> [ 3161.300147] BUG kmalloc-512: Freechain corrupt
>>
>> The qemu process is then frozen, unkillable but reported in state "R"
>>
>> 13881 ? R 3:27 /usr/bin/qemu -S -M pc-0.13 -enable-kvm -m 1024 -smp
>> 2,sockets=2,cores=1,threads=1 -nam
>>
>> The machine then progressively dies until it's frozen solid with no
>> further error messages.
>>
>> I stupidly forgot to do an alt-sysrq-t prior to doing an alt-sysrq-b,
>> but at least it responded to that.
>>
>> On the bright side I can reproduce it at will.
>
> Please try slub_debug=FZPU; that should point the finger (hopefully at
> somebody else).
>

Well the first attempt locked the machine solid. No network, no console..

I saw 
"=========================================================================="

on the console.. nothing after that. Would not respond to sysrq-t or any 
other sysrq combination other than -b, which rebooted the box.


No output on netconsole at all, I had to walk to the other building to 
look at the monitor and reboot it.

The second attempt jammed netconsole again, but I managed to get this 
from an ssh session I already had established. The machine died a slow 
and horrible death, but remained interactive enough for me to reboot it with

echo b > /proc/sysrq-trigger

Nothing else worked.



[  376.269051] 
=============================================================================
[  413.755328] 
=============================================================================
[  413.755337] BUG kmalloc-1024: Object padding overwritten
[  413.755341] 
-----------------------------------------------------------------------------
[  413.755344]
[  413.755349] INFO: 0xffff880417179566-0xffff880417179567. First byte 
0x0 instead of 0x5a
[  413.755363] INFO: Allocated in tcp_send_ack+0x26/0x120 age=5320 cpu=5 
pid=0
[  413.755374] INFO: Freed in __kfree_skb+0x11/0x90 age=5320 cpu=5 pid=0
[  413.755380] INFO: Slab 0xffffea000e50d240 objects=29 used=5 
fp=0xffff880417179120 flags=0x80000000000040c1
[  413.755386] INFO: Object 0xffff880417179120 @offset=4384 
fp=0xffff8804171799b0
[  413.755389]
[  413.755392] Bytes b4 0xffff880417179110:  d9 2b 00 00 01 00 00 00 5a 
5a 5a 5a 5a 5a 5a 5a Ù+......ZZZZZZZZ
[  413.755408]   Object 0xffff880417179120:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755423]   Object 0xffff880417179130:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755438]   Object 0xffff880417179140:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755452]   Object 0xffff880417179150:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755466]   Object 0xffff880417179160:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755480]   Object 0xffff880417179170:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755494]   Object 0xffff880417179180:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755508]   Object 0xffff880417179190:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755522]   Object 0xffff8804171791a0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755536]   Object 0xffff8804171791b0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755551]   Object 0xffff8804171791c0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755565]   Object 0xffff8804171791d0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755579]   Object 0xffff8804171791e0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755593]   Object 0xffff8804171791f0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755607]   Object 0xffff880417179200:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755621]   Object 0xffff880417179210:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755635]   Object 0xffff880417179220:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755650]   Object 0xffff880417179230:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755664]   Object 0xffff880417179240:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755678]   Object 0xffff880417179250:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755692]   Object 0xffff880417179260:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755706]   Object 0xffff880417179270:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755720]   Object 0xffff880417179280:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755734]   Object 0xffff880417179290:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755749]   Object 0xffff8804171792a0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755763]   Object 0xffff8804171792b0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755777]   Object 0xffff8804171792c0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755791]   Object 0xffff8804171792d0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755805]   Object 0xffff8804171792e0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755819]   Object 0xffff8804171792f0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755834]   Object 0xffff880417179300:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755848]   Object 0xffff880417179310:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755862]   Object 0xffff880417179320:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755876]   Object 0xffff880417179330:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755890]   Object 0xffff880417179340:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755904]   Object 0xffff880417179350:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755919]   Object 0xffff880417179360:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755933]   Object 0xffff880417179370:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755947]   Object 0xffff880417179380:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755961]   Object 0xffff880417179390:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755975]   Object 0xffff8804171793a0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755989]   Object 0xffff8804171793b0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.756004]   Object 0xffff8804171793c0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.756018]   Object 0xffff8804171793d0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.756032]   Object 0xffff8804171793e0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.756046]   Object 0xffff8804171793f0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.756060]   Object 0xffff880417179400:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.756074]   Object 0xffff880417179410:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.756089]   Object 0xffff880417179420:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.756103]   Object 0xffff880417179430:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.756117]   Object 0xffff880417179440:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.756131]   Object 0xffff880417179450:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.756145]   Object 0xffff880417179460:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.756160]   Object 0xffff880417179470:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.756174]   Object 0xffff880417179480:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.756188]   Object 0xffff880417179490:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.756202]   Object 0xffff8804171794a0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.756216]   Object 0xffff8804171794b0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.756230]   Object 0xffff8804171794c0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.756245]   Object 0xffff8804171794d0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.756259]   Object 0xffff8804171794e0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.756273]   Object 0xffff8804171794f0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.756287]   Object 0xffff880417179500:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.756301]   Object 0xffff880417179510:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b a5 kkkkkkkkkkkkkkk¥
[  413.756316]  Redzone 0xffff880417179520:  bb bb bb bb bb bb bb bb 
                      »»»»»»»»
[  413.756329]  Padding 0xffff880417179560:  5a 5a 5a 5a 5a 5a 00 00 
                      ZZZZZZ..
[  413.756345] Pid: 5247, comm: qemu Not tainted 2.6.39 #2
[  413.756349] Call Trace:
[  413.756353]  <IRQ>  [<ffffffff810b7ccd>] ? 
check_bytes_and_report+0x10d/0x150
[  413.756372]  [<ffffffff81318f1c>] ? pskb_expand_head+0x15c/0x250
[  413.756379]  [<ffffffff810b7db9>] ? check_object+0xa9/0x260
[  413.756387]  [<ffffffff81318f1c>] ? pskb_expand_head+0x15c/0x250
[  413.756393]  [<ffffffff810b86f4>] ? alloc_debug_processing+0x104/0x190
[  413.756400]  [<ffffffff810b9ac2>] ? T.912+0x272/0x2d0
[  413.756409]  [<ffffffff810ba59d>] ? __kmalloc+0x10d/0x160
[  413.756416]  [<ffffffff81318f1c>] ? pskb_expand_head+0x15c/0x250
[  413.756424]  [<ffffffff813a6c45>] ? nf_bridge_copy_header+0x145/0x160
[  413.756431]  [<ffffffff8139f78d>] ? br_dev_queue_push_xmit+0x6d/0x80
[  413.756439]  [<ffffffff813a55a0>] ? br_nf_post_routing+0x2a0/0x2f0
[  413.756447]  [<ffffffff81346bc4>] ? nf_iterate+0x84/0xb0
[  413.756453]  [<ffffffff8139f720>] ? br_flood_deliver+0x20/0x20
[  413.756459]  [<ffffffff81346c64>] ? nf_hook_slow+0x74/0x120
[  413.756465]  [<ffffffff8139f720>] ? br_flood_deliver+0x20/0x20
[  413.756472]  [<ffffffff8139f7da>] ? br_forward_finish+0x3a/0x60
[  413.756479]  [<ffffffff813a5758>] ? br_nf_forward_finish+0x168/0x170
[  413.756487]  [<ffffffff813a5c90>] ? br_nf_forward_ip+0x360/0x3a0
[  413.756492]  [<ffffffff81346bc4>] ? nf_iterate+0x84/0xb0
[  413.756498]  [<ffffffff8139f7a0>] ? br_dev_queue_push_xmit+0x80/0x80
[  413.756504]  [<ffffffff81346c64>] ? nf_hook_slow+0x74/0x120
[  413.756510]  [<ffffffff8139f7a0>] ? br_dev_queue_push_xmit+0x80/0x80
[  413.756516]  [<ffffffff8139f800>] ? br_forward_finish+0x60/0x60
[  413.756522]  [<ffffffff8139f800>] ? br_forward_finish+0x60/0x60
[  413.756528]  [<ffffffff8139f875>] ? __br_forward+0x75/0xc0
[  413.756534]  [<ffffffff8139f426>] ? deliver_clone+0x36/0x60
[  413.756540]  [<ffffffff8139f69d>] ? br_flood+0xbd/0x100
[  413.756546]  [<ffffffff813a05b0>] ? br_handle_local_finish+0x40/0x40
[  413.756552]  [<ffffffff813a080e>] ? br_handle_frame_finish+0x25e/0x280
[  413.756560]  [<ffffffff813a60f0>] ? br_nf_pre_routing_finish+0x1a0/0x330
[  413.756568]  [<ffffffff813a6958>] ? br_nf_pre_routing+0x6d8/0x800
[  413.756577]  [<ffffffff8102d46a>] ? enqueue_task+0x3a/0x90
[  413.756582]  [<ffffffff81346bc4>] ? nf_iterate+0x84/0xb0
[  413.756589]  [<ffffffff813a05b0>] ? br_handle_local_finish+0x40/0x40
[  413.756594]  [<ffffffff81346c64>] ? nf_hook_slow+0x74/0x120
[  413.756600]  [<ffffffff813a05b0>] ? br_handle_local_finish+0x40/0x40
[  413.756607]  [<ffffffff810339b0>] ? try_to_wake_up+0x2c0/0x2c0
[  413.756613]  [<ffffffff813a09d9>] ? br_handle_frame+0x1a9/0x280
[  413.756620]  [<ffffffff813a0830>] ? br_handle_frame_finish+0x280/0x280
[  413.756627]  [<ffffffff81320ef7>] ? __netif_receive_skb+0x157/0x5c0
[  413.756634]  [<ffffffff81321443>] ? process_backlog+0xe3/0x1d0
[  413.756641]  [<ffffffff81321da5>] ? net_rx_action+0xc5/0x1d0
[  413.756650]  [<ffffffff8103df11>] ? __do_softirq+0x91/0x120
[  413.756657]  [<ffffffff813d838c>] ? call_softirq+0x1c/0x30
[  413.756660]  <EOI>  [<ffffffff81003cbd>] ? do_softirq+0x4d/0x80
[  413.756673]  [<ffffffff81321ece>] ? netif_rx_ni+0x1e/0x30
[  413.756681]  [<ffffffff812b3ae2>] ? tun_chr_aio_write+0x332/0x4e0
[  413.756688]  [<ffffffff812b37b0>] ? tun_sendmsg+0x4d0/0x4d0
[  413.756697]  [<ffffffff810c24e9>] ? do_sync_readv_writev+0xa9/0xf0
[  413.756704]  [<ffffffff81063f9c>] ? do_futex+0x13c/0xa70
[  413.756711]  [<ffffffff811d6730>] ? timerqueue_add+0x60/0xb0
[  413.756719]  [<ffffffff81056ab7>] ? __hrtimer_start_range_ns+0x1e7/0x410
[  413.756726]  [<ffffffff810c231b>] ? rw_copy_check_uvector+0x7b/0x140
[  413.756734]  [<ffffffff810c2bcf>] ? do_readv_writev+0xdf/0x210
[  413.756742]  [<ffffffff810c2e7e>] ? sys_writev+0x4e/0xc0
[  413.756750]  [<ffffffff813d753b>] ? system_call_fastpath+0x16/0x1b
[  413.756756] FIX kmalloc-1024: Restoring 
0xffff880417179566-0xffff880417179567=0x5a
[  413.756760]
[  556.640033] 
=============================================================================
[  556.640041] BUG kmalloc-512: Object padding overwritten
[  556.640045] 
-----------------------------------------------------------------------------
[  556.640048]
[  556.640053] INFO: 0xffff880403bf091e-0xffff880403bf091f. First byte 
0x0 instead of 0x5a
[  556.640069] INFO: Allocated in sock_alloc_send_pskb+0x1d0/0x320 
age=21401 cpu=2 pid=5630
[  556.640080] INFO: Freed in __kfree_skb+0x11/0x90 age=21386 cpu=2 pid=3753
[  556.640086] INFO: Slab 0xffffea000e0d1c80 objects=28 used=10 
fp=0xffff880403bf06d8 flags=0x80000000000040c1
[  556.640092] INFO: Object 0xffff880403bf06d8 @offset=1752 
fp=0xffff880403bf1488
[  556.640096]
[  556.640099] Bytes b4 0xffff880403bf06c8:  90 c4 ff ff 00 00 00 00 5a 
5a 5a 5a 5a 5a 5a 5a .Äÿÿ....ZZZZZZZZ
[  556.640115]   Object 0xffff880403bf06d8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  556.640130]   Object 0xffff880403bf06e8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  556.640144]   Object 0xffff880403bf06f8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  556.640158]   Object 0xffff880403bf0708:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  556.640172]   Object 0xffff880403bf0718:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  556.640187]   Object 0xffff880403bf0728:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  556.640201]   Object 0xffff880403bf0738:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  556.640215]   Object 0xffff880403bf0748:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  556.640229]   Object 0xffff880403bf0758:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  556.640243]   Object 0xffff880403bf0768:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  556.640257]   Object 0xffff880403bf0778:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  556.640271]   Object 0xffff880403bf0788:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  556.640286]   Object 0xffff880403bf0798:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  556.640300]   Object 0xffff880403bf07a8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  556.640314]   Object 0xffff880403bf07b8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  556.640328]   Object 0xffff880403bf07c8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  556.640343]   Object 0xffff880403bf07d8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  556.640357]   Object 0xffff880403bf07e8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  556.640371]   Object 0xffff880403bf07f8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  556.640385]   Object 0xffff880403bf0808:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  556.640399]   Object 0xffff880403bf0818:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  556.640413]   Object 0xffff880403bf0828:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  556.640428]   Object 0xffff880403bf0838:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  556.640442]   Object 0xffff880403bf0848:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  556.640456]   Object 0xffff880403bf0858:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  556.640471]   Object 0xffff880403bf0868:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  556.640485]   Object 0xffff880403bf0878:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  556.640499]   Object 0xffff880403bf0888:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  556.640513]   Object 0xffff880403bf0898:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  556.640527]   Object 0xffff880403bf08a8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  556.640542]   Object 0xffff880403bf08b8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  556.640556]   Object 0xffff880403bf08c8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b a5 kkkkkkkkkkkkkkk¥
[  556.640570]  Redzone 0xffff880403bf08d8:  bb bb bb bb bb bb bb bb 
                      »»»»»»»»
[  556.640583]  Padding 0xffff880403bf0918:  5a 5a 5a 5a 5a 5a 00 00 
                      ZZZZZZ..
[  556.640599] Pid: 4809, comm: qemu Not tainted 2.6.39 #2
[  556.640603] Call Trace:
[  556.640607]  <IRQ>  [<ffffffff810b7ccd>] ? 
check_bytes_and_report+0x10d/0x150
[  556.640626]  [<ffffffff8131aae7>] ? __netdev_alloc_skb+0x17/0x40
[  556.640632]  [<ffffffff810b7db9>] ? check_object+0xa9/0x260
[  556.640640]  [<ffffffff8131aae7>] ? __netdev_alloc_skb+0x17/0x40
[  556.640647]  [<ffffffff810b86f4>] ? alloc_debug_processing+0x104/0x190
[  556.640654]  [<ffffffff810b9ac2>] ? T.912+0x272/0x2d0
[  556.640661]  [<ffffffff810bb70d>] ? __kmalloc_track_caller+0x10d/0x160
[  556.640668]  [<ffffffff813198c2>] ? __alloc_skb+0x72/0x160
[  556.640676]  [<ffffffff8131aae7>] ? __netdev_alloc_skb+0x17/0x40
[  556.640684]  [<ffffffff813a7b93>] ? 
br_ip4_multicast_alloc_query+0x23/0x1c0
[  556.640690]  [<ffffffff813a7e76>] ? br_multicast_send_query+0x76/0x130
[  556.640698]  [<ffffffff8104dba8>] ? wq_worker_waking_up+0x8/0x30
[  556.640706]  [<ffffffff8102ca81>] ? sched_slice+0x51/0x80
[  556.640711]  [<ffffffff813a7fb5>] ? 
br_multicast_port_query_expired+0x45/0x70
[  556.640719]  [<ffffffff81043f77>] ? run_timer_softirq+0x137/0x270
[  556.640725]  [<ffffffff81033fe9>] ? scheduler_tick+0x289/0x2d0
[  556.640731]  [<ffffffff813a7f70>] ? br_multicast_query_expired+0x40/0x40
[  556.640740]  [<ffffffff8103df11>] ? __do_softirq+0x91/0x120
[  556.640747]  [<ffffffff813d838c>] ? call_softirq+0x1c/0x30
[  556.640755]  [<ffffffff81003cbd>] ? do_softirq+0x4d/0x80
[  556.640762]  [<ffffffff8103ddbe>] ? irq_exit+0x8e/0xb0
[  556.640770]  [<ffffffff81019d1a>] ? smp_apic_timer_interrupt+0x6a/0xa0
[  556.640776]  [<ffffffff813d7e53>] ? apic_timer_interrupt+0x13/0x20
[  556.640780]  <EOI>  [<ffffffff8102965a>] ? 
flush_tlb_others_ipi+0x11a/0x130
[  556.640791]  [<ffffffff81029647>] ? flush_tlb_others_ipi+0x107/0x130
[  556.640799]  [<ffffffff810aadab>] ? ptep_clear_flush+0xb/0x10
[  556.640806]  [<ffffffff8109a40b>] ? do_wp_page+0x34b/0x7a0
[  556.640836]  [<ffffffffa00a188a>] ? kvm_read_guest_page+0x5a/0x70 [kvm]
[  556.640863]  [<ffffffffa00b7db4>] ? 
paging64_walk_addr_generic+0x264/0x4c0 [kvm]
[  556.640871]  [<ffffffff8109acad>] ? handle_pte_fault+0x44d/0x990
[  556.640878]  [<ffffffff8109b4e8>] ? follow_page+0x268/0x440
[  556.640886]  [<ffffffff8109bd4d>] ? __get_user_pages+0x12d/0x530
[  556.640892]  [<ffffffff81028dbb>] ? gup_pud_range+0x12b/0x1b0
[  556.640914]  [<ffffffffa00a2d2e>] ? get_user_page_nowait+0x2e/0x40 [kvm]
[  556.640935]  [<ffffffffa00a2e7b>] ? hva_to_pfn+0x13b/0x440 [kvm]
[  556.640956]  [<ffffffffa00a31e9>] ? __gfn_to_pfn+0x39/0xf0 [kvm]
[  556.640979]  [<ffffffffa00b825d>] ? try_async_pf+0x4d/0x190 [kvm]
[  556.641002]  [<ffffffffa00b92be>] ? tdp_page_fault+0x10e/0x200 [kvm]
[  556.641026]  [<ffffffffa00b968c>] ? kvm_mmu_page_fault+0x1c/0x80 [kvm]
[  556.641050]  [<ffffffffa00b0a7a>] ? 
kvm_arch_vcpu_ioctl_run+0x3fa/0xcf0 [kvm]
[  556.641058]  [<ffffffff8106239f>] ? futex_wake+0x10f/0x120
[  556.641065]  [<ffffffff81063f7b>] ? do_futex+0x11b/0xa70
[  556.641086]  [<ffffffffa00a517f>] ? kvm_vcpu_ioctl+0x4df/0x5e0 [kvm]
[  556.641097]  [<ffffffff81046671>] ? __dequeue_signal+0xe1/0x170
[  556.641104]  [<ffffffff8104821c>] ? do_send_sig_info+0x6c/0x90
[  556.641112]  [<ffffffff81046aac>] ? dequeue_signal+0x3c/0x170
[  556.641119]  [<ffffffff8104551f>] ? copy_siginfo_to_user+0xef/0x1d0
[  556.641125]  [<ffffffff810d26db>] ? do_vfs_ioctl+0x9b/0x4f0
[  556.641132]  [<ffffffff8106494a>] ? sys_futex+0x7a/0x180
[  556.641137]  [<ffffffff810d2b79>] ? sys_ioctl+0x49/0x80
[  556.641145]  [<ffffffff813d753b>] ? system_call_fastpath+0x16/0x1b
[  556.641152] FIX kmalloc-512: Restoring 
0xffff880403bf091e-0xffff880403bf091f=0x5a
[  556.641155]
[  602.610062] INFO: task ksmd:552 blocked for more than 120 seconds.
[  602.610068] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" 
disables this message.
[  602.610073] ksmd            D 0000000000000000     0   552      2 
0x00000000
[  602.610083]  ffff88041db68ca0 0000000000000046 000212d01fc11d00 
ffff880400000000
[  602.610090]  ffffffff81593020 ffff88041c1a9fd8 0000000000004000 
ffff88041c1a8010
[  602.610097]  ffff88041c1a9fd8 ffff88041db68ca0 0000000000000000 
0000000200000002
[  602.610104] Call Trace:
[  602.610121]  [<ffffffff813d5595>] ? schedule_timeout+0x1c5/0x230
[  602.610132]  [<ffffffff8102f69c>] ? enqueue_task_fair+0x14c/0x190
[  602.610141]  [<ffffffff8102db27>] ? task_rq_lock+0x47/0x90
[  602.610148]  [<ffffffff813d4562>] ? wait_for_common+0xd2/0x180
[  602.610154]  [<ffffffff810339b0>] ? try_to_wake_up+0x2c0/0x2c0
[  602.610162]  [<ffffffff8104cb94>] ? flush_work+0x24/0x30
[  602.610167]  [<ffffffff8104be10>] ? do_work_for_cpu+0x20/0x20
[  602.610174]  [<ffffffff8104e0ab>] ? schedule_on_each_cpu+0xab/0xe0
[  602.610181]  [<ffffffff810b5c05>] ? ksm_scan_thread+0x7f5/0xc20
[  602.610189]  [<ffffffff81052a20>] ? wake_up_bit+0x40/0x40
[  602.610194]  [<ffffffff810b5410>] ? 
try_to_merge_with_ksm_page+0x570/0x570
[  602.610200]  [<ffffffff810b5410>] ? 
try_to_merge_with_ksm_page+0x570/0x570
[  602.610207]  [<ffffffff810525b6>] ? kthread+0x96/0xa0
[  602.610214]  [<ffffffff813d8294>] ? kernel_thread_helper+0x4/0x10
[  602.610221]  [<ffffffff81052520>] ? kthread_worker_fn+0x120/0x120
[  602.610227]  [<ffffffff813d8290>] ? gs_change+0xb/0xb
[  602.610233] INFO: task fsnotify_mark:662 blocked for more than 120 
seconds.
[  602.610237] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" 
disables this message.
[  602.610241] fsnotify_mark   D 0000000000000000     0   662      2 
0x00000000
[  602.610248]  ffff88041c31d860 0000000000000046 0000000000000000 
0000000000000000
[  602.610255]  ffffffff81593020 ffff88041c365fd8 0000000000004000 
ffff88041c364010
[  602.610262]  ffff88041c365fd8 ffff88041c31d860 0000000000000000 
0000000000000000
[  602.610268] Call Trace:
[  602.610275]  [<ffffffff81035521>] ? load_balance+0x91/0x5e0
[  602.610282]  [<ffffffff813d5595>] ? schedule_timeout+0x1c5/0x230
[  602.610289]  [<ffffffff8102f323>] ? pick_next_task_fair+0x103/0x190
[  602.610295]  [<ffffffff813d4a6d>] ? schedule+0x28d/0x910
[  602.610302]  [<ffffffff813d4562>] ? wait_for_common+0xd2/0x180
[  602.610307]  [<ffffffff810339b0>] ? try_to_wake_up+0x2c0/0x2c0
[  602.610314]  [<ffffffff81074420>] ? synchronize_rcu_bh+0x50/0x50
[  602.610320]  [<ffffffff8107446a>] ? synchronize_sched+0x4a/0x50
[  602.610326]  [<ffffffff8104f940>] ? find_ge_pid+0x40/0x40
[  602.610333]  [<ffffffff8105742b>] ? __synchronize_srcu+0x5b/0xc0
[  602.610342]  [<ffffffff810f5f63>] ? fsnotify_mark_destroy+0x83/0x150
[  602.610348]  [<ffffffff81052a20>] ? wake_up_bit+0x40/0x40
[  602.610356]  [<ffffffff810f5ee0>] ? 
fsnotify_set_mark_ignored_mask_locked+0x20/0x20
[  602.610364]  [<ffffffff810f5ee0>] ? 
fsnotify_set_mark_ignored_mask_locked+0x20/0x20
[  602.610371]  [<ffffffff810525b6>] ? kthread+0x96/0xa0
[  602.610377]  [<ffffffff813d8294>] ? kernel_thread_helper+0x4/0x10
[  602.610385]  [<ffffffff81052520>] ? kthread_worker_fn+0x120/0x120
[  602.610390]  [<ffffffff813d8290>] ? gs_change+0xb/0xb
[  644.689408] 
=============================================================================
[  644.689416] BUG kmalloc-512: Object padding overwritten
[  644.689420] 
-----------------------------------------------------------------------------
[  644.689423]
[  644.689428] INFO: 0xffff88041411f476-0xffff88041411f477. First byte 
0x0 instead of 0x5a
[  644.689444] INFO: Allocated in load_elf_binary+0xa6c/0x1c00 age=18648 
cpu=3 pid=5915
[  644.689454] INFO: Freed in load_elf_binary+0xab8/0x1c00 age=18648 
cpu=3 pid=5915
[  644.689461] INFO: Slab 0xffffea000e463e20 objects=28 used=10 
fp=0xffff88041411f230 flags=0x80000000000040c1
[  644.689467] INFO: Object 0xffff88041411f230 @offset=12848 
fp=0xffff88041411eda0
[  644.689470]
[  644.689474] Bytes b4 0xffff88041411f220:  22 6f 00 00 01 00 00 00 5a 
5a 5a 5a 5a 5a 5a 5a "o......ZZZZZZZZ
[  644.689490]   Object 0xffff88041411f230:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  644.689505]   Object 0xffff88041411f240:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  644.689519]   Object 0xffff88041411f250:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  644.689533]   Object 0xffff88041411f260:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  644.689547]   Object 0xffff88041411f270:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  644.689562]   Object 0xffff88041411f280:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  644.689576]   Object 0xffff88041411f290:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  644.689590]   Object 0xffff88041411f2a0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  644.689604]   Object 0xffff88041411f2b0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  644.689618]   Object 0xffff88041411f2c0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  644.689632]   Object 0xffff88041411f2d0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  644.689647]   Object 0xffff88041411f2e0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  644.689661]   Object 0xffff88041411f2f0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  644.689675]   Object 0xffff88041411f300:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  644.689689]   Object 0xffff88041411f310:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  644.689703]   Object 0xffff88041411f320:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  644.689717]   Object 0xffff88041411f330:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  644.689732]   Object 0xffff88041411f340:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  644.689746]   Object 0xffff88041411f350:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  644.689760]   Object 0xffff88041411f360:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  644.689774]   Object 0xffff88041411f370:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  644.689788]   Object 0xffff88041411f380:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  644.689802]   Object 0xffff88041411f390:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  644.689817]   Object 0xffff88041411f3a0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  644.689831]   Object 0xffff88041411f3b0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  644.689845]   Object 0xffff88041411f3c0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  644.689859]   Object 0xffff88041411f3d0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  644.689873]   Object 0xffff88041411f3e0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  644.689888]   Object 0xffff88041411f3f0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  644.689902]   Object 0xffff88041411f400:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  644.689916]   Object 0xffff88041411f410:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  644.689930]   Object 0xffff88041411f420:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b a5 kkkkkkkkkkkkkkk¥
[  644.689945]  Redzone 0xffff88041411f430:  bb bb bb bb bb bb bb bb 
                      »»»»»»»»
[  644.689957]  Padding 0xffff88041411f470:  5a 5a 5a 5a 5a 5a 00 00 
                      ZZZZZZ..
[  644.689974] Pid: 6597, comm: awk Not tainted 2.6.39 #2
[  644.689978] Call Trace:
[  644.689989]  [<ffffffff810b7ccd>] ? check_bytes_and_report+0x10d/0x150
[  644.689998]  [<ffffffff8110773c>] ? load_elf_binary+0xa6c/0x1c00
[  644.690005]  [<ffffffff810b7db9>] ? check_object+0xa9/0x260
[  644.690012]  [<ffffffff8110773c>] ? load_elf_binary+0xa6c/0x1c00
[  644.690019]  [<ffffffff810b86f4>] ? alloc_debug_processing+0x104/0x190
[  644.690026]  [<ffffffff810b9ac2>] ? T.912+0x272/0x2d0
[  644.690032]  [<ffffffff810ba59d>] ? __kmalloc+0x10d/0x160
[  644.690039]  [<ffffffff8110773c>] ? load_elf_binary+0xa6c/0x1c00
[  644.690047]  [<ffffffff8109bd9d>] ? __get_user_pages+0x17d/0x530
[  644.690055]  [<ffffffff810c9556>] ? get_arg_page+0x56/0x100
[  644.690062]  [<ffffffff810c8070>] ? search_binary_handler+0x90/0x240
[  644.690069]  [<ffffffff810c9e9f>] ? do_execve+0x22f/0x2f0
[  644.690076]  [<ffffffff810094a6>] ? sys_execve+0x36/0x60
[  644.690085]  [<ffffffff813d78dc>] ? stub_execve+0x6c/0xc0
[  644.690092] FIX kmalloc-512: Restoring 
0xffff88041411f476-0xffff88041411f477=0x5a
[  644.690095]
[  704.979481] 
=============================================================================
[  704.979488] BUG kmalloc-512: Object padding overwritten
[  704.979492] 
-----------------------------------------------------------------------------
[  704.979496]
[  704.979501] INFO: 0xffff8804168c1fee-0xffff8804168c1fef. First byte 
0x0 instead of 0x5a
[  704.979517] INFO: Allocated in load_elf_binary+0xa6c/0x1c00 age=24184 
cpu=0 pid=5999
[  704.979527] INFO: Freed in load_elf_binary+0xab8/0x1c00 age=24184 
cpu=0 pid=5999
[  704.979534] INFO: Slab 0xffffea000e4eea00 objects=28 used=10 
fp=0xffff8804168c1da8 flags=0x80000000000040c1
[  704.979540] INFO: Object 0xffff8804168c1da8 @offset=7592 
fp=0xffff8804168c2910
[  704.979543]
[  704.979547] Bytes b4 0xffff8804168c1d98:  93 98 ff ff 00 00 00 00 5a 
5a 5a 5a 5a 5a 5a 5a ..ÿÿ....ZZZZZZZZ
[  704.979563]   Object 0xffff8804168c1da8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  704.979578]   Object 0xffff8804168c1db8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  704.979592]   Object 0xffff8804168c1dc8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  704.979606]   Object 0xffff8804168c1dd8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  704.979620]   Object 0xffff8804168c1de8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  704.979634]   Object 0xffff8804168c1df8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  704.979648]   Object 0xffff8804168c1e08:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  704.979662]   Object 0xffff8804168c1e18:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  704.979676]   Object 0xffff8804168c1e28:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  704.979690]   Object 0xffff8804168c1e38:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  704.979704]   Object 0xffff8804168c1e48:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  704.979719]   Object 0xffff8804168c1e58:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  704.979733]   Object 0xffff8804168c1e68:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  704.979747]   Object 0xffff8804168c1e78:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  704.979761]   Object 0xffff8804168c1e88:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  704.979775]   Object 0xffff8804168c1e98:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  704.979789]   Object 0xffff8804168c1ea8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  704.979803]   Object 0xffff8804168c1eb8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  704.979817]   Object 0xffff8804168c1ec8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  704.979831]   Object 0xffff8804168c1ed8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  704.979845]   Object 0xffff8804168c1ee8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  704.979859]   Object 0xffff8804168c1ef8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  704.979873]   Object 0xffff8804168c1f08:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  704.979888]   Object 0xffff8804168c1f18:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  704.979902]   Object 0xffff8804168c1f28:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  704.979916]   Object 0xffff8804168c1f38:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  704.979930]   Object 0xffff8804168c1f48:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  704.979944]   Object 0xffff8804168c1f58:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  704.979958]   Object 0xffff8804168c1f68:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  704.979972]   Object 0xffff8804168c1f78:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  704.979986]   Object 0xffff8804168c1f88:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  704.980000]   Object 0xffff8804168c1f98:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b a5 kkkkkkkkkkkkkkk¥
[  704.980015]  Redzone 0xffff8804168c1fa8:  bb bb bb bb bb bb bb bb 
                      »»»»»»»»
[  704.980028]  Padding 0xffff8804168c1fe8:  5a 5a 5a 5a 5a 5a 00 00 
                      ZZZZZZ..
[  704.980044] Pid: 6812, comm: get-rrdtool-dat Not tainted 2.6.39 #2
[  704.980048] Call Trace:
[  704.980060]  [<ffffffff810b7ccd>] ? check_bytes_and_report+0x10d/0x150
[  704.980069]  [<ffffffff81106e49>] ? load_elf_binary+0x179/0x1c00
[  704.980075]  [<ffffffff810b7db9>] ? check_object+0xa9/0x260
[  704.980083]  [<ffffffff81106e49>] ? load_elf_binary+0x179/0x1c00
[  704.980090]  [<ffffffff810b86f4>] ? alloc_debug_processing+0x104/0x190
[  704.980096]  [<ffffffff810b9ac2>] ? T.912+0x272/0x2d0
[  704.980102]  [<ffffffff810ba59d>] ? __kmalloc+0x10d/0x160
[  704.980109]  [<ffffffff81106e49>] ? load_elf_binary+0x179/0x1c00
[  704.980117]  [<ffffffff8109b782>] ? __pte_alloc+0x42/0x130
[  704.980123]  [<ffffffff8109b5a5>] ? follow_page+0x325/0x440
[  704.980130]  [<ffffffff8109b5a5>] ? follow_page+0x325/0x440
[  704.980137]  [<ffffffff8109bd9d>] ? __get_user_pages+0x17d/0x530
[  704.980145]  [<ffffffff810c9556>] ? get_arg_page+0x56/0x100
[  704.980152]  [<ffffffff810c8070>] ? search_binary_handler+0x90/0x240
[  704.980158]  [<ffffffff810c9e9f>] ? do_execve+0x22f/0x2f0
[  704.980166]  [<ffffffff810094a6>] ? sys_execve+0x36/0x60
[  704.980175]  [<ffffffff813d78dc>] ? stub_execve+0x6c/0xc0
[  704.980182] FIX kmalloc-512: Restoring 
0xffff8804168c1fee-0xffff8804168c1fef=0x5a
[  704.980185]
[  722.610058] INFO: task ksmd:552 blocked for more than 120 seconds.
[  722.610063] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" 
disables this message.
[  722.610069] ksmd            D 0000000000000000     0   552      2 
0x00000000
[  722.610078]  ffff88041db68ca0 0000000000000046 000212d01fc11d00 
ffff880400000000
[  722.610085]  ffffffff81593020 ffff88041c1a9fd8 0000000000004000 
ffff88041c1a8010
[  722.610092]  ffff88041c1a9fd8 ffff88041db68ca0 0000000000000000 
0000000200000002
[  722.610099] Call Trace:
[  722.610115]  [<ffffffff813d5595>] ? schedule_timeout+0x1c5/0x230
[  722.610127]  [<ffffffff8102f69c>] ? enqueue_task_fair+0x14c/0x190
[  722.610135]  [<ffffffff8102db27>] ? task_rq_lock+0x47/0x90
[  722.610142]  [<ffffffff813d4562>] ? wait_for_common+0xd2/0x180
[  722.610148]  [<ffffffff810339b0>] ? try_to_wake_up+0x2c0/0x2c0
[  722.610156]  [<ffffffff8104cb94>] ? flush_work+0x24/0x30
[  722.610162]  [<ffffffff8104be10>] ? do_work_for_cpu+0x20/0x20
[  722.610168]  [<ffffffff8104e0ab>] ? schedule_on_each_cpu+0xab/0xe0
[  722.610175]  [<ffffffff810b5c05>] ? ksm_scan_thread+0x7f5/0xc20
[  722.610182]  [<ffffffff81052a20>] ? wake_up_bit+0x40/0x40
[  722.610188]  [<ffffffff810b5410>] ? 
try_to_merge_with_ksm_page+0x570/0x570
[  722.610194]  [<ffffffff810b5410>] ? 
try_to_merge_with_ksm_page+0x570/0x570
[  722.610201]  [<ffffffff810525b6>] ? kthread+0x96/0xa0
[  722.610208]  [<ffffffff813d8294>] ? kernel_thread_helper+0x4/0x10
[  722.610215]  [<ffffffff81052520>] ? kthread_worker_fn+0x120/0x120
[  722.610221]  [<ffffffff813d8290>] ? gs_change+0xb/0xb
[  722.610226] INFO: task fsnotify_mark:662 blocked for more than 120 
seconds.
[  722.610230] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" 
disables this message.
[  722.610235] fsnotify_mark   D 0000000000000000     0   662      2 
0x00000000
[  722.610241]  ffff88041c31d860 0000000000000046 0000000000000000 
0000000000000000
[  722.610248]  ffffffff81593020 ffff88041c365fd8 0000000000004000 
ffff88041c364010
[  722.610255]  ffff88041c365fd8 ffff88041c31d860 0000000000000000 
0000000000000000
[  722.610261] Call Trace:
[  722.610268]  [<ffffffff81035521>] ? load_balance+0x91/0x5e0
[  722.610275]  [<ffffffff813d5595>] ? schedule_timeout+0x1c5/0x230
[  722.610282]  [<ffffffff8102f323>] ? pick_next_task_fair+0x103/0x190
[  722.610289]  [<ffffffff813d4a6d>] ? schedule+0x28d/0x910
[  722.610295]  [<ffffffff813d4562>] ? wait_for_common+0xd2/0x180
[  722.610301]  [<ffffffff810339b0>] ? try_to_wake_up+0x2c0/0x2c0
[  722.610308]  [<ffffffff81074420>] ? synchronize_rcu_bh+0x50/0x50
[  722.610313]  [<ffffffff8107446a>] ? synchronize_sched+0x4a/0x50
[  722.610319]  [<ffffffff8104f940>] ? find_ge_pid+0x40/0x40
[  722.610326]  [<ffffffff8105742b>] ? __synchronize_srcu+0x5b/0xc0
[  722.610335]  [<ffffffff810f5f63>] ? fsnotify_mark_destroy+0x83/0x150
[  722.610342]  [<ffffffff81052a20>] ? wake_up_bit+0x40/0x40
[  722.610350]  [<ffffffff810f5ee0>] ? 
fsnotify_set_mark_ignored_mask_locked+0x20/0x20
[  722.610358]  [<ffffffff810f5ee0>] ? 
fsnotify_set_mark_ignored_mask_locked+0x20/0x20
[  722.610365]  [<ffffffff810525b6>] ? kthread+0x96/0xa0
[  722.610371]  [<ffffffff813d8294>] ? kernel_thread_helper+0x4/0x10
[  722.610378]  [<ffffffff81052520>] ? kthread_worker_fn+0x120/0x120
[  722.610384]  [<ffffffff813d8290>] ? gs_change+0xb/0xb
[  722.610392] INFO: task jbd2/md0-8:2528 blocked for more than 120 seconds.
[  722.610396] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" 
disables this message.
[  722.610400] jbd2/md0-8      D 0000000000000005     0  2528      2 
0x00000000
[  722.610407]  ffff88041d91c570 0000000000000046 ffff88041baeea28 
ffffea0000000000
[  722.610413]  ffff88041d91a5e0 ffff880419f9bfd8 0000000000004000 
ffff880419f9a010
[  722.610420]  ffff880419f9bfd8 ffff88041d91c570 ffff88041baeea28 
ffffffff810b8495
[  722.610426] Call Trace:
[  722.610432]  [<ffffffff810b8495>] ? init_object+0x85/0xa0
[  722.610438]  [<ffffffff810b8916>] ? free_debug_processing+0x196/0x250
[  722.610447]  [<ffffffff8105adae>] ? ktime_get_ts+0x6e/0xf0
[  722.610455]  [<ffffffff810810f0>] ? __lock_page+0x70/0x70
[  722.610461]  [<ffffffff813d5174>] ? io_schedule+0x84/0xd0
[  722.610469]  [<ffffffff811d4403>] ? 
radix_tree_gang_lookup_tag_slot+0x93/0xf0
[  722.610476]  [<ffffffff810810f9>] ? sleep_on_page+0x9/0x10
[  722.610482]  [<ffffffff813d57df>] ? __wait_on_bit+0x4f/0x80
[  722.610489]  [<ffffffff810812eb>] ? wait_on_page_bit+0x6b/0x80
[  722.610496]  [<ffffffff81052a50>] ? autoremove_wake_function+0x30/0x30
[  722.610504]  [<ffffffff8108a458>] ? pagevec_lookup_tag+0x18/0x20
[  722.610509]  [<ffffffff81081f2a>] ? filemap_fdatawait_range+0xfa/0x180
[  722.610518]  [<ffffffff811be09f>] ? submit_bio+0x6f/0xf0
[  722.610526]  [<ffffffff81176276>] ? 
jbd2_journal_commit_transaction+0x796/0x1270
[  722.610536]  [<ffffffff81179ed1>] ? kjournald2+0xb1/0x1e0
[  722.610542]  [<ffffffff81052a20>] ? wake_up_bit+0x40/0x40
[  722.610549]  [<ffffffff81179e20>] ? commit_timeout+0x10/0x10
[  722.610556]  [<ffffffff81179e20>] ? commit_timeout+0x10/0x10
[  722.610563]  [<ffffffff810525b6>] ? kthread+0x96/0xa0
[  722.610569]  [<ffffffff813d8294>] ? kernel_thread_helper+0x4/0x10
[  722.610576]  [<ffffffff81052520>] ? kthread_worker_fn+0x120/0x120
[  722.610582]  [<ffffffff813d8290>] ? gs_change+0xb/0xb
[  722.610591] INFO: task nfsd:4326 blocked for more than 120 seconds.
[  722.610594] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" 
disables this message.
[  722.610598] nfsd            D 0000000000000000     0  4326      2 
0x00000000
[  722.610605]  ffff88041b604bc0 0000000000000046 0000000000000016 
ffffffff00000000
[  722.610611]  ffff88041d8fa5e0 ffff88041c527fd8 0000000000004000 
ffff88041c526010
[  722.610617]  ffff88041c527fd8 ffff88041b604bc0 000000000000009c 
ffff880000000000
[  722.610623] Call Trace:
[  722.610630]  [<ffffffff8101e845>] ? amd_flush_garts+0x105/0x140
[  722.610637]  [<ffffffff8101fa80>] ? gart_map_sg+0x480/0x480
[  722.610643]  [<ffffffff8101f5d3>] ? flush_gart+0x23/0x50
[  722.610650]  [<ffffffff81080f98>] ? find_get_page+0x18/0x90
[  722.610657]  [<ffffffff81174b95>] ? do_get_write_access+0x265/0x4a0
[  722.610665]  [<ffffffff81052a50>] ? autoremove_wake_function+0x30/0x30
[  722.610672]  [<ffffffff81174ef9>] ? 
jbd2_journal_get_write_access+0x29/0x50
[  722.610680]  [<ffffffff8115f122>] ? 
__ext4_journal_get_write_access+0x32/0x80
[  722.610689]  [<ffffffff81143908>] ? ext4_reserve_inode_write+0x78/0xa0
[  722.610696]  [<ffffffff81143970>] ? ext4_mark_inode_dirty+0x40/0x1e0
[  722.610703]  [<ffffffff81156c0b>] ? ext4_journal_start_sb+0x6b/0x160
[  722.610711]  [<ffffffff81322645>] ? dev_hard_start_xmit+0x305/0x5f0
[  722.610719]  [<ffffffff81352fd0>] ? ip_finish_output2+0x290/0x290
[  722.610726]  [<ffffffff81143c65>] ? ext4_dirty_inode+0x35/0x70
[  722.610733]  [<ffffffff810e4a08>] ? __mark_inode_dirty+0x38/0x210
[  722.610741]  [<ffffffff810d9317>] ? file_update_time+0xf7/0x180
[  722.610747]  [<ffffffff81082458>] ? __generic_file_aio_write+0x1f8/0x430
[  722.610755]  [<ffffffff81373ef9>] ? udp_sendmsg+0x3c9/0x7e0
[  722.610763]  [<ffffffff81314260>] ? sock_alloc_send_pskb+0x1d0/0x320
[  722.610769]  [<ffffffff81082703>] ? generic_file_aio_write+0x73/0xf0
[  722.610776]  [<ffffffff8113f74e>] ? ext4_file_write+0x6e/0x2b0
[  722.610783]  [<ffffffff810da8ac>] ? iget_locked+0x4c/0x140
[  722.610789]  [<ffffffff8119a590>] ? fh_compose+0x4c0/0x4c0
[  722.610795]  [<ffffffff8113f6e0>] ? ext4_llseek+0x110/0x110
[  722.610803]  [<ffffffff810c24e9>] ? do_sync_readv_writev+0xa9/0xf0
[  722.610811]  [<ffffffff810c231b>] ? rw_copy_check_uvector+0x7b/0x140
[  722.610819]  [<ffffffff810c2bcf>] ? do_readv_writev+0xdf/0x210
[  722.610825]  [<ffffffff810b8735>] ? alloc_debug_processing+0x145/0x190
[  722.610831]  [<ffffffff810b9bc5>] ? kmem_cache_alloc+0xa5/0xb0
[  722.610838]  [<ffffffff8113f463>] ? ext4_file_open+0x63/0x180
[  722.610844]  [<ffffffff8119c21d>] ? nfsd_vfs_write+0xed/0x3a0
[  722.610851]  [<ffffffff810c1127>] ? __dentry_open+0x1f7/0x2b0
[  722.610857]  [<ffffffff8119c892>] ? nfsd_open+0xf2/0x1b0
[  722.610862]  [<ffffffff8119cd34>] ? nfsd_write+0xf4/0x110
[  722.610868]  [<ffffffff81199930>] ? nfsd_proc_write+0xb0/0x120
[  722.610876]  [<ffffffff811971c5>] ? nfsd_dispatch+0xf5/0x230
[  722.610882]  [<ffffffff813b456f>] ? svc_process+0x4af/0x820
[  722.610887]  [<ffffffff810339b0>] ? try_to_wake_up+0x2c0/0x2c0
[  722.610894]  [<ffffffff811977a0>] ? nfsd_svc+0x1b0/0x1b0
[  722.610901]  [<ffffffff8119784d>] ? nfsd+0xad/0x150
[  722.610907]  [<ffffffff810525b6>] ? kthread+0x96/0xa0
[  722.610913]  [<ffffffff813d8294>] ? kernel_thread_helper+0x4/0x10
[  722.610921]  [<ffffffff81052520>] ? kthread_worker_fn+0x120/0x120
[  722.610926]  [<ffffffff813d8290>] ? gs_change+0xb/0xb
[  765.241265] 
=============================================================================
[  765.241267] BUG kmalloc-512: Object padding overwritten
[  765.241268] 
-----------------------------------------------------------------------------
[  765.241269]
[  765.241271] INFO: 0xffff8804168c1486-0xffff8804168c1487. First byte 
0x0 instead of 0x5a
[  765.241277] INFO: Allocated in bio_kmalloc+0x2d/0x70 age=73010 cpu=5 
pid=2495
[  765.241281] INFO: Freed in r10buf_pool_free+0x71/0x90 age=72999 cpu=5 
pid=1728
[  765.241283] INFO: Slab 0xffffea000e4eea00 objects=28 used=13 
fp=0xffff8804168c1240 flags=0x80000000000040c1
[  765.241284] INFO: Object 0xffff8804168c1240 @offset=4672 
fp=0xffff8804168c0ff8
[  765.241285]
[  765.241286] Bytes b4 0xffff8804168c1230:  95 98 ff ff 00 00 00 00 5a 
5a 5a 5a 5a 5a 5a 5a ..ÿÿ....ZZZZZZZZ
[  765.241291]   Object 0xffff8804168c1240:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  765.241294]   Object 0xffff8804168c1250:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  765.241298]   Object 0xffff8804168c1260:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  765.241301]   Object 0xffff8804168c1270:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  765.241305]   Object 0xffff8804168c1280:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  765.241308]   Object 0xffff8804168c1290:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  765.241312]   Object 0xffff8804168c12a0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  765.241315]   Object 0xffff8804168c12b0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  765.241319]   Object 0xffff8804168c12c0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  765.241322]   Object 0xffff8804168c12d0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  765.241326]   Object 0xffff8804168c12e0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  765.241329]   Object 0xffff8804168c12f0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  765.241332]   Object 0xffff8804168c1300:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  765.241336]   Object 0xffff8804168c1310:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  765.241339]   Object 0xffff8804168c1320:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  765.241343]   Object 0xffff8804168c1330:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  765.241346]   Object 0xffff8804168c1340:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  765.241350]   Object 0xffff8804168c1350:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  765.241353]   Object 0xffff8804168c1360:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  765.241357]   Object 0xffff8804168c1370:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  765.241360]   Object 0xffff8804168c1380:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  765.241364]   Object 0xffff8804168c1390:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  765.241367]   Object 0xffff8804168c13a0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  765.241371]   Object 0xffff8804168c13b0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  765.241374]   Object 0xffff8804168c13c0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  765.241378]   Object 0xffff8804168c13d0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  765.241381]   Object 0xffff8804168c13e0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  765.241385]   Object 0xffff8804168c13f0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  765.241388]   Object 0xffff8804168c1400:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  765.241392]   Object 0xffff8804168c1410:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  765.241395]   Object 0xffff8804168c1420:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  765.241399]   Object 0xffff8804168c1430:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b a5 kkkkkkkkkkkkkkk¥
[  765.241402]  Redzone 0xffff8804168c1440:  bb bb bb bb bb bb bb bb 
                      »»»»»»»»
[  765.241406]  Padding 0xffff8804168c1480:  5a 5a 5a 5a 5a 5a 00 00 
                      ZZZZZZ..
[  765.241410] Pid: 7147, comm: apcaccess Not tainted 2.6.39 #2
[  765.241411] Call Trace:
[  765.241416]  [<ffffffff810b7ccd>] ? check_bytes_and_report+0x10d/0x150
[  765.241419]  [<ffffffff8110773c>] ? load_elf_binary+0xa6c/0x1c00
[  765.241421]  [<ffffffff810b7db9>] ? check_object+0xa9/0x260
[  765.241423]  [<ffffffff8110773c>] ? load_elf_binary+0xa6c/0x1c00
[  765.241424]  [<ffffffff810b86f4>] ? alloc_debug_processing+0x104/0x190
[  765.241426]  [<ffffffff810b9ac2>] ? T.912+0x272/0x2d0
[  765.241428]  [<ffffffff810ba59d>] ? __kmalloc+0x10d/0x160
[  765.241429]  [<ffffffff8110773c>] ? load_elf_binary+0xa6c/0x1c00
[  765.241432]  [<ffffffff8109bd9d>] ? __get_user_pages+0x17d/0x530
[  765.241434]  [<ffffffff810c9556>] ? get_arg_page+0x56/0x100
[  765.241437]  [<ffffffff811da4ad>] ? strnlen_user+0x2d/0x80
[  765.241439]  [<ffffffff810c8070>] ? search_binary_handler+0x90/0x240
[  765.241440]  [<ffffffff810c9e9f>] ? do_execve+0x22f/0x2f0
[  765.241443]  [<ffffffff810094a6>] ? sys_execve+0x36/0x60
[  765.241446]  [<ffffffff813d78dc>] ? stub_execve+0x6c/0xc0
[  765.241448] FIX kmalloc-512: Restoring 
0xffff8804168c1486-0xffff8804168c1487=0x5a
[  765.241449]
[  789.536436] 
=============================================================================
[  789.536444] BUG kmalloc-2048: Object padding overwritten
[  789.536448] 
-----------------------------------------------------------------------------
[  789.536451]
[  789.536456] INFO: 0xffff8802c8b1a11e-0xffff8802c8b1a11f. First byte 
0x0 instead of 0x5a
[  789.536471] INFO: Allocated in sk_stream_alloc_skb+0x3a/0x110 age=63 
cpu=3 pid=5169
[  789.536482] INFO: Freed in __kfree_skb+0x11/0x90 age=63 cpu=5 pid=0
[  789.536488] INFO: Slab 0xffffea0009be6d40 objects=15 used=4 
fp=0xffff8802c8b198d8 flags=0x80000000000040c1
[  789.536494] INFO: Object 0xffff8802c8b198d8 @offset=6360 
fp=0xffff8802c8b1a968
[  789.536497]
[  789.536501] Bytes b4 0xffff8802c8b198c8:  cc be 00 00 01 00 00 00 5a 
5a 5a 5a 5a 5a 5a 5a ÌŸ......ZZZZZZZZ
[  789.536517]   Object 0xffff8802c8b198d8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.536532]   Object 0xffff8802c8b198e8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.536546]   Object 0xffff8802c8b198f8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.536560]   Object 0xffff8802c8b19908:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.536575]   Object 0xffff8802c8b19918:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.536589]   Object 0xffff8802c8b19928:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.536603]   Object 0xffff8802c8b19938:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.536617]   Object 0xffff8802c8b19948:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.536631]   Object 0xffff8802c8b19958:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.536645]   Object 0xffff8802c8b19968:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.536660]   Object 0xffff8802c8b19978:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.536674]   Object 0xffff8802c8b19988:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.536688]   Object 0xffff8802c8b19998:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.536702]   Object 0xffff8802c8b199a8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.536716]   Object 0xffff8802c8b199b8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.536730]   Object 0xffff8802c8b199c8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.536745]   Object 0xffff8802c8b199d8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.536759]   Object 0xffff8802c8b199e8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.536773]   Object 0xffff8802c8b199f8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.536787]   Object 0xffff8802c8b19a08:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.536801]   Object 0xffff8802c8b19a18:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.536816]   Object 0xffff8802c8b19a28:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.536830]   Object 0xffff8802c8b19a38:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.536844]   Object 0xffff8802c8b19a48:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.536858]   Object 0xffff8802c8b19a58:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.536872]   Object 0xffff8802c8b19a68:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.536887]   Object 0xffff8802c8b19a78:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.536901]   Object 0xffff8802c8b19a88:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.536915]   Object 0xffff8802c8b19a98:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.536929]   Object 0xffff8802c8b19aa8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.536943]   Object 0xffff8802c8b19ab8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.536958]   Object 0xffff8802c8b19ac8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.536972]   Object 0xffff8802c8b19ad8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.536986]   Object 0xffff8802c8b19ae8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537000]   Object 0xffff8802c8b19af8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537014]   Object 0xffff8802c8b19b08:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537029]   Object 0xffff8802c8b19b18:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537043]   Object 0xffff8802c8b19b28:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537057]   Object 0xffff8802c8b19b38:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537071]   Object 0xffff8802c8b19b48:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537085]   Object 0xffff8802c8b19b58:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537100]   Object 0xffff8802c8b19b68:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537114]   Object 0xffff8802c8b19b78:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537128]   Object 0xffff8802c8b19b88:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537143]   Object 0xffff8802c8b19b98:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537158]   Object 0xffff8802c8b19ba8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537172]   Object 0xffff8802c8b19bb8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537187]   Object 0xffff8802c8b19bc8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537201]   Object 0xffff8802c8b19bd8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537215]   Object 0xffff8802c8b19be8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537229]   Object 0xffff8802c8b19bf8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537244]   Object 0xffff8802c8b19c08:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537258]   Object 0xffff8802c8b19c18:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537272]   Object 0xffff8802c8b19c28:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537286]   Object 0xffff8802c8b19c38:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537301]   Object 0xffff8802c8b19c48:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537315]   Object 0xffff8802c8b19c58:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537329]   Object 0xffff8802c8b19c68:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537343]   Object 0xffff8802c8b19c78:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537357]   Object 0xffff8802c8b19c88:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537372]   Object 0xffff8802c8b19c98:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537386]   Object 0xffff8802c8b19ca8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537400]   Object 0xffff8802c8b19cb8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537414]   Object 0xffff8802c8b19cc8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537428]   Object 0xffff8802c8b19cd8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537443]   Object 0xffff8802c8b19ce8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537457]   Object 0xffff8802c8b19cf8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537471]   Object 0xffff8802c8b19d08:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537485]   Object 0xffff8802c8b19d18:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537500]   Object 0xffff8802c8b19d28:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537514]   Object 0xffff8802c8b19d38:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537528]   Object 0xffff8802c8b19d48:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537542]   Object 0xffff8802c8b19d58:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537557]   Object 0xffff8802c8b19d68:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537571]   Object 0xffff8802c8b19d78:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537585]   Object 0xffff8802c8b19d88:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537599]   Object 0xffff8802c8b19d98:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537613]   Object 0xffff8802c8b19da8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537628]   Object 0xffff8802c8b19db8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537642]   Object 0xffff8802c8b19dc8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537656]   Object 0xffff8802c8b19dd8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537670]   Object 0xffff8802c8b19de8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537685]   Object 0xffff8802c8b19df8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537699]   Object 0xffff8802c8b19e08:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537713]   Object 0xffff8802c8b19e18:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537727]   Object 0xffff8802c8b19e28:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537742]   Object 0xffff8802c8b19e38:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537756]   Object 0xffff8802c8b19e48:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537770]   Object 0xffff8802c8b19e58:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537784]   Object 0xffff8802c8b19e68:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537799]   Object 0xffff8802c8b19e78:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537813]   Object 0xffff8802c8b19e88:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537827]   Object 0xffff8802c8b19e98:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537841]   Object 0xffff8802c8b19ea8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537856]   Object 0xffff8802c8b19eb8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537870]   Object 0xffff8802c8b19ec8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537884]   Object 0xffff8802c8b19ed8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537898]   Object 0xffff8802c8b19ee8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537912]   Object 0xffff8802c8b19ef8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537927]   Object 0xffff8802c8b19f08:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537941]   Object 0xffff8802c8b19f18:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537955]   Object 0xffff8802c8b19f28:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537969]   Object 0xffff8802c8b19f38:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537984]   Object 0xffff8802c8b19f48:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537998]   Object 0xffff8802c8b19f58:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.538012]   Object 0xffff8802c8b19f68:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.538026]   Object 0xffff8802c8b19f78:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.538041]   Object 0xffff8802c8b19f88:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.538055]   Object 0xffff8802c8b19f98:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.538069]   Object 0xffff8802c8b19fa8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.538083]   Object 0xffff8802c8b19fb8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.538097]   Object 0xffff8802c8b19fc8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.538112]   Object 0xffff8802c8b19fd8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.538126]   Object 0xffff8802c8b19fe8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.538140]   Object 0xffff8802c8b19ff8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.538154]   Object 0xffff8802c8b1a008:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.538169]   Object 0xffff8802c8b1a018:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.538183]   Object 0xffff8802c8b1a028:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.538197]   Object 0xffff8802c8b1a038:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.538211]   Object 0xffff8802c8b1a048:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.538226]   Object 0xffff8802c8b1a058:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.538240]   Object 0xffff8802c8b1a068:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.538254]   Object 0xffff8802c8b1a078:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.538268]   Object 0xffff8802c8b1a088:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.538283]   Object 0xffff8802c8b1a098:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.538297]   Object 0xffff8802c8b1a0a8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.538311]   Object 0xffff8802c8b1a0b8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.538325]   Object 0xffff8802c8b1a0c8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b a5 kkkkkkkkkkkkkkk¥
[  789.538340]  Redzone 0xffff8802c8b1a0d8:  bb bb bb bb bb bb bb bb 
                      »»»»»»»»
[  789.538353]  Padding 0xffff8802c8b1a118:  5a 5a 5a 5a 5a 5a 00 00 
                      ZZZZZZ..
[  789.538369] Pid: 5247, comm: qemu Not tainted 2.6.39 #2
[  789.538373] Call Trace:
[  789.538385]  [<ffffffff810b7ccd>] ? check_bytes_and_report+0x10d/0x150
[  789.538394]  [<ffffffff81314260>] ? sock_alloc_send_pskb+0x1d0/0x320
[  789.538400]  [<ffffffff810b7db9>] ? check_object+0xa9/0x260
[  789.538407]  [<ffffffff81314260>] ? sock_alloc_send_pskb+0x1d0/0x320
[  789.538414]  [<ffffffff810b86f4>] ? alloc_debug_processing+0x104/0x190
[  789.538420]  [<ffffffff810b9ac2>] ? T.912+0x272/0x2d0
[  789.538427]  [<ffffffff810bb70d>] ? __kmalloc_track_caller+0x10d/0x160
[  789.538434]  [<ffffffff813198c2>] ? __alloc_skb+0x72/0x160
[  789.538441]  [<ffffffff81314260>] ? sock_alloc_send_pskb+0x1d0/0x320
[  789.538450]  [<ffffffff8105af7b>] ? getnstimeofday+0x4b/0xd0
[  789.538457]  [<ffffffff81320d32>] ? netif_rx+0xb2/0x120
[  789.538465]  [<ffffffff812b393e>] ? tun_chr_aio_write+0x18e/0x4e0
[  789.538472]  [<ffffffff812b37b0>] ? tun_sendmsg+0x4d0/0x4d0
[  789.538481]  [<ffffffff810c24e9>] ? do_sync_readv_writev+0xa9/0xf0
[  789.538489]  [<ffffffff810c231b>] ? rw_copy_check_uvector+0x7b/0x140
[  789.538496]  [<ffffffff810c2bcf>] ? do_readv_writev+0xdf/0x210
[  789.538504]  [<ffffffff810c2e7e>] ? sys_writev+0x4e/0xc0
[  789.538513]  [<ffffffff813d753b>] ? system_call_fastpath+0x16/0x1b
[  789.538519] FIX kmalloc-2048: Restoring 
0xffff8802c8b1a11e-0xffff8802c8b1a11f=0x5a
[  789.538522]
[  825.601291] 
=============================================================================
[  825.601298] BUG kmalloc-512: Object padding overwritten
[  825.601302] 
-----------------------------------------------------------------------------
[  825.601306]
[  825.601311] INFO: 0xffff88041411efe6-0xffff88041411efe7. First byte 
0x0 instead of 0x5a
[  825.601327] INFO: Allocated in sock_alloc_send_pskb+0x1d0/0x320 
age=36742 cpu=3 pid=4846
[  825.601338] INFO: Freed in __kfree_skb+0x11/0x90 age=36742 cpu=3 pid=3781
[  825.601344] INFO: Slab 0xffffea000e463e20 objects=28 used=11 
fp=0xffff88041411eda0 flags=0x80000000000040c1
[  825.601350] INFO: Object 0xffff88041411eda0 @offset=11680 
fp=0xffff88041411cdb0
[  825.601353]
[  825.601357] Bytes b4 0xffff88041411ed90:  e2 96 ff ff 00 00 00 00 5a 
5a 5a 5a 5a 5a 5a 5a â.ÿÿ....ZZZZZZZZ
[  825.601373]   Object 0xffff88041411eda0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  825.601388]   Object 0xffff88041411edb0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  825.601402]   Object 0xffff88041411edc0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  825.601416]   Object 0xffff88041411edd0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  825.601430]   Object 0xffff88041411ede0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  825.601444]   Object 0xffff88041411edf0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  825.601459]   Object 0xffff88041411ee00:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  825.601473]   Object 0xffff88041411ee10:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  825.601487]   Object 0xffff88041411ee20:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  825.601501]   Object 0xffff88041411ee30:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  825.601515]   Object 0xffff88041411ee40:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  825.601529]   Object 0xffff88041411ee50:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  825.601543]   Object 0xffff88041411ee60:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  825.601557]   Object 0xffff88041411ee70:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  825.601571]   Object 0xffff88041411ee80:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  825.601586]   Object 0xffff88041411ee90:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  825.601600]   Object 0xffff88041411eea0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  825.601614]   Object 0xffff88041411eeb0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  825.601628]   Object 0xffff88041411eec0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  825.601642]   Object 0xffff88041411eed0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  825.601656]   Object 0xffff88041411eee0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  825.601670]   Object 0xffff88041411eef0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  825.601684]   Object 0xffff88041411ef00:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  825.601699]   Object 0xffff88041411ef10:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  825.601713]   Object 0xffff88041411ef20:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  825.601727]   Object 0xffff88041411ef30:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  825.601741]   Object 0xffff88041411ef40:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  825.601755]   Object 0xffff88041411ef50:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  825.601769]   Object 0xffff88041411ef60:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  825.601783]   Object 0xffff88041411ef70:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  825.601797]   Object 0xffff88041411ef80:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  825.601812]   Object 0xffff88041411ef90:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b a5 kkkkkkkkkkkkkkk¥
[  825.601826]  Redzone 0xffff88041411efa0:  bb bb bb bb bb bb bb bb 
                      »»»»»»»»
[  825.601839]  Padding 0xffff88041411efe0:  5a 5a 5a 5a 5a 5a 00 00 
                      ZZZZZZ..
[  825.601855] Pid: 7353, comm: apcaccess Not tainted 2.6.39 #2
[  825.601859] Call Trace:
[  825.601871]  [<ffffffff810b7ccd>] ? check_bytes_and_report+0x10d/0x150
[  825.601881]  [<ffffffff8110773c>] ? load_elf_binary+0xa6c/0x1c00
[  825.601887]  [<ffffffff810b7db9>] ? check_object+0xa9/0x260
[  825.601895]  [<ffffffff8110773c>] ? load_elf_binary+0xa6c/0x1c00
[  825.601902]  [<ffffffff810b86f4>] ? alloc_debug_processing+0x104/0x190
[  825.601908]  [<ffffffff810b9ac2>] ? T.912+0x272/0x2d0
[  825.601914]  [<ffffffff810ba59d>] ? __kmalloc+0x10d/0x160
[  825.601922]  [<ffffffff8110773c>] ? load_elf_binary+0xa6c/0x1c00
[  825.601930]  [<ffffffff8109bd9d>] ? __get_user_pages+0x17d/0x530
[  825.601938]  [<ffffffff810c9556>] ? get_arg_page+0x56/0x100
[  825.601945]  [<ffffffff810c8070>] ? search_binary_handler+0x90/0x240
[  825.601951]  [<ffffffff810c9e9f>] ? do_execve+0x22f/0x2f0
[  825.601959]  [<ffffffff810094a6>] ? sys_execve+0x36/0x60
[  825.601968]  [<ffffffff813d78dc>] ? stub_execve+0x6c/0xc0
[  825.601975] FIX kmalloc-512: Restoring 
0xffff88041411efe6-0xffff88041411efe7=0x5a
[  825.601978]
[  842.610057] INFO: task ksmd:552 blocked for more than 120 seconds.
[  842.610063] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" 
disables this message.
[  842.610069] ksmd            D 0000000000000000     0   552      2 
0x00000000
[  842.610078]  ffff88041db68ca0 0000000000000046 000212d01fc11d00 
ffff880400000000
[  842.610086]  ffffffff81593020 ffff88041c1a9fd8 0000000000004000 
ffff88041c1a8010
[  842.610093]  ffff88041c1a9fd8 ffff88041db68ca0 0000000000000000 
0000000200000002
[  842.610100] Call Trace:
[  842.610116]  [<ffffffff813d5595>] ? schedule_timeout+0x1c5/0x230
[  842.610128]  [<ffffffff8102f69c>] ? enqueue_task_fair+0x14c/0x190
[  842.610137]  [<ffffffff8102db27>] ? task_rq_lock+0x47/0x90
[  842.610143]  [<ffffffff813d4562>] ? wait_for_common+0xd2/0x180
[  842.610150]  [<ffffffff810339b0>] ? try_to_wake_up+0x2c0/0x2c0
[  842.610157]  [<ffffffff8104cb94>] ? flush_work+0x24/0x30
[  842.610163]  [<ffffffff8104be10>] ? do_work_for_cpu+0x20/0x20
[  842.610170]  [<ffffffff8104e0ab>] ? schedule_on_each_cpu+0xab/0xe0
[  842.610177]  [<ffffffff810b5c05>] ? ksm_scan_thread+0x7f5/0xc20
[  842.610184]  [<ffffffff81052a20>] ? wake_up_bit+0x40/0x40
[  842.610190]  [<ffffffff810b5410>] ? 
try_to_merge_with_ksm_page+0x570/0x570
[  842.610196]  [<ffffffff810b5410>] ? 
try_to_merge_with_ksm_page+0x570/0x570
[  842.610203]  [<ffffffff810525b6>] ? kthread+0x96/0xa0
[  842.610210]  [<ffffffff813d8294>] ? kernel_thread_helper+0x4/0x10
[  842.610218]  [<ffffffff81052520>] ? kthread_worker_fn+0x120/0x120
[  842.610223]  [<ffffffff813d8290>] ? gs_change+0xb/0xb
[  842.610229] INFO: task fsnotify_mark:662 blocked for more than 120 
seconds.
[  842.610233] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" 
disables this message.
[  842.610237] fsnotify_mark   D 0000000000000000     0   662      2 
0x00000000
[  842.610244]  ffff88041c31d860 0000000000000046 0000000000000000 
0000000000000000
[  842.610251]  ffffffff81593020 ffff88041c365fd8 0000000000004000 
ffff88041c364010
[  842.610257]  ffff88041c365fd8 ffff88041c31d860 0000000000000000 
0000000000000000
[  842.610263] Call Trace:
[  842.610270]  [<ffffffff81035521>] ? load_balance+0x91/0x5e0
[  842.610278]  [<ffffffff813d5595>] ? schedule_timeout+0x1c5/0x230
[  842.610285]  [<ffffffff8102f323>] ? pick_next_task_fair+0x103/0x190
[  842.610291]  [<ffffffff813d4a6d>] ? schedule+0x28d/0x910
[  842.610297]  [<ffffffff813d4562>] ? wait_for_common+0xd2/0x180
[  842.610303]  [<ffffffff810339b0>] ? try_to_wake_up+0x2c0/0x2c0
[  842.610310]  [<ffffffff81074420>] ? synchronize_rcu_bh+0x50/0x50
[  842.610316]  [<ffffffff8107446a>] ? synchronize_sched+0x4a/0x50
[  842.610322]  [<ffffffff8104f940>] ? find_ge_pid+0x40/0x40
[  842.610329]  [<ffffffff8105742b>] ? __synchronize_srcu+0x5b/0xc0
[  842.610338]  [<ffffffff810f5f63>] ? fsnotify_mark_destroy+0x83/0x150
[  842.610344]  [<ffffffff81052a20>] ? wake_up_bit+0x40/0x40
[  842.610352]  [<ffffffff810f5ee0>] ? 
fsnotify_set_mark_ignored_mask_locked+0x20/0x20
[  842.610360]  [<ffffffff810f5ee0>] ? 
fsnotify_set_mark_ignored_mask_locked+0x20/0x20
[  842.610367]  [<ffffffff810525b6>] ? kthread+0x96/0xa0
[  842.610373]  [<ffffffff813d8294>] ? kernel_thread_helper+0x4/0x10
[  842.610381]  [<ffffffff81052520>] ? kthread_worker_fn+0x120/0x120
[  842.610386]  [<ffffffff813d8290>] ? gs_change+0xb/0xb
[  842.610395] INFO: task jbd2/md0-8:2528 blocked for more than 120 seconds.
[  842.610399] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" 
disables this message.
[  842.610403] jbd2/md0-8      D 0000000000000005     0  2528      2 
0x00000000
[  842.610410]  ffff88041d91c570 0000000000000046 ffff88041baeea28 
ffffea0000000000
[  842.610417]  ffff88041d91a5e0 ffff880419f9bfd8 0000000000004000 
ffff880419f9a010
[  842.610423]  ffff880419f9bfd8 ffff88041d91c570 ffff88041baeea28 
ffffffff810b8495
[  842.610429] Call Trace:
[  842.610435]  [<ffffffff810b8495>] ? init_object+0x85/0xa0
[  842.610442]  [<ffffffff810b8916>] ? free_debug_processing+0x196/0x250
[  842.610450]  [<ffffffff8105adae>] ? ktime_get_ts+0x6e/0xf0
[  842.610458]  [<ffffffff810810f0>] ? __lock_page+0x70/0x70
[  842.610464]  [<ffffffff813d5174>] ? io_schedule+0x84/0xd0
[  842.610473]  [<ffffffff811d4403>] ? 
radix_tree_gang_lookup_tag_slot+0x93/0xf0
[  842.610480]  [<ffffffff810810f9>] ? sleep_on_page+0x9/0x10
[  842.610486]  [<ffffffff813d57df>] ? __wait_on_bit+0x4f/0x80
[  842.610492]  [<ffffffff810812eb>] ? wait_on_page_bit+0x6b/0x80
[  842.610499]  [<ffffffff81052a50>] ? autoremove_wake_function+0x30/0x30
[  842.610507]  [<ffffffff8108a458>] ? pagevec_lookup_tag+0x18/0x20
[  842.610512]  [<ffffffff81081f2a>] ? filemap_fdatawait_range+0xfa/0x180
[  842.610521]  [<ffffffff811be09f>] ? submit_bio+0x6f/0xf0
[  842.610530]  [<ffffffff81176276>] ? 
jbd2_journal_commit_transaction+0x796/0x1270
[  842.610539]  [<ffffffff81179ed1>] ? kjournald2+0xb1/0x1e0
[  842.610546]  [<ffffffff81052a20>] ? wake_up_bit+0x40/0x40
[  842.610553]  [<ffffffff81179e20>] ? commit_timeout+0x10/0x10
[  842.610560]  [<ffffffff81179e20>] ? commit_timeout+0x10/0x10
[  842.610566]  [<ffffffff810525b6>] ? kthread+0x96/0xa0
[  842.610572]  [<ffffffff813d8294>] ? kernel_thread_helper+0x4/0x10
[  842.610580]  [<ffffffff81052520>] ? kthread_worker_fn+0x120/0x120
[  842.610585]  [<ffffffff813d8290>] ? gs_change+0xb/0xb
[  842.610594] INFO: task nfsd:4326 blocked for more than 120 seconds.
[  842.610598] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" 
disables this message.
[  842.610602] nfsd            D 0000000000000000     0  4326      2 
0x00000000
[  842.610608]  ffff88041b604bc0 0000000000000046 0000000000000016 
ffffffff00000000
[  842.610614]  ffff88041d8fa5e0 ffff88041c527fd8 0000000000004000 
ffff88041c526010
[  842.610621]  ffff88041c527fd8 ffff88041b604bc0 000000000000009c 
ffff880000000000
[  842.610626] Call Trace:
[  842.610634]  [<ffffffff8101e845>] ? amd_flush_garts+0x105/0x140
[  842.610640]  [<ffffffff8101fa80>] ? gart_map_sg+0x480/0x480
[  842.610646]  [<ffffffff8101f5d3>] ? flush_gart+0x23/0x50
[  842.610653]  [<ffffffff81080f98>] ? find_get_page+0x18/0x90
[  842.610660]  [<ffffffff81174b95>] ? do_get_write_access+0x265/0x4a0
[  842.610668]  [<ffffffff81052a50>] ? autoremove_wake_function+0x30/0x30
[  842.610675]  [<ffffffff81174ef9>] ? 
jbd2_journal_get_write_access+0x29/0x50
[  842.610684]  [<ffffffff8115f122>] ? 
__ext4_journal_get_write_access+0x32/0x80
[  842.610692]  [<ffffffff81143908>] ? ext4_reserve_inode_write+0x78/0xa0
[  842.610700]  [<ffffffff81143970>] ? ext4_mark_inode_dirty+0x40/0x1e0
[  842.610706]  [<ffffffff81156c0b>] ? ext4_journal_start_sb+0x6b/0x160
[  842.610715]  [<ffffffff81322645>] ? dev_hard_start_xmit+0x305/0x5f0
[  842.610723]  [<ffffffff81352fd0>] ? ip_finish_output2+0x290/0x290
[  842.610730]  [<ffffffff81143c65>] ? ext4_dirty_inode+0x35/0x70
[  842.610738]  [<ffffffff810e4a08>] ? __mark_inode_dirty+0x38/0x210
[  842.610745]  [<ffffffff810d9317>] ? file_update_time+0xf7/0x180
[  842.610751]  [<ffffffff81082458>] ? __generic_file_aio_write+0x1f8/0x430
[  842.610760]  [<ffffffff81373ef9>] ? udp_sendmsg+0x3c9/0x7e0
[  842.610767]  [<ffffffff81314260>] ? sock_alloc_send_pskb+0x1d0/0x320
[  842.610774]  [<ffffffff81082703>] ? generic_file_aio_write+0x73/0xf0
[  842.610781]  [<ffffffff8113f74e>] ? ext4_file_write+0x6e/0x2b0
[  842.610788]  [<ffffffff810da8ac>] ? iget_locked+0x4c/0x140
[  842.610794]  [<ffffffff8119a590>] ? fh_compose+0x4c0/0x4c0
[  842.610800]  [<ffffffff8113f6e0>] ? ext4_llseek+0x110/0x110
[  842.610808]  [<ffffffff810c24e9>] ? do_sync_readv_writev+0xa9/0xf0
[  842.610816]  [<ffffffff810c231b>] ? rw_copy_check_uvector+0x7b/0x140
[  842.610824]  [<ffffffff810c2bcf>] ? do_readv_writev+0xdf/0x210
[  842.610830]  [<ffffffff810b8735>] ? alloc_debug_processing+0x145/0x190
[  842.610836]  [<ffffffff810b9bc5>] ? kmem_cache_alloc+0xa5/0xb0
[  842.610843]  [<ffffffff8113f463>] ? ext4_file_open+0x63/0x180
[  842.610849]  [<ffffffff8119c21d>] ? nfsd_vfs_write+0xed/0x3a0
[  842.610856]  [<ffffffff810c1127>] ? __dentry_open+0x1f7/0x2b0
[  842.610862]  [<ffffffff8119c892>] ? nfsd_open+0xf2/0x1b0
[  842.610867]  [<ffffffff8119cd34>] ? nfsd_write+0xf4/0x110
[  842.610873]  [<ffffffff81199930>] ? nfsd_proc_write+0xb0/0x120
[  842.610880]  [<ffffffff811971c5>] ? nfsd_dispatch+0xf5/0x230
[  842.610886]  [<ffffffff813b456f>] ? svc_process+0x4af/0x820
[  842.610892]  [<ffffffff810339b0>] ? try_to_wake_up+0x2c0/0x2c0
[  842.610899]  [<ffffffff811977a0>] ? nfsd_svc+0x1b0/0x1b0
[  842.610906]  [<ffffffff8119784d>] ? nfsd+0xad/0x150
[  842.610912]  [<ffffffff810525b6>] ? kthread+0x96/0xa0
[  842.610918]  [<ffffffff813d8294>] ? kernel_thread_helper+0x4/0x10
[  842.610926]  [<ffffffff81052520>] ? kthread_worker_fn+0x120/0x120
[  842.610931]  [<ffffffff813d8290>] ? gs_change+0xb/0xb
[  854.241450] 
=============================================================================
[  854.241459] BUG kmalloc-1024: Object padding overwritten
[  854.241463] 
-----------------------------------------------------------------------------
[  854.241466]
[  854.241471] INFO: 0xffff8802c86059e6-0xffff8802c86059e7. First byte 
0x0 instead of 0x5a
[  854.241488] INFO: Allocated in __blockdev_direct_IO+0x16c/0xa90 
age=518 cpu=5 pid=7094
[  854.241500] INFO: Freed in __kfree_skb+0x11/0x90 age=519 cpu=5 pid=5247
[  854.241507] INFO: Slab 0xffffea0009bd5000 objects=29 used=22 
fp=0xffff8802c8606f50 flags=0x80000000000040c1
[  854.241513] INFO: Object 0xffff8802c86055a0 @offset=21920 
fp=0xffff8802c86066c0
[  854.241516]
[  854.241520] Bytes b4 0xffff8802c8605590:  00 00 00 00 00 00 00 00 5a 
5a 5a 5a 5a 5a 5a 5a ........ZZZZZZZZ
[  854.241536]   Object 0xffff8802c86055a0:  00 00 00 00 00 00 00 00 e0 
61 29 fa 03 88 ff ff ........àa)ú..ÿÿ
[  854.241551]   Object 0xffff8802c86055b0:  00 00 00 00 00 00 00 00 00 
00 41 ce 04 00 00 00 ..........AÎ....
[  854.241565]   Object 0xffff8802c86055c0:  03 00 00 00 0c 00 00 00 00 
00 00 00 01 00 00 00 ................
[  854.241579]   Object 0xffff8802c86055d0:  00 00 00 00 00 00 00 00 00 
60 00 00 00 00 00 00 .........`......
[  854.241593]   Object 0xffff8802c86055e0:  b7 2c 0a 00 00 00 00 00 00 
00 00 00 00 00 00 00 ·,..............
[  854.241607]   Object 0xffff8802c86055f0:  b7 2c 0a 00 00 00 00 00 00 
00 00 00 00 00 00 00 ·,..............
[  854.241621]   Object 0xffff8802c8605600:  01 00 00 00 00 00 00 00 f0 
77 14 81 ff ff ff ff ........ðw..ÿÿÿÿ
[  854.241635]   Object 0xffff8802c8605610:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[  854.241649]   Object 0xffff8802c8605620:  00 00 00 00 00 00 00 00 b7 
54 3b 02 00 00 00 00 ........·T;.....
[  854.241663]   Object 0xffff8802c8605630:  b7 54 3b 02 00 00 00 00 20 
00 00 00 00 00 00 00 ·T;.............
[  854.241677]   Object 0xffff8802c8605640:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[  854.241691]   Object 0xffff8802c8605650:  b6 54 3b 02 00 00 00 00 00 
10 00 00 00 00 00 00 ¶T;.............
[  854.241705]   Object 0xffff8802c8605660:  00 00 00 00 00 00 00 00 00 
80 8e 1a 04 88 ff ff ..............ÿÿ
[  854.241719]   Object 0xffff8802c8605670:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[  854.241733]   Object 0xffff8802c8605680:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[  854.241747]   Object 0xffff8802c8605690:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[  854.241761]   Object 0xffff8802c86056a0:  00 00 00 00 00 00 00 00 00 
00 00 00 00 10 00 00 ................
[  854.241775]   Object 0xffff8802c86056b0:  b6 54 3b 02 00 00 00 00 00 
60 cb a2 00 00 00 00 ¶T;......`Ë¢....
[  854.241789]   Object 0xffff8802c86056c0:  06 06 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[  854.241803]   Object 0xffff8802c86056d0:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[  854.241817]   Object 0xffff8802c86056e0:  68 fd 1c c9 02 88 ff ff 00 
00 00 00 00 00 00 00 hý.É..ÿÿ........
[  854.241831]   Object 0xffff8802c86056f0:  00 60 00 00 00 00 00 00 01 
00 00 00 01 00 00 00 .`..............
[  854.241845]   Object 0xffff8802c8605700:  00 80 c9 dd 39 7f 00 00 01 
00 00 00 01 00 00 00 ..ÉÝ9...........
[  854.241860]   Object 0xffff8802c8605710:  00 00 00 00 00 00 00 00 08 
81 bb 0c 00 ea ff ff ..........»..êÿÿ
[  854.241874]   Object 0xffff8802c8605720:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.241888]   Object 0xffff8802c8605730:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.241902]   Object 0xffff8802c8605740:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.241916]   Object 0xffff8802c8605750:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.241930]   Object 0xffff8802c8605760:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.241944]   Object 0xffff8802c8605770:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.241959]   Object 0xffff8802c8605780:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.241973]   Object 0xffff8802c8605790:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.241987]   Object 0xffff8802c86057a0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.242001]   Object 0xffff8802c86057b0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.242015]   Object 0xffff8802c86057c0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.242029]   Object 0xffff8802c86057d0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.242043]   Object 0xffff8802c86057e0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.242058]   Object 0xffff8802c86057f0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.242072]   Object 0xffff8802c8605800:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.242086]   Object 0xffff8802c8605810:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.242100]   Object 0xffff8802c8605820:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.242114]   Object 0xffff8802c8605830:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.242128]   Object 0xffff8802c8605840:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.242142]   Object 0xffff8802c8605850:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.242157]   Object 0xffff8802c8605860:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.242171]   Object 0xffff8802c8605870:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.242185]   Object 0xffff8802c8605880:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.242199]   Object 0xffff8802c8605890:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.242213]   Object 0xffff8802c86058a0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.242227]   Object 0xffff8802c86058b0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.242241]   Object 0xffff8802c86058c0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.242256]   Object 0xffff8802c86058d0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.242270]   Object 0xffff8802c86058e0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.242284]   Object 0xffff8802c86058f0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.242298]   Object 0xffff8802c8605900:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.242312]   Object 0xffff8802c8605910:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.242326]   Object 0xffff8802c8605920:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.242340]   Object 0xffff8802c8605930:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.242355]   Object 0xffff8802c8605940:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.242369]   Object 0xffff8802c8605950:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.242383]   Object 0xffff8802c8605960:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.242397]   Object 0xffff8802c8605970:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.242411]   Object 0xffff8802c8605980:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.242425]   Object 0xffff8802c8605990:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b a5 kkkkkkkkkkkkkkk¥
[  854.242440]  Redzone 0xffff8802c86059a0:  cc cc cc cc cc cc cc cc 
                      ÌÌÌÌÌÌÌÌ
[  854.242453]  Padding 0xffff8802c86059e0:  5a 5a 5a 5a 5a 5a 00 00 
                      ZZZZZZ..
[  854.242470] Pid: 7094, comm: qemu Not tainted 2.6.39 #2
[  854.242474] Call Trace:
[  854.242485]  [<ffffffff810b7ccd>] ? check_bytes_and_report+0x10d/0x150
[  854.242492]  [<ffffffff810b7db9>] ? check_object+0xa9/0x260
[  854.242500]  [<ffffffff810f3126>] ? __blockdev_direct_IO+0xa16/0xa90
[  854.242507]  [<ffffffff810b88b3>] ? free_debug_processing+0x133/0x250
[  854.242513]  [<ffffffff810b8afb>] ? __slab_free+0x12b/0x140
[  854.242520]  [<ffffffff810f3126>] ? __blockdev_direct_IO+0xa16/0xa90
[  854.242530]  [<ffffffff81144a17>] ? ext4_ind_direct_IO+0xf7/0x410
[  854.242536]  [<ffffffff811477f0>] ? noalloc_get_block_write+0x30/0x30
[  854.242543]  [<ffffffff81082029>] ? __filemap_fdatawrite_range+0x49/0x50
[  854.242550]  [<ffffffff810830c3>] ? generic_file_aio_read+0x653/0x6d0
[  854.242558]  [<ffffffff810567a2>] ? hrtimer_cancel+0x12/0x20
[  854.242565]  [<ffffffff81062137>] ? futex_wait+0x197/0x240
[  854.242572]  [<ffffffff81082a70>] ? delete_from_page_cache+0x70/0x70
[  854.242580]  [<ffffffff810c24e9>] ? do_sync_readv_writev+0xa9/0xf0
[  854.242587]  [<ffffffff8106239f>] ? futex_wake+0x10f/0x120
[  854.242594]  [<ffffffff81063f7b>] ? do_futex+0x11b/0xa70
[  854.242601]  [<ffffffff810471cc>] ? T.680+0x13c/0x280
[  854.242608]  [<ffffffff810c231b>] ? rw_copy_check_uvector+0x7b/0x140
[  854.242616]  [<ffffffff810c2bcf>] ? do_readv_writev+0xdf/0x210
[  854.242624]  [<ffffffff810c3003>] ? sys_preadv+0xc3/0xd0
[  854.242633]  [<ffffffff813d753b>] ? system_call_fastpath+0x16/0x1b
[  854.242640] FIX kmalloc-1024: Restoring 
0xffff8802c86059e6-0xffff8802c86059e7=0x5a
[  854.242643]
[  860.203918] 
=============================================================================
[  860.203927] BUG kmalloc-1024: Object padding overwritten
[  860.203931] 
-----------------------------------------------------------------------------
[  860.203934]
[  860.203939] INFO: 0xffff88041ad15156-0xffff88041ad15157. First byte 
0x0 instead of 0x5a
[  860.203955] INFO: Allocated in __blockdev_direct_IO+0x16c/0xa90 
age=1609 cpu=3 pid=7094
[  860.203965] INFO: Freed in __blockdev_direct_IO+0xa16/0xa90 age=1609 
cpu=3 pid=7094
[  860.203972] INFO: Slab 0xffffea000e5ddb80 objects=29 used=25 
fp=0xffff88041ad14d10 flags=0x80000000000040c1
[  860.203977] INFO: Object 0xffff88041ad14d10 @offset=19728 
fp=0xffff88041ad10000
[  860.203981]
[  860.203984] Bytes b4 0xffff88041ad14d00:  62 28 00 00 01 00 00 00 5a 
5a 5a 5a 5a 5a 5a 5a b(......ZZZZZZZZ
[  860.204000]   Object 0xffff88041ad14d10:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204015]   Object 0xffff88041ad14d20:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204030]   Object 0xffff88041ad14d30:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204044]   Object 0xffff88041ad14d40:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204058]   Object 0xffff88041ad14d50:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204073]   Object 0xffff88041ad14d60:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204087]   Object 0xffff88041ad14d70:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204101]   Object 0xffff88041ad14d80:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204116]   Object 0xffff88041ad14d90:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204130]   Object 0xffff88041ad14da0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204144]   Object 0xffff88041ad14db0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204159]   Object 0xffff88041ad14dc0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204173]   Object 0xffff88041ad14dd0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204187]   Object 0xffff88041ad14de0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204202]   Object 0xffff88041ad14df0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204216]   Object 0xffff88041ad14e00:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204230]   Object 0xffff88041ad14e10:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204245]   Object 0xffff88041ad14e20:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204259]   Object 0xffff88041ad14e30:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204273]   Object 0xffff88041ad14e40:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204288]   Object 0xffff88041ad14e50:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204302]   Object 0xffff88041ad14e60:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204316]   Object 0xffff88041ad14e70:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204331]   Object 0xffff88041ad14e80:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204345]   Object 0xffff88041ad14e90:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204359]   Object 0xffff88041ad14ea0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204374]   Object 0xffff88041ad14eb0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204388]   Object 0xffff88041ad14ec0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204402]   Object 0xffff88041ad14ed0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204417]   Object 0xffff88041ad14ee0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204431]   Object 0xffff88041ad14ef0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204445]   Object 0xffff88041ad14f00:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204460]   Object 0xffff88041ad14f10:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204474]   Object 0xffff88041ad14f20:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204488]   Object 0xffff88041ad14f30:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204503]   Object 0xffff88041ad14f40:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204517]   Object 0xffff88041ad14f50:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204531]   Object 0xffff88041ad14f60:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204546]   Object 0xffff88041ad14f70:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204560]   Object 0xffff88041ad14f80:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204574]   Object 0xffff88041ad14f90:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204589]   Object 0xffff88041ad14fa0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204603]   Object 0xffff88041ad14fb0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204618]   Object 0xffff88041ad14fc0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204632]   Object 0xffff88041ad14fd0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204646]   Object 0xffff88041ad14fe0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204661]   Object 0xffff88041ad14ff0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204675]   Object 0xffff88041ad15000:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204689]   Object 0xffff88041ad15010:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204704]   Object 0xffff88041ad15020:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204718]   Object 0xffff88041ad15030:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204732]   Object 0xffff88041ad15040:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204747]   Object 0xffff88041ad15050:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204761]   Object 0xffff88041ad15060:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204775]   Object 0xffff88041ad15070:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204790]   Object 0xffff88041ad15080:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204804]   Object 0xffff88041ad15090:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204818]   Object 0xffff88041ad150a0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204833]   Object 0xffff88041ad150b0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204847]   Object 0xffff88041ad150c0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204861]   Object 0xffff88041ad150d0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204876]   Object 0xffff88041ad150e0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204890]   Object 0xffff88041ad150f0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204904]   Object 0xffff88041ad15100:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b a5 kkkkkkkkkkkkkkk¥
[  860.204919]  Redzone 0xffff88041ad15110:  bb bb bb bb bb bb bb bb 
                      »»»»»»»»
[  860.204932]  Padding 0xffff88041ad15150:  5a 5a 5a 5a 5a 5a 00 00 
                      ZZZZZZ..
[  860.204949] Pid: 7100, comm: qemu Not tainted 2.6.39 #2
[  860.204953] Call Trace:
[  860.204964]  [<ffffffff810b7ccd>] ? check_bytes_and_report+0x10d/0x150
[  860.204973]  [<ffffffff810f287c>] ? __blockdev_direct_IO+0x16c/0xa90
[  860.204979]  [<ffffffff810b7db9>] ? check_object+0xa9/0x260
[  860.204986]  [<ffffffff810f287c>] ? __blockdev_direct_IO+0x16c/0xa90
[  860.204993]  [<ffffffff810b86f4>] ? alloc_debug_processing+0x104/0x190
[  860.204999]  [<ffffffff810b9ac2>] ? T.912+0x272/0x2d0
[  860.205006]  [<ffffffff810b9bc5>] ? kmem_cache_alloc+0xa5/0xb0
[  860.205013]  [<ffffffff810f287c>] ? __blockdev_direct_IO+0x16c/0xa90
[  860.205022]  [<ffffffff8108a475>] ? pagevec_lookup+0x15/0x20
[  860.205029]  [<ffffffff8108b5b7>] ? 
invalidate_inode_pages2_range+0x87/0x300
[  860.205039]  [<ffffffff81144e41>] ? ext4_direct_IO+0x111/0x1f0
[  860.205045]  [<ffffffff811477a0>] ? _ext4_get_block+0x160/0x160
[  860.205052]  [<ffffffff81144f20>] ? ext4_direct_IO+0x1f0/0x1f0
[  860.205058]  [<ffffffff81082182>] ? generic_file_direct_write+0xd2/0x1b0
[  860.205064]  [<ffffffff8108252b>] ? __generic_file_aio_write+0x2cb/0x430
[  860.205071]  [<ffffffff81082703>] ? generic_file_aio_write+0x73/0xf0
[  860.205079]  [<ffffffff8113f74e>] ? ext4_file_write+0x6e/0x2b0
[  860.205087]  [<ffffffff810c25ef>] ? do_sync_write+0xbf/0x100
[  860.205095]  [<ffffffff810471cc>] ? T.680+0x13c/0x280
[  860.205103]  [<ffffffff810485a8>] ? kill_pid_info+0x38/0x60
[  860.205110]  [<ffffffff81048854>] ? sys_kill+0x94/0x1b0
[  860.205117]  [<ffffffff810c3198>] ? vfs_write+0xc8/0x190
[  860.205124]  [<ffffffff810c32fb>] ? sys_pwrite64+0x9b/0xb0
[  860.205134]  [<ffffffff813d753b>] ? system_call_fastpath+0x16/0x1b
[  860.205140] FIX kmalloc-1024: Restoring 
0xffff88041ad15156-0xffff88041ad15157=0x5a
[  860.205143]

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
@ 2011-06-01  9:29                       ` Brad Campbell
  0 siblings, 0 replies; 111+ messages in thread
From: Brad Campbell @ 2011-06-01  9:29 UTC (permalink / raw)
  To: Avi Kivity
  Cc: Hugh Dickins, Andrea Arcangeli, Borislav Petkov, linux-kernel,
	kvm, linux-mm

On 01/06/11 14:56, Avi Kivity wrote:
> On 06/01/2011 09:31 AM, Brad Campbell wrote:
>> On 01/06/11 12:52, Hugh Dickins wrote:
>>
>>>
>>> I guess Brad could try SLUB debugging, boot with slub_debug=P
>>> for poisoning perhaps; though it might upset alignments and
>>> drive the problem underground. Or see if the same happens
>>> with SLAB instead of SLUB.
>>
>> Not much use I'm afraid.
>> This is all I get in the log
>>
>> [ 3161.300073]
>> =============================================================================
>>
>> [ 3161.300147] BUG kmalloc-512: Freechain corrupt
>>
>> The qemu process is then frozen, unkillable but reported in state "R"
>>
>> 13881 ? R 3:27 /usr/bin/qemu -S -M pc-0.13 -enable-kvm -m 1024 -smp
>> 2,sockets=2,cores=1,threads=1 -nam
>>
>> The machine then progressively dies until it's frozen solid with no
>> further error messages.
>>
>> I stupidly forgot to do an alt-sysrq-t prior to doing an alt-sysrq-b,
>> but at least it responded to that.
>>
>> On the bright side I can reproduce it at will.
>
> Please try slub_debug=FZPU; that should point the finger (hopefully at
> somebody else).
>

Well the first attempt locked the machine solid. No network, no console..

I saw 
"=========================================================================="

on the console.. nothing after that. Would not respond to sysrq-t or any 
other sysrq combination other than -b, which rebooted the box.


No output on netconsole at all, I had to walk to the other building to 
look at the monitor and reboot it.

The second attempt jammed netconsole again, but I managed to get this 
from an ssh session I already had established. The machine died a slow 
and horrible death, but remained interactive enough for me to reboot it with

echo b > /proc/sysrq-trigger

Nothing else worked.



[  376.269051] 
=============================================================================
[  413.755328] 
=============================================================================
[  413.755337] BUG kmalloc-1024: Object padding overwritten
[  413.755341] 
-----------------------------------------------------------------------------
[  413.755344]
[  413.755349] INFO: 0xffff880417179566-0xffff880417179567. First byte 
0x0 instead of 0x5a
[  413.755363] INFO: Allocated in tcp_send_ack+0x26/0x120 age=5320 cpu=5 
pid=0
[  413.755374] INFO: Freed in __kfree_skb+0x11/0x90 age=5320 cpu=5 pid=0
[  413.755380] INFO: Slab 0xffffea000e50d240 objects=29 used=5 
fp=0xffff880417179120 flags=0x80000000000040c1
[  413.755386] INFO: Object 0xffff880417179120 @offset=4384 
fp=0xffff8804171799b0
[  413.755389]
[  413.755392] Bytes b4 0xffff880417179110:  d9 2b 00 00 01 00 00 00 5a 
5a 5a 5a 5a 5a 5a 5a A?+......ZZZZZZZZ
[  413.755408]   Object 0xffff880417179120:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755423]   Object 0xffff880417179130:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755438]   Object 0xffff880417179140:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755452]   Object 0xffff880417179150:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755466]   Object 0xffff880417179160:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755480]   Object 0xffff880417179170:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755494]   Object 0xffff880417179180:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755508]   Object 0xffff880417179190:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755522]   Object 0xffff8804171791a0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755536]   Object 0xffff8804171791b0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755551]   Object 0xffff8804171791c0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755565]   Object 0xffff8804171791d0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755579]   Object 0xffff8804171791e0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755593]   Object 0xffff8804171791f0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755607]   Object 0xffff880417179200:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755621]   Object 0xffff880417179210:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755635]   Object 0xffff880417179220:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755650]   Object 0xffff880417179230:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755664]   Object 0xffff880417179240:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755678]   Object 0xffff880417179250:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755692]   Object 0xffff880417179260:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755706]   Object 0xffff880417179270:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755720]   Object 0xffff880417179280:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755734]   Object 0xffff880417179290:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755749]   Object 0xffff8804171792a0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755763]   Object 0xffff8804171792b0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755777]   Object 0xffff8804171792c0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755791]   Object 0xffff8804171792d0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755805]   Object 0xffff8804171792e0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755819]   Object 0xffff8804171792f0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755834]   Object 0xffff880417179300:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755848]   Object 0xffff880417179310:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755862]   Object 0xffff880417179320:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755876]   Object 0xffff880417179330:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755890]   Object 0xffff880417179340:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755904]   Object 0xffff880417179350:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755919]   Object 0xffff880417179360:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755933]   Object 0xffff880417179370:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755947]   Object 0xffff880417179380:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755961]   Object 0xffff880417179390:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755975]   Object 0xffff8804171793a0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755989]   Object 0xffff8804171793b0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.756004]   Object 0xffff8804171793c0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.756018]   Object 0xffff8804171793d0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.756032]   Object 0xffff8804171793e0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.756046]   Object 0xffff8804171793f0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.756060]   Object 0xffff880417179400:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.756074]   Object 0xffff880417179410:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.756089]   Object 0xffff880417179420:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.756103]   Object 0xffff880417179430:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.756117]   Object 0xffff880417179440:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.756131]   Object 0xffff880417179450:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.756145]   Object 0xffff880417179460:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.756160]   Object 0xffff880417179470:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.756174]   Object 0xffff880417179480:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.756188]   Object 0xffff880417179490:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.756202]   Object 0xffff8804171794a0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.756216]   Object 0xffff8804171794b0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.756230]   Object 0xffff8804171794c0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.756245]   Object 0xffff8804171794d0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.756259]   Object 0xffff8804171794e0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.756273]   Object 0xffff8804171794f0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.756287]   Object 0xffff880417179500:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.756301]   Object 0xffff880417179510:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b a5 kkkkkkkkkkkkkkkAJPY
[  413.756316]  Redzone 0xffff880417179520:  bb bb bb bb bb bb bb bb 
                      A>>A>>A>>A>>A>>A>>A>>A>>
[  413.756329]  Padding 0xffff880417179560:  5a 5a 5a 5a 5a 5a 00 00 
                      ZZZZZZ..
[  413.756345] Pid: 5247, comm: qemu Not tainted 2.6.39 #2
[  413.756349] Call Trace:
[  413.756353]  <IRQ>  [<ffffffff810b7ccd>] ? 
check_bytes_and_report+0x10d/0x150
[  413.756372]  [<ffffffff81318f1c>] ? pskb_expand_head+0x15c/0x250
[  413.756379]  [<ffffffff810b7db9>] ? check_object+0xa9/0x260
[  413.756387]  [<ffffffff81318f1c>] ? pskb_expand_head+0x15c/0x250
[  413.756393]  [<ffffffff810b86f4>] ? alloc_debug_processing+0x104/0x190
[  413.756400]  [<ffffffff810b9ac2>] ? T.912+0x272/0x2d0
[  413.756409]  [<ffffffff810ba59d>] ? __kmalloc+0x10d/0x160
[  413.756416]  [<ffffffff81318f1c>] ? pskb_expand_head+0x15c/0x250
[  413.756424]  [<ffffffff813a6c45>] ? nf_bridge_copy_header+0x145/0x160
[  413.756431]  [<ffffffff8139f78d>] ? br_dev_queue_push_xmit+0x6d/0x80
[  413.756439]  [<ffffffff813a55a0>] ? br_nf_post_routing+0x2a0/0x2f0
[  413.756447]  [<ffffffff81346bc4>] ? nf_iterate+0x84/0xb0
[  413.756453]  [<ffffffff8139f720>] ? br_flood_deliver+0x20/0x20
[  413.756459]  [<ffffffff81346c64>] ? nf_hook_slow+0x74/0x120
[  413.756465]  [<ffffffff8139f720>] ? br_flood_deliver+0x20/0x20
[  413.756472]  [<ffffffff8139f7da>] ? br_forward_finish+0x3a/0x60
[  413.756479]  [<ffffffff813a5758>] ? br_nf_forward_finish+0x168/0x170
[  413.756487]  [<ffffffff813a5c90>] ? br_nf_forward_ip+0x360/0x3a0
[  413.756492]  [<ffffffff81346bc4>] ? nf_iterate+0x84/0xb0
[  413.756498]  [<ffffffff8139f7a0>] ? br_dev_queue_push_xmit+0x80/0x80
[  413.756504]  [<ffffffff81346c64>] ? nf_hook_slow+0x74/0x120
[  413.756510]  [<ffffffff8139f7a0>] ? br_dev_queue_push_xmit+0x80/0x80
[  413.756516]  [<ffffffff8139f800>] ? br_forward_finish+0x60/0x60
[  413.756522]  [<ffffffff8139f800>] ? br_forward_finish+0x60/0x60
[  413.756528]  [<ffffffff8139f875>] ? __br_forward+0x75/0xc0
[  413.756534]  [<ffffffff8139f426>] ? deliver_clone+0x36/0x60
[  413.756540]  [<ffffffff8139f69d>] ? br_flood+0xbd/0x100
[  413.756546]  [<ffffffff813a05b0>] ? br_handle_local_finish+0x40/0x40
[  413.756552]  [<ffffffff813a080e>] ? br_handle_frame_finish+0x25e/0x280
[  413.756560]  [<ffffffff813a60f0>] ? br_nf_pre_routing_finish+0x1a0/0x330
[  413.756568]  [<ffffffff813a6958>] ? br_nf_pre_routing+0x6d8/0x800
[  413.756577]  [<ffffffff8102d46a>] ? enqueue_task+0x3a/0x90
[  413.756582]  [<ffffffff81346bc4>] ? nf_iterate+0x84/0xb0
[  413.756589]  [<ffffffff813a05b0>] ? br_handle_local_finish+0x40/0x40
[  413.756594]  [<ffffffff81346c64>] ? nf_hook_slow+0x74/0x120
[  413.756600]  [<ffffffff813a05b0>] ? br_handle_local_finish+0x40/0x40
[  413.756607]  [<ffffffff810339b0>] ? try_to_wake_up+0x2c0/0x2c0
[  413.756613]  [<ffffffff813a09d9>] ? br_handle_frame+0x1a9/0x280
[  413.756620]  [<ffffffff813a0830>] ? br_handle_frame_finish+0x280/0x280
[  413.756627]  [<ffffffff81320ef7>] ? __netif_receive_skb+0x157/0x5c0
[  413.756634]  [<ffffffff81321443>] ? process_backlog+0xe3/0x1d0
[  413.756641]  [<ffffffff81321da5>] ? net_rx_action+0xc5/0x1d0
[  413.756650]  [<ffffffff8103df11>] ? __do_softirq+0x91/0x120
[  413.756657]  [<ffffffff813d838c>] ? call_softirq+0x1c/0x30
[  413.756660]  <EOI>  [<ffffffff81003cbd>] ? do_softirq+0x4d/0x80
[  413.756673]  [<ffffffff81321ece>] ? netif_rx_ni+0x1e/0x30
[  413.756681]  [<ffffffff812b3ae2>] ? tun_chr_aio_write+0x332/0x4e0
[  413.756688]  [<ffffffff812b37b0>] ? tun_sendmsg+0x4d0/0x4d0
[  413.756697]  [<ffffffff810c24e9>] ? do_sync_readv_writev+0xa9/0xf0
[  413.756704]  [<ffffffff81063f9c>] ? do_futex+0x13c/0xa70
[  413.756711]  [<ffffffff811d6730>] ? timerqueue_add+0x60/0xb0
[  413.756719]  [<ffffffff81056ab7>] ? __hrtimer_start_range_ns+0x1e7/0x410
[  413.756726]  [<ffffffff810c231b>] ? rw_copy_check_uvector+0x7b/0x140
[  413.756734]  [<ffffffff810c2bcf>] ? do_readv_writev+0xdf/0x210
[  413.756742]  [<ffffffff810c2e7e>] ? sys_writev+0x4e/0xc0
[  413.756750]  [<ffffffff813d753b>] ? system_call_fastpath+0x16/0x1b
[  413.756756] FIX kmalloc-1024: Restoring 
0xffff880417179566-0xffff880417179567=0x5a
[  413.756760]
[  556.640033] 
=============================================================================
[  556.640041] BUG kmalloc-512: Object padding overwritten
[  556.640045] 
-----------------------------------------------------------------------------
[  556.640048]
[  556.640053] INFO: 0xffff880403bf091e-0xffff880403bf091f. First byte 
0x0 instead of 0x5a
[  556.640069] INFO: Allocated in sock_alloc_send_pskb+0x1d0/0x320 
age=21401 cpu=2 pid=5630
[  556.640080] INFO: Freed in __kfree_skb+0x11/0x90 age=21386 cpu=2 pid=3753
[  556.640086] INFO: Slab 0xffffea000e0d1c80 objects=28 used=10 
fp=0xffff880403bf06d8 flags=0x80000000000040c1
[  556.640092] INFO: Object 0xffff880403bf06d8 @offset=1752 
fp=0xffff880403bf1488
[  556.640096]
[  556.640099] Bytes b4 0xffff880403bf06c8:  90 c4 ff ff 00 00 00 00 5a 
5a 5a 5a 5a 5a 5a 5a .A?A?A?....ZZZZZZZZ
[  556.640115]   Object 0xffff880403bf06d8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  556.640130]   Object 0xffff880403bf06e8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  556.640144]   Object 0xffff880403bf06f8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  556.640158]   Object 0xffff880403bf0708:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  556.640172]   Object 0xffff880403bf0718:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  556.640187]   Object 0xffff880403bf0728:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  556.640201]   Object 0xffff880403bf0738:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  556.640215]   Object 0xffff880403bf0748:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  556.640229]   Object 0xffff880403bf0758:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  556.640243]   Object 0xffff880403bf0768:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  556.640257]   Object 0xffff880403bf0778:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  556.640271]   Object 0xffff880403bf0788:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  556.640286]   Object 0xffff880403bf0798:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  556.640300]   Object 0xffff880403bf07a8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  556.640314]   Object 0xffff880403bf07b8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  556.640328]   Object 0xffff880403bf07c8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  556.640343]   Object 0xffff880403bf07d8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  556.640357]   Object 0xffff880403bf07e8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  556.640371]   Object 0xffff880403bf07f8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  556.640385]   Object 0xffff880403bf0808:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  556.640399]   Object 0xffff880403bf0818:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  556.640413]   Object 0xffff880403bf0828:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  556.640428]   Object 0xffff880403bf0838:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  556.640442]   Object 0xffff880403bf0848:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  556.640456]   Object 0xffff880403bf0858:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  556.640471]   Object 0xffff880403bf0868:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  556.640485]   Object 0xffff880403bf0878:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  556.640499]   Object 0xffff880403bf0888:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  556.640513]   Object 0xffff880403bf0898:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  556.640527]   Object 0xffff880403bf08a8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  556.640542]   Object 0xffff880403bf08b8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  556.640556]   Object 0xffff880403bf08c8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b a5 kkkkkkkkkkkkkkkAJPY
[  556.640570]  Redzone 0xffff880403bf08d8:  bb bb bb bb bb bb bb bb 
                      A>>A>>A>>A>>A>>A>>A>>A>>
[  556.640583]  Padding 0xffff880403bf0918:  5a 5a 5a 5a 5a 5a 00 00 
                      ZZZZZZ..
[  556.640599] Pid: 4809, comm: qemu Not tainted 2.6.39 #2
[  556.640603] Call Trace:
[  556.640607]  <IRQ>  [<ffffffff810b7ccd>] ? 
check_bytes_and_report+0x10d/0x150
[  556.640626]  [<ffffffff8131aae7>] ? __netdev_alloc_skb+0x17/0x40
[  556.640632]  [<ffffffff810b7db9>] ? check_object+0xa9/0x260
[  556.640640]  [<ffffffff8131aae7>] ? __netdev_alloc_skb+0x17/0x40
[  556.640647]  [<ffffffff810b86f4>] ? alloc_debug_processing+0x104/0x190
[  556.640654]  [<ffffffff810b9ac2>] ? T.912+0x272/0x2d0
[  556.640661]  [<ffffffff810bb70d>] ? __kmalloc_track_caller+0x10d/0x160
[  556.640668]  [<ffffffff813198c2>] ? __alloc_skb+0x72/0x160
[  556.640676]  [<ffffffff8131aae7>] ? __netdev_alloc_skb+0x17/0x40
[  556.640684]  [<ffffffff813a7b93>] ? 
br_ip4_multicast_alloc_query+0x23/0x1c0
[  556.640690]  [<ffffffff813a7e76>] ? br_multicast_send_query+0x76/0x130
[  556.640698]  [<ffffffff8104dba8>] ? wq_worker_waking_up+0x8/0x30
[  556.640706]  [<ffffffff8102ca81>] ? sched_slice+0x51/0x80
[  556.640711]  [<ffffffff813a7fb5>] ? 
br_multicast_port_query_expired+0x45/0x70
[  556.640719]  [<ffffffff81043f77>] ? run_timer_softirq+0x137/0x270
[  556.640725]  [<ffffffff81033fe9>] ? scheduler_tick+0x289/0x2d0
[  556.640731]  [<ffffffff813a7f70>] ? br_multicast_query_expired+0x40/0x40
[  556.640740]  [<ffffffff8103df11>] ? __do_softirq+0x91/0x120
[  556.640747]  [<ffffffff813d838c>] ? call_softirq+0x1c/0x30
[  556.640755]  [<ffffffff81003cbd>] ? do_softirq+0x4d/0x80
[  556.640762]  [<ffffffff8103ddbe>] ? irq_exit+0x8e/0xb0
[  556.640770]  [<ffffffff81019d1a>] ? smp_apic_timer_interrupt+0x6a/0xa0
[  556.640776]  [<ffffffff813d7e53>] ? apic_timer_interrupt+0x13/0x20
[  556.640780]  <EOI>  [<ffffffff8102965a>] ? 
flush_tlb_others_ipi+0x11a/0x130
[  556.640791]  [<ffffffff81029647>] ? flush_tlb_others_ipi+0x107/0x130
[  556.640799]  [<ffffffff810aadab>] ? ptep_clear_flush+0xb/0x10
[  556.640806]  [<ffffffff8109a40b>] ? do_wp_page+0x34b/0x7a0
[  556.640836]  [<ffffffffa00a188a>] ? kvm_read_guest_page+0x5a/0x70 [kvm]
[  556.640863]  [<ffffffffa00b7db4>] ? 
paging64_walk_addr_generic+0x264/0x4c0 [kvm]
[  556.640871]  [<ffffffff8109acad>] ? handle_pte_fault+0x44d/0x990
[  556.640878]  [<ffffffff8109b4e8>] ? follow_page+0x268/0x440
[  556.640886]  [<ffffffff8109bd4d>] ? __get_user_pages+0x12d/0x530
[  556.640892]  [<ffffffff81028dbb>] ? gup_pud_range+0x12b/0x1b0
[  556.640914]  [<ffffffffa00a2d2e>] ? get_user_page_nowait+0x2e/0x40 [kvm]
[  556.640935]  [<ffffffffa00a2e7b>] ? hva_to_pfn+0x13b/0x440 [kvm]
[  556.640956]  [<ffffffffa00a31e9>] ? __gfn_to_pfn+0x39/0xf0 [kvm]
[  556.640979]  [<ffffffffa00b825d>] ? try_async_pf+0x4d/0x190 [kvm]
[  556.641002]  [<ffffffffa00b92be>] ? tdp_page_fault+0x10e/0x200 [kvm]
[  556.641026]  [<ffffffffa00b968c>] ? kvm_mmu_page_fault+0x1c/0x80 [kvm]
[  556.641050]  [<ffffffffa00b0a7a>] ? 
kvm_arch_vcpu_ioctl_run+0x3fa/0xcf0 [kvm]
[  556.641058]  [<ffffffff8106239f>] ? futex_wake+0x10f/0x120
[  556.641065]  [<ffffffff81063f7b>] ? do_futex+0x11b/0xa70
[  556.641086]  [<ffffffffa00a517f>] ? kvm_vcpu_ioctl+0x4df/0x5e0 [kvm]
[  556.641097]  [<ffffffff81046671>] ? __dequeue_signal+0xe1/0x170
[  556.641104]  [<ffffffff8104821c>] ? do_send_sig_info+0x6c/0x90
[  556.641112]  [<ffffffff81046aac>] ? dequeue_signal+0x3c/0x170
[  556.641119]  [<ffffffff8104551f>] ? copy_siginfo_to_user+0xef/0x1d0
[  556.641125]  [<ffffffff810d26db>] ? do_vfs_ioctl+0x9b/0x4f0
[  556.641132]  [<ffffffff8106494a>] ? sys_futex+0x7a/0x180
[  556.641137]  [<ffffffff810d2b79>] ? sys_ioctl+0x49/0x80
[  556.641145]  [<ffffffff813d753b>] ? system_call_fastpath+0x16/0x1b
[  556.641152] FIX kmalloc-512: Restoring 
0xffff880403bf091e-0xffff880403bf091f=0x5a
[  556.641155]
[  602.610062] INFO: task ksmd:552 blocked for more than 120 seconds.
[  602.610068] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" 
disables this message.
[  602.610073] ksmd            D 0000000000000000     0   552      2 
0x00000000
[  602.610083]  ffff88041db68ca0 0000000000000046 000212d01fc11d00 
ffff880400000000
[  602.610090]  ffffffff81593020 ffff88041c1a9fd8 0000000000004000 
ffff88041c1a8010
[  602.610097]  ffff88041c1a9fd8 ffff88041db68ca0 0000000000000000 
0000000200000002
[  602.610104] Call Trace:
[  602.610121]  [<ffffffff813d5595>] ? schedule_timeout+0x1c5/0x230
[  602.610132]  [<ffffffff8102f69c>] ? enqueue_task_fair+0x14c/0x190
[  602.610141]  [<ffffffff8102db27>] ? task_rq_lock+0x47/0x90
[  602.610148]  [<ffffffff813d4562>] ? wait_for_common+0xd2/0x180
[  602.610154]  [<ffffffff810339b0>] ? try_to_wake_up+0x2c0/0x2c0
[  602.610162]  [<ffffffff8104cb94>] ? flush_work+0x24/0x30
[  602.610167]  [<ffffffff8104be10>] ? do_work_for_cpu+0x20/0x20
[  602.610174]  [<ffffffff8104e0ab>] ? schedule_on_each_cpu+0xab/0xe0
[  602.610181]  [<ffffffff810b5c05>] ? ksm_scan_thread+0x7f5/0xc20
[  602.610189]  [<ffffffff81052a20>] ? wake_up_bit+0x40/0x40
[  602.610194]  [<ffffffff810b5410>] ? 
try_to_merge_with_ksm_page+0x570/0x570
[  602.610200]  [<ffffffff810b5410>] ? 
try_to_merge_with_ksm_page+0x570/0x570
[  602.610207]  [<ffffffff810525b6>] ? kthread+0x96/0xa0
[  602.610214]  [<ffffffff813d8294>] ? kernel_thread_helper+0x4/0x10
[  602.610221]  [<ffffffff81052520>] ? kthread_worker_fn+0x120/0x120
[  602.610227]  [<ffffffff813d8290>] ? gs_change+0xb/0xb
[  602.610233] INFO: task fsnotify_mark:662 blocked for more than 120 
seconds.
[  602.610237] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" 
disables this message.
[  602.610241] fsnotify_mark   D 0000000000000000     0   662      2 
0x00000000
[  602.610248]  ffff88041c31d860 0000000000000046 0000000000000000 
0000000000000000
[  602.610255]  ffffffff81593020 ffff88041c365fd8 0000000000004000 
ffff88041c364010
[  602.610262]  ffff88041c365fd8 ffff88041c31d860 0000000000000000 
0000000000000000
[  602.610268] Call Trace:
[  602.610275]  [<ffffffff81035521>] ? load_balance+0x91/0x5e0
[  602.610282]  [<ffffffff813d5595>] ? schedule_timeout+0x1c5/0x230
[  602.610289]  [<ffffffff8102f323>] ? pick_next_task_fair+0x103/0x190
[  602.610295]  [<ffffffff813d4a6d>] ? schedule+0x28d/0x910
[  602.610302]  [<ffffffff813d4562>] ? wait_for_common+0xd2/0x180
[  602.610307]  [<ffffffff810339b0>] ? try_to_wake_up+0x2c0/0x2c0
[  602.610314]  [<ffffffff81074420>] ? synchronize_rcu_bh+0x50/0x50
[  602.610320]  [<ffffffff8107446a>] ? synchronize_sched+0x4a/0x50
[  602.610326]  [<ffffffff8104f940>] ? find_ge_pid+0x40/0x40
[  602.610333]  [<ffffffff8105742b>] ? __synchronize_srcu+0x5b/0xc0
[  602.610342]  [<ffffffff810f5f63>] ? fsnotify_mark_destroy+0x83/0x150
[  602.610348]  [<ffffffff81052a20>] ? wake_up_bit+0x40/0x40
[  602.610356]  [<ffffffff810f5ee0>] ? 
fsnotify_set_mark_ignored_mask_locked+0x20/0x20
[  602.610364]  [<ffffffff810f5ee0>] ? 
fsnotify_set_mark_ignored_mask_locked+0x20/0x20
[  602.610371]  [<ffffffff810525b6>] ? kthread+0x96/0xa0
[  602.610377]  [<ffffffff813d8294>] ? kernel_thread_helper+0x4/0x10
[  602.610385]  [<ffffffff81052520>] ? kthread_worker_fn+0x120/0x120
[  602.610390]  [<ffffffff813d8290>] ? gs_change+0xb/0xb
[  644.689408] 
=============================================================================
[  644.689416] BUG kmalloc-512: Object padding overwritten
[  644.689420] 
-----------------------------------------------------------------------------
[  644.689423]
[  644.689428] INFO: 0xffff88041411f476-0xffff88041411f477. First byte 
0x0 instead of 0x5a
[  644.689444] INFO: Allocated in load_elf_binary+0xa6c/0x1c00 age=18648 
cpu=3 pid=5915
[  644.689454] INFO: Freed in load_elf_binary+0xab8/0x1c00 age=18648 
cpu=3 pid=5915
[  644.689461] INFO: Slab 0xffffea000e463e20 objects=28 used=10 
fp=0xffff88041411f230 flags=0x80000000000040c1
[  644.689467] INFO: Object 0xffff88041411f230 @offset=12848 
fp=0xffff88041411eda0
[  644.689470]
[  644.689474] Bytes b4 0xffff88041411f220:  22 6f 00 00 01 00 00 00 5a 
5a 5a 5a 5a 5a 5a 5a "o......ZZZZZZZZ
[  644.689490]   Object 0xffff88041411f230:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  644.689505]   Object 0xffff88041411f240:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  644.689519]   Object 0xffff88041411f250:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  644.689533]   Object 0xffff88041411f260:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  644.689547]   Object 0xffff88041411f270:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  644.689562]   Object 0xffff88041411f280:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  644.689576]   Object 0xffff88041411f290:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  644.689590]   Object 0xffff88041411f2a0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  644.689604]   Object 0xffff88041411f2b0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  644.689618]   Object 0xffff88041411f2c0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  644.689632]   Object 0xffff88041411f2d0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  644.689647]   Object 0xffff88041411f2e0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  644.689661]   Object 0xffff88041411f2f0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  644.689675]   Object 0xffff88041411f300:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  644.689689]   Object 0xffff88041411f310:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  644.689703]   Object 0xffff88041411f320:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  644.689717]   Object 0xffff88041411f330:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  644.689732]   Object 0xffff88041411f340:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  644.689746]   Object 0xffff88041411f350:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  644.689760]   Object 0xffff88041411f360:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  644.689774]   Object 0xffff88041411f370:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  644.689788]   Object 0xffff88041411f380:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  644.689802]   Object 0xffff88041411f390:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  644.689817]   Object 0xffff88041411f3a0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  644.689831]   Object 0xffff88041411f3b0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  644.689845]   Object 0xffff88041411f3c0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  644.689859]   Object 0xffff88041411f3d0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  644.689873]   Object 0xffff88041411f3e0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  644.689888]   Object 0xffff88041411f3f0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  644.689902]   Object 0xffff88041411f400:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  644.689916]   Object 0xffff88041411f410:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  644.689930]   Object 0xffff88041411f420:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b a5 kkkkkkkkkkkkkkkAJPY
[  644.689945]  Redzone 0xffff88041411f430:  bb bb bb bb bb bb bb bb 
                      A>>A>>A>>A>>A>>A>>A>>A>>
[  644.689957]  Padding 0xffff88041411f470:  5a 5a 5a 5a 5a 5a 00 00 
                      ZZZZZZ..
[  644.689974] Pid: 6597, comm: awk Not tainted 2.6.39 #2
[  644.689978] Call Trace:
[  644.689989]  [<ffffffff810b7ccd>] ? check_bytes_and_report+0x10d/0x150
[  644.689998]  [<ffffffff8110773c>] ? load_elf_binary+0xa6c/0x1c00
[  644.690005]  [<ffffffff810b7db9>] ? check_object+0xa9/0x260
[  644.690012]  [<ffffffff8110773c>] ? load_elf_binary+0xa6c/0x1c00
[  644.690019]  [<ffffffff810b86f4>] ? alloc_debug_processing+0x104/0x190
[  644.690026]  [<ffffffff810b9ac2>] ? T.912+0x272/0x2d0
[  644.690032]  [<ffffffff810ba59d>] ? __kmalloc+0x10d/0x160
[  644.690039]  [<ffffffff8110773c>] ? load_elf_binary+0xa6c/0x1c00
[  644.690047]  [<ffffffff8109bd9d>] ? __get_user_pages+0x17d/0x530
[  644.690055]  [<ffffffff810c9556>] ? get_arg_page+0x56/0x100
[  644.690062]  [<ffffffff810c8070>] ? search_binary_handler+0x90/0x240
[  644.690069]  [<ffffffff810c9e9f>] ? do_execve+0x22f/0x2f0
[  644.690076]  [<ffffffff810094a6>] ? sys_execve+0x36/0x60
[  644.690085]  [<ffffffff813d78dc>] ? stub_execve+0x6c/0xc0
[  644.690092] FIX kmalloc-512: Restoring 
0xffff88041411f476-0xffff88041411f477=0x5a
[  644.690095]
[  704.979481] 
=============================================================================
[  704.979488] BUG kmalloc-512: Object padding overwritten
[  704.979492] 
-----------------------------------------------------------------------------
[  704.979496]
[  704.979501] INFO: 0xffff8804168c1fee-0xffff8804168c1fef. First byte 
0x0 instead of 0x5a
[  704.979517] INFO: Allocated in load_elf_binary+0xa6c/0x1c00 age=24184 
cpu=0 pid=5999
[  704.979527] INFO: Freed in load_elf_binary+0xab8/0x1c00 age=24184 
cpu=0 pid=5999
[  704.979534] INFO: Slab 0xffffea000e4eea00 objects=28 used=10 
fp=0xffff8804168c1da8 flags=0x80000000000040c1
[  704.979540] INFO: Object 0xffff8804168c1da8 @offset=7592 
fp=0xffff8804168c2910
[  704.979543]
[  704.979547] Bytes b4 0xffff8804168c1d98:  93 98 ff ff 00 00 00 00 5a 
5a 5a 5a 5a 5a 5a 5a ..A?A?....ZZZZZZZZ
[  704.979563]   Object 0xffff8804168c1da8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  704.979578]   Object 0xffff8804168c1db8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  704.979592]   Object 0xffff8804168c1dc8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  704.979606]   Object 0xffff8804168c1dd8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  704.979620]   Object 0xffff8804168c1de8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  704.979634]   Object 0xffff8804168c1df8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  704.979648]   Object 0xffff8804168c1e08:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  704.979662]   Object 0xffff8804168c1e18:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  704.979676]   Object 0xffff8804168c1e28:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  704.979690]   Object 0xffff8804168c1e38:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  704.979704]   Object 0xffff8804168c1e48:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  704.979719]   Object 0xffff8804168c1e58:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  704.979733]   Object 0xffff8804168c1e68:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  704.979747]   Object 0xffff8804168c1e78:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  704.979761]   Object 0xffff8804168c1e88:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  704.979775]   Object 0xffff8804168c1e98:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  704.979789]   Object 0xffff8804168c1ea8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  704.979803]   Object 0xffff8804168c1eb8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  704.979817]   Object 0xffff8804168c1ec8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  704.979831]   Object 0xffff8804168c1ed8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  704.979845]   Object 0xffff8804168c1ee8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  704.979859]   Object 0xffff8804168c1ef8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  704.979873]   Object 0xffff8804168c1f08:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  704.979888]   Object 0xffff8804168c1f18:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  704.979902]   Object 0xffff8804168c1f28:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  704.979916]   Object 0xffff8804168c1f38:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  704.979930]   Object 0xffff8804168c1f48:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  704.979944]   Object 0xffff8804168c1f58:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  704.979958]   Object 0xffff8804168c1f68:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  704.979972]   Object 0xffff8804168c1f78:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  704.979986]   Object 0xffff8804168c1f88:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  704.980000]   Object 0xffff8804168c1f98:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b a5 kkkkkkkkkkkkkkkAJPY
[  704.980015]  Redzone 0xffff8804168c1fa8:  bb bb bb bb bb bb bb bb 
                      A>>A>>A>>A>>A>>A>>A>>A>>
[  704.980028]  Padding 0xffff8804168c1fe8:  5a 5a 5a 5a 5a 5a 00 00 
                      ZZZZZZ..
[  704.980044] Pid: 6812, comm: get-rrdtool-dat Not tainted 2.6.39 #2
[  704.980048] Call Trace:
[  704.980060]  [<ffffffff810b7ccd>] ? check_bytes_and_report+0x10d/0x150
[  704.980069]  [<ffffffff81106e49>] ? load_elf_binary+0x179/0x1c00
[  704.980075]  [<ffffffff810b7db9>] ? check_object+0xa9/0x260
[  704.980083]  [<ffffffff81106e49>] ? load_elf_binary+0x179/0x1c00
[  704.980090]  [<ffffffff810b86f4>] ? alloc_debug_processing+0x104/0x190
[  704.980096]  [<ffffffff810b9ac2>] ? T.912+0x272/0x2d0
[  704.980102]  [<ffffffff810ba59d>] ? __kmalloc+0x10d/0x160
[  704.980109]  [<ffffffff81106e49>] ? load_elf_binary+0x179/0x1c00
[  704.980117]  [<ffffffff8109b782>] ? __pte_alloc+0x42/0x130
[  704.980123]  [<ffffffff8109b5a5>] ? follow_page+0x325/0x440
[  704.980130]  [<ffffffff8109b5a5>] ? follow_page+0x325/0x440
[  704.980137]  [<ffffffff8109bd9d>] ? __get_user_pages+0x17d/0x530
[  704.980145]  [<ffffffff810c9556>] ? get_arg_page+0x56/0x100
[  704.980152]  [<ffffffff810c8070>] ? search_binary_handler+0x90/0x240
[  704.980158]  [<ffffffff810c9e9f>] ? do_execve+0x22f/0x2f0
[  704.980166]  [<ffffffff810094a6>] ? sys_execve+0x36/0x60
[  704.980175]  [<ffffffff813d78dc>] ? stub_execve+0x6c/0xc0
[  704.980182] FIX kmalloc-512: Restoring 
0xffff8804168c1fee-0xffff8804168c1fef=0x5a
[  704.980185]
[  722.610058] INFO: task ksmd:552 blocked for more than 120 seconds.
[  722.610063] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" 
disables this message.
[  722.610069] ksmd            D 0000000000000000     0   552      2 
0x00000000
[  722.610078]  ffff88041db68ca0 0000000000000046 000212d01fc11d00 
ffff880400000000
[  722.610085]  ffffffff81593020 ffff88041c1a9fd8 0000000000004000 
ffff88041c1a8010
[  722.610092]  ffff88041c1a9fd8 ffff88041db68ca0 0000000000000000 
0000000200000002
[  722.610099] Call Trace:
[  722.610115]  [<ffffffff813d5595>] ? schedule_timeout+0x1c5/0x230
[  722.610127]  [<ffffffff8102f69c>] ? enqueue_task_fair+0x14c/0x190
[  722.610135]  [<ffffffff8102db27>] ? task_rq_lock+0x47/0x90
[  722.610142]  [<ffffffff813d4562>] ? wait_for_common+0xd2/0x180
[  722.610148]  [<ffffffff810339b0>] ? try_to_wake_up+0x2c0/0x2c0
[  722.610156]  [<ffffffff8104cb94>] ? flush_work+0x24/0x30
[  722.610162]  [<ffffffff8104be10>] ? do_work_for_cpu+0x20/0x20
[  722.610168]  [<ffffffff8104e0ab>] ? schedule_on_each_cpu+0xab/0xe0
[  722.610175]  [<ffffffff810b5c05>] ? ksm_scan_thread+0x7f5/0xc20
[  722.610182]  [<ffffffff81052a20>] ? wake_up_bit+0x40/0x40
[  722.610188]  [<ffffffff810b5410>] ? 
try_to_merge_with_ksm_page+0x570/0x570
[  722.610194]  [<ffffffff810b5410>] ? 
try_to_merge_with_ksm_page+0x570/0x570
[  722.610201]  [<ffffffff810525b6>] ? kthread+0x96/0xa0
[  722.610208]  [<ffffffff813d8294>] ? kernel_thread_helper+0x4/0x10
[  722.610215]  [<ffffffff81052520>] ? kthread_worker_fn+0x120/0x120
[  722.610221]  [<ffffffff813d8290>] ? gs_change+0xb/0xb
[  722.610226] INFO: task fsnotify_mark:662 blocked for more than 120 
seconds.
[  722.610230] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" 
disables this message.
[  722.610235] fsnotify_mark   D 0000000000000000     0   662      2 
0x00000000
[  722.610241]  ffff88041c31d860 0000000000000046 0000000000000000 
0000000000000000
[  722.610248]  ffffffff81593020 ffff88041c365fd8 0000000000004000 
ffff88041c364010
[  722.610255]  ffff88041c365fd8 ffff88041c31d860 0000000000000000 
0000000000000000
[  722.610261] Call Trace:
[  722.610268]  [<ffffffff81035521>] ? load_balance+0x91/0x5e0
[  722.610275]  [<ffffffff813d5595>] ? schedule_timeout+0x1c5/0x230
[  722.610282]  [<ffffffff8102f323>] ? pick_next_task_fair+0x103/0x190
[  722.610289]  [<ffffffff813d4a6d>] ? schedule+0x28d/0x910
[  722.610295]  [<ffffffff813d4562>] ? wait_for_common+0xd2/0x180
[  722.610301]  [<ffffffff810339b0>] ? try_to_wake_up+0x2c0/0x2c0
[  722.610308]  [<ffffffff81074420>] ? synchronize_rcu_bh+0x50/0x50
[  722.610313]  [<ffffffff8107446a>] ? synchronize_sched+0x4a/0x50
[  722.610319]  [<ffffffff8104f940>] ? find_ge_pid+0x40/0x40
[  722.610326]  [<ffffffff8105742b>] ? __synchronize_srcu+0x5b/0xc0
[  722.610335]  [<ffffffff810f5f63>] ? fsnotify_mark_destroy+0x83/0x150
[  722.610342]  [<ffffffff81052a20>] ? wake_up_bit+0x40/0x40
[  722.610350]  [<ffffffff810f5ee0>] ? 
fsnotify_set_mark_ignored_mask_locked+0x20/0x20
[  722.610358]  [<ffffffff810f5ee0>] ? 
fsnotify_set_mark_ignored_mask_locked+0x20/0x20
[  722.610365]  [<ffffffff810525b6>] ? kthread+0x96/0xa0
[  722.610371]  [<ffffffff813d8294>] ? kernel_thread_helper+0x4/0x10
[  722.610378]  [<ffffffff81052520>] ? kthread_worker_fn+0x120/0x120
[  722.610384]  [<ffffffff813d8290>] ? gs_change+0xb/0xb
[  722.610392] INFO: task jbd2/md0-8:2528 blocked for more than 120 seconds.
[  722.610396] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" 
disables this message.
[  722.610400] jbd2/md0-8      D 0000000000000005     0  2528      2 
0x00000000
[  722.610407]  ffff88041d91c570 0000000000000046 ffff88041baeea28 
ffffea0000000000
[  722.610413]  ffff88041d91a5e0 ffff880419f9bfd8 0000000000004000 
ffff880419f9a010
[  722.610420]  ffff880419f9bfd8 ffff88041d91c570 ffff88041baeea28 
ffffffff810b8495
[  722.610426] Call Trace:
[  722.610432]  [<ffffffff810b8495>] ? init_object+0x85/0xa0
[  722.610438]  [<ffffffff810b8916>] ? free_debug_processing+0x196/0x250
[  722.610447]  [<ffffffff8105adae>] ? ktime_get_ts+0x6e/0xf0
[  722.610455]  [<ffffffff810810f0>] ? __lock_page+0x70/0x70
[  722.610461]  [<ffffffff813d5174>] ? io_schedule+0x84/0xd0
[  722.610469]  [<ffffffff811d4403>] ? 
radix_tree_gang_lookup_tag_slot+0x93/0xf0
[  722.610476]  [<ffffffff810810f9>] ? sleep_on_page+0x9/0x10
[  722.610482]  [<ffffffff813d57df>] ? __wait_on_bit+0x4f/0x80
[  722.610489]  [<ffffffff810812eb>] ? wait_on_page_bit+0x6b/0x80
[  722.610496]  [<ffffffff81052a50>] ? autoremove_wake_function+0x30/0x30
[  722.610504]  [<ffffffff8108a458>] ? pagevec_lookup_tag+0x18/0x20
[  722.610509]  [<ffffffff81081f2a>] ? filemap_fdatawait_range+0xfa/0x180
[  722.610518]  [<ffffffff811be09f>] ? submit_bio+0x6f/0xf0
[  722.610526]  [<ffffffff81176276>] ? 
jbd2_journal_commit_transaction+0x796/0x1270
[  722.610536]  [<ffffffff81179ed1>] ? kjournald2+0xb1/0x1e0
[  722.610542]  [<ffffffff81052a20>] ? wake_up_bit+0x40/0x40
[  722.610549]  [<ffffffff81179e20>] ? commit_timeout+0x10/0x10
[  722.610556]  [<ffffffff81179e20>] ? commit_timeout+0x10/0x10
[  722.610563]  [<ffffffff810525b6>] ? kthread+0x96/0xa0
[  722.610569]  [<ffffffff813d8294>] ? kernel_thread_helper+0x4/0x10
[  722.610576]  [<ffffffff81052520>] ? kthread_worker_fn+0x120/0x120
[  722.610582]  [<ffffffff813d8290>] ? gs_change+0xb/0xb
[  722.610591] INFO: task nfsd:4326 blocked for more than 120 seconds.
[  722.610594] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" 
disables this message.
[  722.610598] nfsd            D 0000000000000000     0  4326      2 
0x00000000
[  722.610605]  ffff88041b604bc0 0000000000000046 0000000000000016 
ffffffff00000000
[  722.610611]  ffff88041d8fa5e0 ffff88041c527fd8 0000000000004000 
ffff88041c526010
[  722.610617]  ffff88041c527fd8 ffff88041b604bc0 000000000000009c 
ffff880000000000
[  722.610623] Call Trace:
[  722.610630]  [<ffffffff8101e845>] ? amd_flush_garts+0x105/0x140
[  722.610637]  [<ffffffff8101fa80>] ? gart_map_sg+0x480/0x480
[  722.610643]  [<ffffffff8101f5d3>] ? flush_gart+0x23/0x50
[  722.610650]  [<ffffffff81080f98>] ? find_get_page+0x18/0x90
[  722.610657]  [<ffffffff81174b95>] ? do_get_write_access+0x265/0x4a0
[  722.610665]  [<ffffffff81052a50>] ? autoremove_wake_function+0x30/0x30
[  722.610672]  [<ffffffff81174ef9>] ? 
jbd2_journal_get_write_access+0x29/0x50
[  722.610680]  [<ffffffff8115f122>] ? 
__ext4_journal_get_write_access+0x32/0x80
[  722.610689]  [<ffffffff81143908>] ? ext4_reserve_inode_write+0x78/0xa0
[  722.610696]  [<ffffffff81143970>] ? ext4_mark_inode_dirty+0x40/0x1e0
[  722.610703]  [<ffffffff81156c0b>] ? ext4_journal_start_sb+0x6b/0x160
[  722.610711]  [<ffffffff81322645>] ? dev_hard_start_xmit+0x305/0x5f0
[  722.610719]  [<ffffffff81352fd0>] ? ip_finish_output2+0x290/0x290
[  722.610726]  [<ffffffff81143c65>] ? ext4_dirty_inode+0x35/0x70
[  722.610733]  [<ffffffff810e4a08>] ? __mark_inode_dirty+0x38/0x210
[  722.610741]  [<ffffffff810d9317>] ? file_update_time+0xf7/0x180
[  722.610747]  [<ffffffff81082458>] ? __generic_file_aio_write+0x1f8/0x430
[  722.610755]  [<ffffffff81373ef9>] ? udp_sendmsg+0x3c9/0x7e0
[  722.610763]  [<ffffffff81314260>] ? sock_alloc_send_pskb+0x1d0/0x320
[  722.610769]  [<ffffffff81082703>] ? generic_file_aio_write+0x73/0xf0
[  722.610776]  [<ffffffff8113f74e>] ? ext4_file_write+0x6e/0x2b0
[  722.610783]  [<ffffffff810da8ac>] ? iget_locked+0x4c/0x140
[  722.610789]  [<ffffffff8119a590>] ? fh_compose+0x4c0/0x4c0
[  722.610795]  [<ffffffff8113f6e0>] ? ext4_llseek+0x110/0x110
[  722.610803]  [<ffffffff810c24e9>] ? do_sync_readv_writev+0xa9/0xf0
[  722.610811]  [<ffffffff810c231b>] ? rw_copy_check_uvector+0x7b/0x140
[  722.610819]  [<ffffffff810c2bcf>] ? do_readv_writev+0xdf/0x210
[  722.610825]  [<ffffffff810b8735>] ? alloc_debug_processing+0x145/0x190
[  722.610831]  [<ffffffff810b9bc5>] ? kmem_cache_alloc+0xa5/0xb0
[  722.610838]  [<ffffffff8113f463>] ? ext4_file_open+0x63/0x180
[  722.610844]  [<ffffffff8119c21d>] ? nfsd_vfs_write+0xed/0x3a0
[  722.610851]  [<ffffffff810c1127>] ? __dentry_open+0x1f7/0x2b0
[  722.610857]  [<ffffffff8119c892>] ? nfsd_open+0xf2/0x1b0
[  722.610862]  [<ffffffff8119cd34>] ? nfsd_write+0xf4/0x110
[  722.610868]  [<ffffffff81199930>] ? nfsd_proc_write+0xb0/0x120
[  722.610876]  [<ffffffff811971c5>] ? nfsd_dispatch+0xf5/0x230
[  722.610882]  [<ffffffff813b456f>] ? svc_process+0x4af/0x820
[  722.610887]  [<ffffffff810339b0>] ? try_to_wake_up+0x2c0/0x2c0
[  722.610894]  [<ffffffff811977a0>] ? nfsd_svc+0x1b0/0x1b0
[  722.610901]  [<ffffffff8119784d>] ? nfsd+0xad/0x150
[  722.610907]  [<ffffffff810525b6>] ? kthread+0x96/0xa0
[  722.610913]  [<ffffffff813d8294>] ? kernel_thread_helper+0x4/0x10
[  722.610921]  [<ffffffff81052520>] ? kthread_worker_fn+0x120/0x120
[  722.610926]  [<ffffffff813d8290>] ? gs_change+0xb/0xb
[  765.241265] 
=============================================================================
[  765.241267] BUG kmalloc-512: Object padding overwritten
[  765.241268] 
-----------------------------------------------------------------------------
[  765.241269]
[  765.241271] INFO: 0xffff8804168c1486-0xffff8804168c1487. First byte 
0x0 instead of 0x5a
[  765.241277] INFO: Allocated in bio_kmalloc+0x2d/0x70 age=73010 cpu=5 
pid=2495
[  765.241281] INFO: Freed in r10buf_pool_free+0x71/0x90 age=72999 cpu=5 
pid=1728
[  765.241283] INFO: Slab 0xffffea000e4eea00 objects=28 used=13 
fp=0xffff8804168c1240 flags=0x80000000000040c1
[  765.241284] INFO: Object 0xffff8804168c1240 @offset=4672 
fp=0xffff8804168c0ff8
[  765.241285]
[  765.241286] Bytes b4 0xffff8804168c1230:  95 98 ff ff 00 00 00 00 5a 
5a 5a 5a 5a 5a 5a 5a ..A?A?....ZZZZZZZZ
[  765.241291]   Object 0xffff8804168c1240:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  765.241294]   Object 0xffff8804168c1250:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  765.241298]   Object 0xffff8804168c1260:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  765.241301]   Object 0xffff8804168c1270:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  765.241305]   Object 0xffff8804168c1280:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  765.241308]   Object 0xffff8804168c1290:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  765.241312]   Object 0xffff8804168c12a0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  765.241315]   Object 0xffff8804168c12b0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  765.241319]   Object 0xffff8804168c12c0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  765.241322]   Object 0xffff8804168c12d0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  765.241326]   Object 0xffff8804168c12e0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  765.241329]   Object 0xffff8804168c12f0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  765.241332]   Object 0xffff8804168c1300:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  765.241336]   Object 0xffff8804168c1310:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  765.241339]   Object 0xffff8804168c1320:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  765.241343]   Object 0xffff8804168c1330:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  765.241346]   Object 0xffff8804168c1340:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  765.241350]   Object 0xffff8804168c1350:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  765.241353]   Object 0xffff8804168c1360:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  765.241357]   Object 0xffff8804168c1370:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  765.241360]   Object 0xffff8804168c1380:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  765.241364]   Object 0xffff8804168c1390:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  765.241367]   Object 0xffff8804168c13a0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  765.241371]   Object 0xffff8804168c13b0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  765.241374]   Object 0xffff8804168c13c0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  765.241378]   Object 0xffff8804168c13d0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  765.241381]   Object 0xffff8804168c13e0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  765.241385]   Object 0xffff8804168c13f0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  765.241388]   Object 0xffff8804168c1400:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  765.241392]   Object 0xffff8804168c1410:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  765.241395]   Object 0xffff8804168c1420:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  765.241399]   Object 0xffff8804168c1430:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b a5 kkkkkkkkkkkkkkkAJPY
[  765.241402]  Redzone 0xffff8804168c1440:  bb bb bb bb bb bb bb bb 
                      A>>A>>A>>A>>A>>A>>A>>A>>
[  765.241406]  Padding 0xffff8804168c1480:  5a 5a 5a 5a 5a 5a 00 00 
                      ZZZZZZ..
[  765.241410] Pid: 7147, comm: apcaccess Not tainted 2.6.39 #2
[  765.241411] Call Trace:
[  765.241416]  [<ffffffff810b7ccd>] ? check_bytes_and_report+0x10d/0x150
[  765.241419]  [<ffffffff8110773c>] ? load_elf_binary+0xa6c/0x1c00
[  765.241421]  [<ffffffff810b7db9>] ? check_object+0xa9/0x260
[  765.241423]  [<ffffffff8110773c>] ? load_elf_binary+0xa6c/0x1c00
[  765.241424]  [<ffffffff810b86f4>] ? alloc_debug_processing+0x104/0x190
[  765.241426]  [<ffffffff810b9ac2>] ? T.912+0x272/0x2d0
[  765.241428]  [<ffffffff810ba59d>] ? __kmalloc+0x10d/0x160
[  765.241429]  [<ffffffff8110773c>] ? load_elf_binary+0xa6c/0x1c00
[  765.241432]  [<ffffffff8109bd9d>] ? __get_user_pages+0x17d/0x530
[  765.241434]  [<ffffffff810c9556>] ? get_arg_page+0x56/0x100
[  765.241437]  [<ffffffff811da4ad>] ? strnlen_user+0x2d/0x80
[  765.241439]  [<ffffffff810c8070>] ? search_binary_handler+0x90/0x240
[  765.241440]  [<ffffffff810c9e9f>] ? do_execve+0x22f/0x2f0
[  765.241443]  [<ffffffff810094a6>] ? sys_execve+0x36/0x60
[  765.241446]  [<ffffffff813d78dc>] ? stub_execve+0x6c/0xc0
[  765.241448] FIX kmalloc-512: Restoring 
0xffff8804168c1486-0xffff8804168c1487=0x5a
[  765.241449]
[  789.536436] 
=============================================================================
[  789.536444] BUG kmalloc-2048: Object padding overwritten
[  789.536448] 
-----------------------------------------------------------------------------
[  789.536451]
[  789.536456] INFO: 0xffff8802c8b1a11e-0xffff8802c8b1a11f. First byte 
0x0 instead of 0x5a
[  789.536471] INFO: Allocated in sk_stream_alloc_skb+0x3a/0x110 age=63 
cpu=3 pid=5169
[  789.536482] INFO: Freed in __kfree_skb+0x11/0x90 age=63 cpu=5 pid=0
[  789.536488] INFO: Slab 0xffffea0009be6d40 objects=15 used=4 
fp=0xffff8802c8b198d8 flags=0x80000000000040c1
[  789.536494] INFO: Object 0xffff8802c8b198d8 @offset=6360 
fp=0xffff8802c8b1a968
[  789.536497]
[  789.536501] Bytes b4 0xffff8802c8b198c8:  cc be 00 00 01 00 00 00 5a 
5a 5a 5a 5a 5a 5a 5a A?A,......ZZZZZZZZ
[  789.536517]   Object 0xffff8802c8b198d8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.536532]   Object 0xffff8802c8b198e8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.536546]   Object 0xffff8802c8b198f8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.536560]   Object 0xffff8802c8b19908:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.536575]   Object 0xffff8802c8b19918:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.536589]   Object 0xffff8802c8b19928:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.536603]   Object 0xffff8802c8b19938:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.536617]   Object 0xffff8802c8b19948:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.536631]   Object 0xffff8802c8b19958:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.536645]   Object 0xffff8802c8b19968:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.536660]   Object 0xffff8802c8b19978:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.536674]   Object 0xffff8802c8b19988:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.536688]   Object 0xffff8802c8b19998:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.536702]   Object 0xffff8802c8b199a8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.536716]   Object 0xffff8802c8b199b8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.536730]   Object 0xffff8802c8b199c8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.536745]   Object 0xffff8802c8b199d8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.536759]   Object 0xffff8802c8b199e8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.536773]   Object 0xffff8802c8b199f8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.536787]   Object 0xffff8802c8b19a08:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.536801]   Object 0xffff8802c8b19a18:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.536816]   Object 0xffff8802c8b19a28:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.536830]   Object 0xffff8802c8b19a38:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.536844]   Object 0xffff8802c8b19a48:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.536858]   Object 0xffff8802c8b19a58:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.536872]   Object 0xffff8802c8b19a68:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.536887]   Object 0xffff8802c8b19a78:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.536901]   Object 0xffff8802c8b19a88:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.536915]   Object 0xffff8802c8b19a98:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.536929]   Object 0xffff8802c8b19aa8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.536943]   Object 0xffff8802c8b19ab8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.536958]   Object 0xffff8802c8b19ac8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.536972]   Object 0xffff8802c8b19ad8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.536986]   Object 0xffff8802c8b19ae8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537000]   Object 0xffff8802c8b19af8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537014]   Object 0xffff8802c8b19b08:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537029]   Object 0xffff8802c8b19b18:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537043]   Object 0xffff8802c8b19b28:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537057]   Object 0xffff8802c8b19b38:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537071]   Object 0xffff8802c8b19b48:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537085]   Object 0xffff8802c8b19b58:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537100]   Object 0xffff8802c8b19b68:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537114]   Object 0xffff8802c8b19b78:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537128]   Object 0xffff8802c8b19b88:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537143]   Object 0xffff8802c8b19b98:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537158]   Object 0xffff8802c8b19ba8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537172]   Object 0xffff8802c8b19bb8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537187]   Object 0xffff8802c8b19bc8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537201]   Object 0xffff8802c8b19bd8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537215]   Object 0xffff8802c8b19be8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537229]   Object 0xffff8802c8b19bf8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537244]   Object 0xffff8802c8b19c08:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537258]   Object 0xffff8802c8b19c18:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537272]   Object 0xffff8802c8b19c28:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537286]   Object 0xffff8802c8b19c38:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537301]   Object 0xffff8802c8b19c48:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537315]   Object 0xffff8802c8b19c58:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537329]   Object 0xffff8802c8b19c68:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537343]   Object 0xffff8802c8b19c78:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537357]   Object 0xffff8802c8b19c88:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537372]   Object 0xffff8802c8b19c98:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537386]   Object 0xffff8802c8b19ca8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537400]   Object 0xffff8802c8b19cb8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537414]   Object 0xffff8802c8b19cc8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537428]   Object 0xffff8802c8b19cd8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537443]   Object 0xffff8802c8b19ce8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537457]   Object 0xffff8802c8b19cf8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537471]   Object 0xffff8802c8b19d08:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537485]   Object 0xffff8802c8b19d18:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537500]   Object 0xffff8802c8b19d28:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537514]   Object 0xffff8802c8b19d38:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537528]   Object 0xffff8802c8b19d48:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537542]   Object 0xffff8802c8b19d58:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537557]   Object 0xffff8802c8b19d68:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537571]   Object 0xffff8802c8b19d78:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537585]   Object 0xffff8802c8b19d88:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537599]   Object 0xffff8802c8b19d98:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537613]   Object 0xffff8802c8b19da8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537628]   Object 0xffff8802c8b19db8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537642]   Object 0xffff8802c8b19dc8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537656]   Object 0xffff8802c8b19dd8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537670]   Object 0xffff8802c8b19de8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537685]   Object 0xffff8802c8b19df8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537699]   Object 0xffff8802c8b19e08:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537713]   Object 0xffff8802c8b19e18:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537727]   Object 0xffff8802c8b19e28:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537742]   Object 0xffff8802c8b19e38:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537756]   Object 0xffff8802c8b19e48:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537770]   Object 0xffff8802c8b19e58:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537784]   Object 0xffff8802c8b19e68:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537799]   Object 0xffff8802c8b19e78:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537813]   Object 0xffff8802c8b19e88:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537827]   Object 0xffff8802c8b19e98:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537841]   Object 0xffff8802c8b19ea8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537856]   Object 0xffff8802c8b19eb8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537870]   Object 0xffff8802c8b19ec8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537884]   Object 0xffff8802c8b19ed8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537898]   Object 0xffff8802c8b19ee8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537912]   Object 0xffff8802c8b19ef8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537927]   Object 0xffff8802c8b19f08:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537941]   Object 0xffff8802c8b19f18:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537955]   Object 0xffff8802c8b19f28:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537969]   Object 0xffff8802c8b19f38:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537984]   Object 0xffff8802c8b19f48:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537998]   Object 0xffff8802c8b19f58:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.538012]   Object 0xffff8802c8b19f68:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.538026]   Object 0xffff8802c8b19f78:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.538041]   Object 0xffff8802c8b19f88:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.538055]   Object 0xffff8802c8b19f98:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.538069]   Object 0xffff8802c8b19fa8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.538083]   Object 0xffff8802c8b19fb8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.538097]   Object 0xffff8802c8b19fc8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.538112]   Object 0xffff8802c8b19fd8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.538126]   Object 0xffff8802c8b19fe8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.538140]   Object 0xffff8802c8b19ff8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.538154]   Object 0xffff8802c8b1a008:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.538169]   Object 0xffff8802c8b1a018:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.538183]   Object 0xffff8802c8b1a028:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.538197]   Object 0xffff8802c8b1a038:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.538211]   Object 0xffff8802c8b1a048:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.538226]   Object 0xffff8802c8b1a058:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.538240]   Object 0xffff8802c8b1a068:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.538254]   Object 0xffff8802c8b1a078:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.538268]   Object 0xffff8802c8b1a088:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.538283]   Object 0xffff8802c8b1a098:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.538297]   Object 0xffff8802c8b1a0a8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.538311]   Object 0xffff8802c8b1a0b8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.538325]   Object 0xffff8802c8b1a0c8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b a5 kkkkkkkkkkkkkkkAJPY
[  789.538340]  Redzone 0xffff8802c8b1a0d8:  bb bb bb bb bb bb bb bb 
                      A>>A>>A>>A>>A>>A>>A>>A>>
[  789.538353]  Padding 0xffff8802c8b1a118:  5a 5a 5a 5a 5a 5a 00 00 
                      ZZZZZZ..
[  789.538369] Pid: 5247, comm: qemu Not tainted 2.6.39 #2
[  789.538373] Call Trace:
[  789.538385]  [<ffffffff810b7ccd>] ? check_bytes_and_report+0x10d/0x150
[  789.538394]  [<ffffffff81314260>] ? sock_alloc_send_pskb+0x1d0/0x320
[  789.538400]  [<ffffffff810b7db9>] ? check_object+0xa9/0x260
[  789.538407]  [<ffffffff81314260>] ? sock_alloc_send_pskb+0x1d0/0x320
[  789.538414]  [<ffffffff810b86f4>] ? alloc_debug_processing+0x104/0x190
[  789.538420]  [<ffffffff810b9ac2>] ? T.912+0x272/0x2d0
[  789.538427]  [<ffffffff810bb70d>] ? __kmalloc_track_caller+0x10d/0x160
[  789.538434]  [<ffffffff813198c2>] ? __alloc_skb+0x72/0x160
[  789.538441]  [<ffffffff81314260>] ? sock_alloc_send_pskb+0x1d0/0x320
[  789.538450]  [<ffffffff8105af7b>] ? getnstimeofday+0x4b/0xd0
[  789.538457]  [<ffffffff81320d32>] ? netif_rx+0xb2/0x120
[  789.538465]  [<ffffffff812b393e>] ? tun_chr_aio_write+0x18e/0x4e0
[  789.538472]  [<ffffffff812b37b0>] ? tun_sendmsg+0x4d0/0x4d0
[  789.538481]  [<ffffffff810c24e9>] ? do_sync_readv_writev+0xa9/0xf0
[  789.538489]  [<ffffffff810c231b>] ? rw_copy_check_uvector+0x7b/0x140
[  789.538496]  [<ffffffff810c2bcf>] ? do_readv_writev+0xdf/0x210
[  789.538504]  [<ffffffff810c2e7e>] ? sys_writev+0x4e/0xc0
[  789.538513]  [<ffffffff813d753b>] ? system_call_fastpath+0x16/0x1b
[  789.538519] FIX kmalloc-2048: Restoring 
0xffff8802c8b1a11e-0xffff8802c8b1a11f=0x5a
[  789.538522]
[  825.601291] 
=============================================================================
[  825.601298] BUG kmalloc-512: Object padding overwritten
[  825.601302] 
-----------------------------------------------------------------------------
[  825.601306]
[  825.601311] INFO: 0xffff88041411efe6-0xffff88041411efe7. First byte 
0x0 instead of 0x5a
[  825.601327] INFO: Allocated in sock_alloc_send_pskb+0x1d0/0x320 
age=36742 cpu=3 pid=4846
[  825.601338] INFO: Freed in __kfree_skb+0x11/0x90 age=36742 cpu=3 pid=3781
[  825.601344] INFO: Slab 0xffffea000e463e20 objects=28 used=11 
fp=0xffff88041411eda0 flags=0x80000000000040c1
[  825.601350] INFO: Object 0xffff88041411eda0 @offset=11680 
fp=0xffff88041411cdb0
[  825.601353]
[  825.601357] Bytes b4 0xffff88041411ed90:  e2 96 ff ff 00 00 00 00 5a 
5a 5a 5a 5a 5a 5a 5a Ac.A?A?....ZZZZZZZZ
[  825.601373]   Object 0xffff88041411eda0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  825.601388]   Object 0xffff88041411edb0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  825.601402]   Object 0xffff88041411edc0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  825.601416]   Object 0xffff88041411edd0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  825.601430]   Object 0xffff88041411ede0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  825.601444]   Object 0xffff88041411edf0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  825.601459]   Object 0xffff88041411ee00:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  825.601473]   Object 0xffff88041411ee10:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  825.601487]   Object 0xffff88041411ee20:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  825.601501]   Object 0xffff88041411ee30:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  825.601515]   Object 0xffff88041411ee40:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  825.601529]   Object 0xffff88041411ee50:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  825.601543]   Object 0xffff88041411ee60:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  825.601557]   Object 0xffff88041411ee70:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  825.601571]   Object 0xffff88041411ee80:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  825.601586]   Object 0xffff88041411ee90:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  825.601600]   Object 0xffff88041411eea0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  825.601614]   Object 0xffff88041411eeb0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  825.601628]   Object 0xffff88041411eec0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  825.601642]   Object 0xffff88041411eed0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  825.601656]   Object 0xffff88041411eee0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  825.601670]   Object 0xffff88041411eef0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  825.601684]   Object 0xffff88041411ef00:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  825.601699]   Object 0xffff88041411ef10:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  825.601713]   Object 0xffff88041411ef20:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  825.601727]   Object 0xffff88041411ef30:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  825.601741]   Object 0xffff88041411ef40:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  825.601755]   Object 0xffff88041411ef50:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  825.601769]   Object 0xffff88041411ef60:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  825.601783]   Object 0xffff88041411ef70:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  825.601797]   Object 0xffff88041411ef80:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  825.601812]   Object 0xffff88041411ef90:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b a5 kkkkkkkkkkkkkkkAJPY
[  825.601826]  Redzone 0xffff88041411efa0:  bb bb bb bb bb bb bb bb 
                      A>>A>>A>>A>>A>>A>>A>>A>>
[  825.601839]  Padding 0xffff88041411efe0:  5a 5a 5a 5a 5a 5a 00 00 
                      ZZZZZZ..
[  825.601855] Pid: 7353, comm: apcaccess Not tainted 2.6.39 #2
[  825.601859] Call Trace:
[  825.601871]  [<ffffffff810b7ccd>] ? check_bytes_and_report+0x10d/0x150
[  825.601881]  [<ffffffff8110773c>] ? load_elf_binary+0xa6c/0x1c00
[  825.601887]  [<ffffffff810b7db9>] ? check_object+0xa9/0x260
[  825.601895]  [<ffffffff8110773c>] ? load_elf_binary+0xa6c/0x1c00
[  825.601902]  [<ffffffff810b86f4>] ? alloc_debug_processing+0x104/0x190
[  825.601908]  [<ffffffff810b9ac2>] ? T.912+0x272/0x2d0
[  825.601914]  [<ffffffff810ba59d>] ? __kmalloc+0x10d/0x160
[  825.601922]  [<ffffffff8110773c>] ? load_elf_binary+0xa6c/0x1c00
[  825.601930]  [<ffffffff8109bd9d>] ? __get_user_pages+0x17d/0x530
[  825.601938]  [<ffffffff810c9556>] ? get_arg_page+0x56/0x100
[  825.601945]  [<ffffffff810c8070>] ? search_binary_handler+0x90/0x240
[  825.601951]  [<ffffffff810c9e9f>] ? do_execve+0x22f/0x2f0
[  825.601959]  [<ffffffff810094a6>] ? sys_execve+0x36/0x60
[  825.601968]  [<ffffffff813d78dc>] ? stub_execve+0x6c/0xc0
[  825.601975] FIX kmalloc-512: Restoring 
0xffff88041411efe6-0xffff88041411efe7=0x5a
[  825.601978]
[  842.610057] INFO: task ksmd:552 blocked for more than 120 seconds.
[  842.610063] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" 
disables this message.
[  842.610069] ksmd            D 0000000000000000     0   552      2 
0x00000000
[  842.610078]  ffff88041db68ca0 0000000000000046 000212d01fc11d00 
ffff880400000000
[  842.610086]  ffffffff81593020 ffff88041c1a9fd8 0000000000004000 
ffff88041c1a8010
[  842.610093]  ffff88041c1a9fd8 ffff88041db68ca0 0000000000000000 
0000000200000002
[  842.610100] Call Trace:
[  842.610116]  [<ffffffff813d5595>] ? schedule_timeout+0x1c5/0x230
[  842.610128]  [<ffffffff8102f69c>] ? enqueue_task_fair+0x14c/0x190
[  842.610137]  [<ffffffff8102db27>] ? task_rq_lock+0x47/0x90
[  842.610143]  [<ffffffff813d4562>] ? wait_for_common+0xd2/0x180
[  842.610150]  [<ffffffff810339b0>] ? try_to_wake_up+0x2c0/0x2c0
[  842.610157]  [<ffffffff8104cb94>] ? flush_work+0x24/0x30
[  842.610163]  [<ffffffff8104be10>] ? do_work_for_cpu+0x20/0x20
[  842.610170]  [<ffffffff8104e0ab>] ? schedule_on_each_cpu+0xab/0xe0
[  842.610177]  [<ffffffff810b5c05>] ? ksm_scan_thread+0x7f5/0xc20
[  842.610184]  [<ffffffff81052a20>] ? wake_up_bit+0x40/0x40
[  842.610190]  [<ffffffff810b5410>] ? 
try_to_merge_with_ksm_page+0x570/0x570
[  842.610196]  [<ffffffff810b5410>] ? 
try_to_merge_with_ksm_page+0x570/0x570
[  842.610203]  [<ffffffff810525b6>] ? kthread+0x96/0xa0
[  842.610210]  [<ffffffff813d8294>] ? kernel_thread_helper+0x4/0x10
[  842.610218]  [<ffffffff81052520>] ? kthread_worker_fn+0x120/0x120
[  842.610223]  [<ffffffff813d8290>] ? gs_change+0xb/0xb
[  842.610229] INFO: task fsnotify_mark:662 blocked for more than 120 
seconds.
[  842.610233] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" 
disables this message.
[  842.610237] fsnotify_mark   D 0000000000000000     0   662      2 
0x00000000
[  842.610244]  ffff88041c31d860 0000000000000046 0000000000000000 
0000000000000000
[  842.610251]  ffffffff81593020 ffff88041c365fd8 0000000000004000 
ffff88041c364010
[  842.610257]  ffff88041c365fd8 ffff88041c31d860 0000000000000000 
0000000000000000
[  842.610263] Call Trace:
[  842.610270]  [<ffffffff81035521>] ? load_balance+0x91/0x5e0
[  842.610278]  [<ffffffff813d5595>] ? schedule_timeout+0x1c5/0x230
[  842.610285]  [<ffffffff8102f323>] ? pick_next_task_fair+0x103/0x190
[  842.610291]  [<ffffffff813d4a6d>] ? schedule+0x28d/0x910
[  842.610297]  [<ffffffff813d4562>] ? wait_for_common+0xd2/0x180
[  842.610303]  [<ffffffff810339b0>] ? try_to_wake_up+0x2c0/0x2c0
[  842.610310]  [<ffffffff81074420>] ? synchronize_rcu_bh+0x50/0x50
[  842.610316]  [<ffffffff8107446a>] ? synchronize_sched+0x4a/0x50
[  842.610322]  [<ffffffff8104f940>] ? find_ge_pid+0x40/0x40
[  842.610329]  [<ffffffff8105742b>] ? __synchronize_srcu+0x5b/0xc0
[  842.610338]  [<ffffffff810f5f63>] ? fsnotify_mark_destroy+0x83/0x150
[  842.610344]  [<ffffffff81052a20>] ? wake_up_bit+0x40/0x40
[  842.610352]  [<ffffffff810f5ee0>] ? 
fsnotify_set_mark_ignored_mask_locked+0x20/0x20
[  842.610360]  [<ffffffff810f5ee0>] ? 
fsnotify_set_mark_ignored_mask_locked+0x20/0x20
[  842.610367]  [<ffffffff810525b6>] ? kthread+0x96/0xa0
[  842.610373]  [<ffffffff813d8294>] ? kernel_thread_helper+0x4/0x10
[  842.610381]  [<ffffffff81052520>] ? kthread_worker_fn+0x120/0x120
[  842.610386]  [<ffffffff813d8290>] ? gs_change+0xb/0xb
[  842.610395] INFO: task jbd2/md0-8:2528 blocked for more than 120 seconds.
[  842.610399] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" 
disables this message.
[  842.610403] jbd2/md0-8      D 0000000000000005     0  2528      2 
0x00000000
[  842.610410]  ffff88041d91c570 0000000000000046 ffff88041baeea28 
ffffea0000000000
[  842.610417]  ffff88041d91a5e0 ffff880419f9bfd8 0000000000004000 
ffff880419f9a010
[  842.610423]  ffff880419f9bfd8 ffff88041d91c570 ffff88041baeea28 
ffffffff810b8495
[  842.610429] Call Trace:
[  842.610435]  [<ffffffff810b8495>] ? init_object+0x85/0xa0
[  842.610442]  [<ffffffff810b8916>] ? free_debug_processing+0x196/0x250
[  842.610450]  [<ffffffff8105adae>] ? ktime_get_ts+0x6e/0xf0
[  842.610458]  [<ffffffff810810f0>] ? __lock_page+0x70/0x70
[  842.610464]  [<ffffffff813d5174>] ? io_schedule+0x84/0xd0
[  842.610473]  [<ffffffff811d4403>] ? 
radix_tree_gang_lookup_tag_slot+0x93/0xf0
[  842.610480]  [<ffffffff810810f9>] ? sleep_on_page+0x9/0x10
[  842.610486]  [<ffffffff813d57df>] ? __wait_on_bit+0x4f/0x80
[  842.610492]  [<ffffffff810812eb>] ? wait_on_page_bit+0x6b/0x80
[  842.610499]  [<ffffffff81052a50>] ? autoremove_wake_function+0x30/0x30
[  842.610507]  [<ffffffff8108a458>] ? pagevec_lookup_tag+0x18/0x20
[  842.610512]  [<ffffffff81081f2a>] ? filemap_fdatawait_range+0xfa/0x180
[  842.610521]  [<ffffffff811be09f>] ? submit_bio+0x6f/0xf0
[  842.610530]  [<ffffffff81176276>] ? 
jbd2_journal_commit_transaction+0x796/0x1270
[  842.610539]  [<ffffffff81179ed1>] ? kjournald2+0xb1/0x1e0
[  842.610546]  [<ffffffff81052a20>] ? wake_up_bit+0x40/0x40
[  842.610553]  [<ffffffff81179e20>] ? commit_timeout+0x10/0x10
[  842.610560]  [<ffffffff81179e20>] ? commit_timeout+0x10/0x10
[  842.610566]  [<ffffffff810525b6>] ? kthread+0x96/0xa0
[  842.610572]  [<ffffffff813d8294>] ? kernel_thread_helper+0x4/0x10
[  842.610580]  [<ffffffff81052520>] ? kthread_worker_fn+0x120/0x120
[  842.610585]  [<ffffffff813d8290>] ? gs_change+0xb/0xb
[  842.610594] INFO: task nfsd:4326 blocked for more than 120 seconds.
[  842.610598] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" 
disables this message.
[  842.610602] nfsd            D 0000000000000000     0  4326      2 
0x00000000
[  842.610608]  ffff88041b604bc0 0000000000000046 0000000000000016 
ffffffff00000000
[  842.610614]  ffff88041d8fa5e0 ffff88041c527fd8 0000000000004000 
ffff88041c526010
[  842.610621]  ffff88041c527fd8 ffff88041b604bc0 000000000000009c 
ffff880000000000
[  842.610626] Call Trace:
[  842.610634]  [<ffffffff8101e845>] ? amd_flush_garts+0x105/0x140
[  842.610640]  [<ffffffff8101fa80>] ? gart_map_sg+0x480/0x480
[  842.610646]  [<ffffffff8101f5d3>] ? flush_gart+0x23/0x50
[  842.610653]  [<ffffffff81080f98>] ? find_get_page+0x18/0x90
[  842.610660]  [<ffffffff81174b95>] ? do_get_write_access+0x265/0x4a0
[  842.610668]  [<ffffffff81052a50>] ? autoremove_wake_function+0x30/0x30
[  842.610675]  [<ffffffff81174ef9>] ? 
jbd2_journal_get_write_access+0x29/0x50
[  842.610684]  [<ffffffff8115f122>] ? 
__ext4_journal_get_write_access+0x32/0x80
[  842.610692]  [<ffffffff81143908>] ? ext4_reserve_inode_write+0x78/0xa0
[  842.610700]  [<ffffffff81143970>] ? ext4_mark_inode_dirty+0x40/0x1e0
[  842.610706]  [<ffffffff81156c0b>] ? ext4_journal_start_sb+0x6b/0x160
[  842.610715]  [<ffffffff81322645>] ? dev_hard_start_xmit+0x305/0x5f0
[  842.610723]  [<ffffffff81352fd0>] ? ip_finish_output2+0x290/0x290
[  842.610730]  [<ffffffff81143c65>] ? ext4_dirty_inode+0x35/0x70
[  842.610738]  [<ffffffff810e4a08>] ? __mark_inode_dirty+0x38/0x210
[  842.610745]  [<ffffffff810d9317>] ? file_update_time+0xf7/0x180
[  842.610751]  [<ffffffff81082458>] ? __generic_file_aio_write+0x1f8/0x430
[  842.610760]  [<ffffffff81373ef9>] ? udp_sendmsg+0x3c9/0x7e0
[  842.610767]  [<ffffffff81314260>] ? sock_alloc_send_pskb+0x1d0/0x320
[  842.610774]  [<ffffffff81082703>] ? generic_file_aio_write+0x73/0xf0
[  842.610781]  [<ffffffff8113f74e>] ? ext4_file_write+0x6e/0x2b0
[  842.610788]  [<ffffffff810da8ac>] ? iget_locked+0x4c/0x140
[  842.610794]  [<ffffffff8119a590>] ? fh_compose+0x4c0/0x4c0
[  842.610800]  [<ffffffff8113f6e0>] ? ext4_llseek+0x110/0x110
[  842.610808]  [<ffffffff810c24e9>] ? do_sync_readv_writev+0xa9/0xf0
[  842.610816]  [<ffffffff810c231b>] ? rw_copy_check_uvector+0x7b/0x140
[  842.610824]  [<ffffffff810c2bcf>] ? do_readv_writev+0xdf/0x210
[  842.610830]  [<ffffffff810b8735>] ? alloc_debug_processing+0x145/0x190
[  842.610836]  [<ffffffff810b9bc5>] ? kmem_cache_alloc+0xa5/0xb0
[  842.610843]  [<ffffffff8113f463>] ? ext4_file_open+0x63/0x180
[  842.610849]  [<ffffffff8119c21d>] ? nfsd_vfs_write+0xed/0x3a0
[  842.610856]  [<ffffffff810c1127>] ? __dentry_open+0x1f7/0x2b0
[  842.610862]  [<ffffffff8119c892>] ? nfsd_open+0xf2/0x1b0
[  842.610867]  [<ffffffff8119cd34>] ? nfsd_write+0xf4/0x110
[  842.610873]  [<ffffffff81199930>] ? nfsd_proc_write+0xb0/0x120
[  842.610880]  [<ffffffff811971c5>] ? nfsd_dispatch+0xf5/0x230
[  842.610886]  [<ffffffff813b456f>] ? svc_process+0x4af/0x820
[  842.610892]  [<ffffffff810339b0>] ? try_to_wake_up+0x2c0/0x2c0
[  842.610899]  [<ffffffff811977a0>] ? nfsd_svc+0x1b0/0x1b0
[  842.610906]  [<ffffffff8119784d>] ? nfsd+0xad/0x150
[  842.610912]  [<ffffffff810525b6>] ? kthread+0x96/0xa0
[  842.610918]  [<ffffffff813d8294>] ? kernel_thread_helper+0x4/0x10
[  842.610926]  [<ffffffff81052520>] ? kthread_worker_fn+0x120/0x120
[  842.610931]  [<ffffffff813d8290>] ? gs_change+0xb/0xb
[  854.241450] 
=============================================================================
[  854.241459] BUG kmalloc-1024: Object padding overwritten
[  854.241463] 
-----------------------------------------------------------------------------
[  854.241466]
[  854.241471] INFO: 0xffff8802c86059e6-0xffff8802c86059e7. First byte 
0x0 instead of 0x5a
[  854.241488] INFO: Allocated in __blockdev_direct_IO+0x16c/0xa90 
age=518 cpu=5 pid=7094
[  854.241500] INFO: Freed in __kfree_skb+0x11/0x90 age=519 cpu=5 pid=5247
[  854.241507] INFO: Slab 0xffffea0009bd5000 objects=29 used=22 
fp=0xffff8802c8606f50 flags=0x80000000000040c1
[  854.241513] INFO: Object 0xffff8802c86055a0 @offset=21920 
fp=0xffff8802c86066c0
[  854.241516]
[  854.241520] Bytes b4 0xffff8802c8605590:  00 00 00 00 00 00 00 00 5a 
5a 5a 5a 5a 5a 5a 5a ........ZZZZZZZZ
[  854.241536]   Object 0xffff8802c86055a0:  00 00 00 00 00 00 00 00 e0 
61 29 fa 03 88 ff ff ........A a)Ao..A?A?
[  854.241551]   Object 0xffff8802c86055b0:  00 00 00 00 00 00 00 00 00 
00 41 ce 04 00 00 00 ..........AA?....
[  854.241565]   Object 0xffff8802c86055c0:  03 00 00 00 0c 00 00 00 00 
00 00 00 01 00 00 00 ................
[  854.241579]   Object 0xffff8802c86055d0:  00 00 00 00 00 00 00 00 00 
60 00 00 00 00 00 00 .........`......
[  854.241593]   Object 0xffff8802c86055e0:  b7 2c 0a 00 00 00 00 00 00 
00 00 00 00 00 00 00 A.,..............
[  854.241607]   Object 0xffff8802c86055f0:  b7 2c 0a 00 00 00 00 00 00 
00 00 00 00 00 00 00 A.,..............
[  854.241621]   Object 0xffff8802c8605600:  01 00 00 00 00 00 00 00 f0 
77 14 81 ff ff ff ff ........A?w..A?A?A?A?
[  854.241635]   Object 0xffff8802c8605610:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[  854.241649]   Object 0xffff8802c8605620:  00 00 00 00 00 00 00 00 b7 
54 3b 02 00 00 00 00 ........A.T;.....
[  854.241663]   Object 0xffff8802c8605630:  b7 54 3b 02 00 00 00 00 20 
00 00 00 00 00 00 00 A.T;.............
[  854.241677]   Object 0xffff8802c8605640:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[  854.241691]   Object 0xffff8802c8605650:  b6 54 3b 02 00 00 00 00 00 
10 00 00 00 00 00 00 A?T;.............
[  854.241705]   Object 0xffff8802c8605660:  00 00 00 00 00 00 00 00 00 
80 8e 1a 04 88 ff ff ..............A?A?
[  854.241719]   Object 0xffff8802c8605670:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[  854.241733]   Object 0xffff8802c8605680:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[  854.241747]   Object 0xffff8802c8605690:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[  854.241761]   Object 0xffff8802c86056a0:  00 00 00 00 00 00 00 00 00 
00 00 00 00 10 00 00 ................
[  854.241775]   Object 0xffff8802c86056b0:  b6 54 3b 02 00 00 00 00 00 
60 cb a2 00 00 00 00 A?T;......`A?Ac....
[  854.241789]   Object 0xffff8802c86056c0:  06 06 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[  854.241803]   Object 0xffff8802c86056d0:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[  854.241817]   Object 0xffff8802c86056e0:  68 fd 1c c9 02 88 ff ff 00 
00 00 00 00 00 00 00 hA 1/2 .A?..A?A?........
[  854.241831]   Object 0xffff8802c86056f0:  00 60 00 00 00 00 00 00 01 
00 00 00 01 00 00 00 .`..............
[  854.241845]   Object 0xffff8802c8605700:  00 80 c9 dd 39 7f 00 00 01 
00 00 00 01 00 00 00 ..A?A?9...........
[  854.241860]   Object 0xffff8802c8605710:  00 00 00 00 00 00 00 00 08 
81 bb 0c 00 ea ff ff ..........A>>..AaA?A?
[  854.241874]   Object 0xffff8802c8605720:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.241888]   Object 0xffff8802c8605730:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.241902]   Object 0xffff8802c8605740:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.241916]   Object 0xffff8802c8605750:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.241930]   Object 0xffff8802c8605760:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.241944]   Object 0xffff8802c8605770:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.241959]   Object 0xffff8802c8605780:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.241973]   Object 0xffff8802c8605790:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.241987]   Object 0xffff8802c86057a0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.242001]   Object 0xffff8802c86057b0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.242015]   Object 0xffff8802c86057c0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.242029]   Object 0xffff8802c86057d0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.242043]   Object 0xffff8802c86057e0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.242058]   Object 0xffff8802c86057f0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.242072]   Object 0xffff8802c8605800:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.242086]   Object 0xffff8802c8605810:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.242100]   Object 0xffff8802c8605820:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.242114]   Object 0xffff8802c8605830:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.242128]   Object 0xffff8802c8605840:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.242142]   Object 0xffff8802c8605850:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.242157]   Object 0xffff8802c8605860:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.242171]   Object 0xffff8802c8605870:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.242185]   Object 0xffff8802c8605880:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.242199]   Object 0xffff8802c8605890:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.242213]   Object 0xffff8802c86058a0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.242227]   Object 0xffff8802c86058b0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.242241]   Object 0xffff8802c86058c0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.242256]   Object 0xffff8802c86058d0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.242270]   Object 0xffff8802c86058e0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.242284]   Object 0xffff8802c86058f0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.242298]   Object 0xffff8802c8605900:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.242312]   Object 0xffff8802c8605910:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.242326]   Object 0xffff8802c8605920:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.242340]   Object 0xffff8802c8605930:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.242355]   Object 0xffff8802c8605940:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.242369]   Object 0xffff8802c8605950:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.242383]   Object 0xffff8802c8605960:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.242397]   Object 0xffff8802c8605970:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.242411]   Object 0xffff8802c8605980:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.242425]   Object 0xffff8802c8605990:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b a5 kkkkkkkkkkkkkkkAJPY
[  854.242440]  Redzone 0xffff8802c86059a0:  cc cc cc cc cc cc cc cc 
                      A?A?A?A?A?A?A?A?
[  854.242453]  Padding 0xffff8802c86059e0:  5a 5a 5a 5a 5a 5a 00 00 
                      ZZZZZZ..
[  854.242470] Pid: 7094, comm: qemu Not tainted 2.6.39 #2
[  854.242474] Call Trace:
[  854.242485]  [<ffffffff810b7ccd>] ? check_bytes_and_report+0x10d/0x150
[  854.242492]  [<ffffffff810b7db9>] ? check_object+0xa9/0x260
[  854.242500]  [<ffffffff810f3126>] ? __blockdev_direct_IO+0xa16/0xa90
[  854.242507]  [<ffffffff810b88b3>] ? free_debug_processing+0x133/0x250
[  854.242513]  [<ffffffff810b8afb>] ? __slab_free+0x12b/0x140
[  854.242520]  [<ffffffff810f3126>] ? __blockdev_direct_IO+0xa16/0xa90
[  854.242530]  [<ffffffff81144a17>] ? ext4_ind_direct_IO+0xf7/0x410
[  854.242536]  [<ffffffff811477f0>] ? noalloc_get_block_write+0x30/0x30
[  854.242543]  [<ffffffff81082029>] ? __filemap_fdatawrite_range+0x49/0x50
[  854.242550]  [<ffffffff810830c3>] ? generic_file_aio_read+0x653/0x6d0
[  854.242558]  [<ffffffff810567a2>] ? hrtimer_cancel+0x12/0x20
[  854.242565]  [<ffffffff81062137>] ? futex_wait+0x197/0x240
[  854.242572]  [<ffffffff81082a70>] ? delete_from_page_cache+0x70/0x70
[  854.242580]  [<ffffffff810c24e9>] ? do_sync_readv_writev+0xa9/0xf0
[  854.242587]  [<ffffffff8106239f>] ? futex_wake+0x10f/0x120
[  854.242594]  [<ffffffff81063f7b>] ? do_futex+0x11b/0xa70
[  854.242601]  [<ffffffff810471cc>] ? T.680+0x13c/0x280
[  854.242608]  [<ffffffff810c231b>] ? rw_copy_check_uvector+0x7b/0x140
[  854.242616]  [<ffffffff810c2bcf>] ? do_readv_writev+0xdf/0x210
[  854.242624]  [<ffffffff810c3003>] ? sys_preadv+0xc3/0xd0
[  854.242633]  [<ffffffff813d753b>] ? system_call_fastpath+0x16/0x1b
[  854.242640] FIX kmalloc-1024: Restoring 
0xffff8802c86059e6-0xffff8802c86059e7=0x5a
[  854.242643]
[  860.203918] 
=============================================================================
[  860.203927] BUG kmalloc-1024: Object padding overwritten
[  860.203931] 
-----------------------------------------------------------------------------
[  860.203934]
[  860.203939] INFO: 0xffff88041ad15156-0xffff88041ad15157. First byte 
0x0 instead of 0x5a
[  860.203955] INFO: Allocated in __blockdev_direct_IO+0x16c/0xa90 
age=1609 cpu=3 pid=7094
[  860.203965] INFO: Freed in __blockdev_direct_IO+0xa16/0xa90 age=1609 
cpu=3 pid=7094
[  860.203972] INFO: Slab 0xffffea000e5ddb80 objects=29 used=25 
fp=0xffff88041ad14d10 flags=0x80000000000040c1
[  860.203977] INFO: Object 0xffff88041ad14d10 @offset=19728 
fp=0xffff88041ad10000
[  860.203981]
[  860.203984] Bytes b4 0xffff88041ad14d00:  62 28 00 00 01 00 00 00 5a 
5a 5a 5a 5a 5a 5a 5a b(......ZZZZZZZZ
[  860.204000]   Object 0xffff88041ad14d10:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204015]   Object 0xffff88041ad14d20:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204030]   Object 0xffff88041ad14d30:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204044]   Object 0xffff88041ad14d40:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204058]   Object 0xffff88041ad14d50:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204073]   Object 0xffff88041ad14d60:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204087]   Object 0xffff88041ad14d70:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204101]   Object 0xffff88041ad14d80:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204116]   Object 0xffff88041ad14d90:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204130]   Object 0xffff88041ad14da0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204144]   Object 0xffff88041ad14db0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204159]   Object 0xffff88041ad14dc0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204173]   Object 0xffff88041ad14dd0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204187]   Object 0xffff88041ad14de0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204202]   Object 0xffff88041ad14df0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204216]   Object 0xffff88041ad14e00:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204230]   Object 0xffff88041ad14e10:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204245]   Object 0xffff88041ad14e20:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204259]   Object 0xffff88041ad14e30:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204273]   Object 0xffff88041ad14e40:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204288]   Object 0xffff88041ad14e50:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204302]   Object 0xffff88041ad14e60:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204316]   Object 0xffff88041ad14e70:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204331]   Object 0xffff88041ad14e80:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204345]   Object 0xffff88041ad14e90:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204359]   Object 0xffff88041ad14ea0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204374]   Object 0xffff88041ad14eb0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204388]   Object 0xffff88041ad14ec0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204402]   Object 0xffff88041ad14ed0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204417]   Object 0xffff88041ad14ee0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204431]   Object 0xffff88041ad14ef0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204445]   Object 0xffff88041ad14f00:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204460]   Object 0xffff88041ad14f10:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204474]   Object 0xffff88041ad14f20:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204488]   Object 0xffff88041ad14f30:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204503]   Object 0xffff88041ad14f40:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204517]   Object 0xffff88041ad14f50:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204531]   Object 0xffff88041ad14f60:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204546]   Object 0xffff88041ad14f70:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204560]   Object 0xffff88041ad14f80:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204574]   Object 0xffff88041ad14f90:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204589]   Object 0xffff88041ad14fa0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204603]   Object 0xffff88041ad14fb0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204618]   Object 0xffff88041ad14fc0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204632]   Object 0xffff88041ad14fd0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204646]   Object 0xffff88041ad14fe0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204661]   Object 0xffff88041ad14ff0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204675]   Object 0xffff88041ad15000:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204689]   Object 0xffff88041ad15010:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204704]   Object 0xffff88041ad15020:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204718]   Object 0xffff88041ad15030:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204732]   Object 0xffff88041ad15040:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204747]   Object 0xffff88041ad15050:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204761]   Object 0xffff88041ad15060:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204775]   Object 0xffff88041ad15070:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204790]   Object 0xffff88041ad15080:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204804]   Object 0xffff88041ad15090:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204818]   Object 0xffff88041ad150a0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204833]   Object 0xffff88041ad150b0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204847]   Object 0xffff88041ad150c0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204861]   Object 0xffff88041ad150d0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204876]   Object 0xffff88041ad150e0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204890]   Object 0xffff88041ad150f0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204904]   Object 0xffff88041ad15100:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b a5 kkkkkkkkkkkkkkkAJPY
[  860.204919]  Redzone 0xffff88041ad15110:  bb bb bb bb bb bb bb bb 
                      A>>A>>A>>A>>A>>A>>A>>A>>
[  860.204932]  Padding 0xffff88041ad15150:  5a 5a 5a 5a 5a 5a 00 00 
                      ZZZZZZ..
[  860.204949] Pid: 7100, comm: qemu Not tainted 2.6.39 #2
[  860.204953] Call Trace:
[  860.204964]  [<ffffffff810b7ccd>] ? check_bytes_and_report+0x10d/0x150
[  860.204973]  [<ffffffff810f287c>] ? __blockdev_direct_IO+0x16c/0xa90
[  860.204979]  [<ffffffff810b7db9>] ? check_object+0xa9/0x260
[  860.204986]  [<ffffffff810f287c>] ? __blockdev_direct_IO+0x16c/0xa90
[  860.204993]  [<ffffffff810b86f4>] ? alloc_debug_processing+0x104/0x190
[  860.204999]  [<ffffffff810b9ac2>] ? T.912+0x272/0x2d0
[  860.205006]  [<ffffffff810b9bc5>] ? kmem_cache_alloc+0xa5/0xb0
[  860.205013]  [<ffffffff810f287c>] ? __blockdev_direct_IO+0x16c/0xa90
[  860.205022]  [<ffffffff8108a475>] ? pagevec_lookup+0x15/0x20
[  860.205029]  [<ffffffff8108b5b7>] ? 
invalidate_inode_pages2_range+0x87/0x300
[  860.205039]  [<ffffffff81144e41>] ? ext4_direct_IO+0x111/0x1f0
[  860.205045]  [<ffffffff811477a0>] ? _ext4_get_block+0x160/0x160
[  860.205052]  [<ffffffff81144f20>] ? ext4_direct_IO+0x1f0/0x1f0
[  860.205058]  [<ffffffff81082182>] ? generic_file_direct_write+0xd2/0x1b0
[  860.205064]  [<ffffffff8108252b>] ? __generic_file_aio_write+0x2cb/0x430
[  860.205071]  [<ffffffff81082703>] ? generic_file_aio_write+0x73/0xf0
[  860.205079]  [<ffffffff8113f74e>] ? ext4_file_write+0x6e/0x2b0
[  860.205087]  [<ffffffff810c25ef>] ? do_sync_write+0xbf/0x100
[  860.205095]  [<ffffffff810471cc>] ? T.680+0x13c/0x280
[  860.205103]  [<ffffffff810485a8>] ? kill_pid_info+0x38/0x60
[  860.205110]  [<ffffffff81048854>] ? sys_kill+0x94/0x1b0
[  860.205117]  [<ffffffff810c3198>] ? vfs_write+0xc8/0x190
[  860.205124]  [<ffffffff810c32fb>] ? sys_pwrite64+0x9b/0xb0
[  860.205134]  [<ffffffff813d753b>] ? system_call_fastpath+0x16/0x1b
[  860.205140] FIX kmalloc-1024: Restoring 
0xffff88041ad15156-0xffff88041ad15157=0x5a
[  860.205143]

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
  2011-06-01  9:29                       ` Brad Campbell
@ 2011-06-01  9:40                         ` Avi Kivity
  -1 siblings, 0 replies; 111+ messages in thread
From: Avi Kivity @ 2011-06-01  9:40 UTC (permalink / raw)
  To: Brad Campbell
  Cc: Hugh Dickins, Andrea Arcangeli, Borislav Petkov, linux-kernel,
	kvm, linux-mm

On 06/01/2011 12:29 PM, Brad Campbell wrote:
> On 01/06/11 14:56, Avi Kivity wrote:
>> On 06/01/2011 09:31 AM, Brad Campbell wrote:
>>> On 01/06/11 12:52, Hugh Dickins wrote:
>>>
>>>>
>>>> I guess Brad could try SLUB debugging, boot with slub_debug=P
>>>> for poisoning perhaps; though it might upset alignments and
>>>> drive the problem underground. Or see if the same happens
>>>> with SLAB instead of SLUB.
>>>
>>> Not much use I'm afraid.
>>> This is all I get in the log
>>>
>>> [ 3161.300073]
>>> ============================================================================= 
>>>
>>>
>>> [ 3161.300147] BUG kmalloc-512: Freechain corrupt
>>>
>>> The qemu process is then frozen, unkillable but reported in state "R"
>>>
>>> 13881 ? R 3:27 /usr/bin/qemu -S -M pc-0.13 -enable-kvm -m 1024 -smp
>>> 2,sockets=2,cores=1,threads=1 -nam
>>>
>>> The machine then progressively dies until it's frozen solid with no
>>> further error messages.
>>>
>>> I stupidly forgot to do an alt-sysrq-t prior to doing an alt-sysrq-b,
>>> but at least it responded to that.
>>>
>>> On the bright side I can reproduce it at will.
>>
>> Please try slub_debug=FZPU; that should point the finger (hopefully at
>> somebody else).
>>
>
> Well the first attempt locked the machine solid. No network, no console..
>
> I saw 
> "=========================================================================="
>
> on the console.. nothing after that. Would not respond to sysrq-t or 
> any other sysrq combination other than -b, which rebooted the box.
>
>
> No output on netconsole at all, I had to walk to the other building to 
> look at the monitor and reboot it.
>
> The second attempt jammed netconsole again, but I managed to get this 
> from an ssh session I already had established. The machine died a slow 
> and horrible death, but remained interactive enough for me to reboot 
> it with
>
> echo b > /proc/sysrq-trigger
>
> Nothing else worked.
>
>
> [  413.756416]  [<ffffffff81318f1c>] ? pskb_expand_head+0x15c/0x250
> [  413.756424]  [<ffffffff813a6c45>] ? nf_bridge_copy_header+0x145/0x160
> [  413.756431]  [<ffffffff8139f78d>] ? br_dev_queue_push_xmit+0x6d/0x80
> [  413.756439]  [<ffffffff813a55a0>] ? br_nf_post_routing+0x2a0/0x2f0
> [  413.756447]  [<ffffffff81346bc4>] ? nf_iterate+0x84/0xb0
> [  413.756453]  [<ffffffff8139f720>] ? br_flood_deliver+0x20/0x20
> [  413.756459]  [<ffffffff81346c64>] ? nf_hook_slow+0x74/0x120
> [  413.756465]  [<ffffffff8139f720>] ? br_flood_deliver+0x20/0x20
> [  413.756472]  [<ffffffff8139f7da>] ? br_forward_finish+0x3a/0x60
> [  413.756479]  [<ffffffff813a5758>] ? br_nf_forward_finish+0x168/0x170
> [  413.756487]  [<ffffffff813a5c90>] ? br_nf_forward_ip+0x360/0x3a0
> [  413.756492]  [<ffffffff81346bc4>] ? nf_iterate+0x84/0xb0
> [  413.756498]  [<ffffffff8139f7a0>] ? br_dev_queue_push_xmit+0x80/0x80
> [  413.756504]  [<ffffffff81346c64>] ? nf_hook_slow+0x74/0x120
> [  413.756510]  [<ffffffff8139f7a0>] ? br_dev_queue_push_xmit+0x80/0x80
> [  413.756516]  [<ffffffff8139f800>] ? br_forward_finish+0x60/0x60
> [  413.756522]  [<ffffffff8139f800>] ? br_forward_finish+0x60/0x60
> [  413.756528]  [<ffffffff8139f875>] ? __br_forward+0x75/0xc0
> [  413.756534]  [<ffffffff8139f426>] ? deliver_clone+0x36/0x60
> [  413.756540]  [<ffffffff8139f69d>] ? br_flood+0xbd/0x100
> [  413.756546]  [<ffffffff813a05b0>] ? br_handle_local_finish+0x40/0x40
> [  413.756552]  [<ffffffff813a080e>] ? br_handle_frame_finish+0x25e/0x280
> [  413.756560]  [<ffffffff813a60f0>] ? 
> br_nf_pre_routing_finish+0x1a0/0x330
> [  413.756568]  [<ffffffff813a6958>] ? br_nf_pre_routing+0x6d8/0x800
> [  413.756577]  [<ffffffff8102d46a>] ? enqueue_task+0x3a/0x90
> [  413.756582]  [<ffffffff81346bc4>] ? nf_iterate+0x84/0xb0
> [  413.756589]  [<ffffffff813a05b0>] ? br_handle_local_finish+0x40/0x40
> [  413.756594]  [<ffffffff81346c64>] ? nf_hook_slow+0x74/0x120
> [  413.756600]  [<ffffffff813a05b0>] ? br_handle_local_finish+0x40/0x40
> [  413.756607]  [<ffffffff810339b0>] ? try_to_wake_up+0x2c0/0x2c0
> [  413.756613]  [<ffffffff813a09d9>] ? br_handle_frame+0x1a9/0x280
> [  413.756620]  [<ffffffff813a0830>] ? br_handle_frame_finish+0x280/0x280
> [  413.756627]  [<ffffffff81320ef7>] ? __netif_receive_skb+0x157/0x5c0
> [  413.756634]  [<ffffffff81321443>] ? process_backlog+0xe3/0x1d0
> [  413.756641]  [<ffffffff81321da5>] ? net_rx_action+0xc5/0x1d0
> [  413.756650]  [<ffffffff8103df11>] ? __do_softirq+0x91/0x120
> [  413.756657]  [<ffffffff813d838c>] ? call_softirq+0x1c/0x30
> [  413.756660] <EOI>  [<ffffffff81003cbd>] ? do_softirq+0x4d/0x80
> [  413.756673]  [<ffffffff81321ece>] ? netif_rx_ni+0x1e/0x30
> [  413.756681]  [<ffffffff812b3ae2>] ? tun_chr_aio_write+0x332/0x4e0
> [  413.756688]  [<ffffffff812b37b0>] ? tun_sendmsg+0x4d0/0x4d0
> [  413.756697]  [<ffffffff810c24e9>] ? do_sync_readv_writev+0xa9/0xf0
> [  413.756704]  [<ffffffff81063f9c>] ? do_futex+0x13c/0xa70
> [  413.756711]  [<ffffffff811d6730>] ? timerqueue_add+0x60/0xb0
> [  413.756719]  [<ffffffff81056ab7>] ? 
> __hrtimer_start_range_ns+0x1e7/0x410
> [  413.756726]  [<ffffffff810c231b>] ? rw_copy_check_uvector+0x7b/0x140
> [  413.756734]  [<ffffffff810c2bcf>] ? do_readv_writev+0xdf/0x210
> [  413.756742]  [<ffffffff810c2e7e>] ? sys_writev+0x4e/0xc0
> [  413.756750]  [<ffffffff813d753b>] ? system_call_fastpath+0x16/0x1b
> [  413.756756] FIX kmalloc-1024: Restoring 
> 0xffff880417179566-0xffff880417179567=0x5a

bridge and netfilter, IIRC this was also the problem last time.

Do you have any ebtables loaded?

Can you try building a kernel without ebtables?  Without netfilter at all?

Please run all tests with slub_debug=FZPU.

-- 
error compiling committee.c: too many arguments to function


^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
@ 2011-06-01  9:40                         ` Avi Kivity
  0 siblings, 0 replies; 111+ messages in thread
From: Avi Kivity @ 2011-06-01  9:40 UTC (permalink / raw)
  To: Brad Campbell
  Cc: Hugh Dickins, Andrea Arcangeli, Borislav Petkov, linux-kernel,
	kvm, linux-mm

On 06/01/2011 12:29 PM, Brad Campbell wrote:
> On 01/06/11 14:56, Avi Kivity wrote:
>> On 06/01/2011 09:31 AM, Brad Campbell wrote:
>>> On 01/06/11 12:52, Hugh Dickins wrote:
>>>
>>>>
>>>> I guess Brad could try SLUB debugging, boot with slub_debug=P
>>>> for poisoning perhaps; though it might upset alignments and
>>>> drive the problem underground. Or see if the same happens
>>>> with SLAB instead of SLUB.
>>>
>>> Not much use I'm afraid.
>>> This is all I get in the log
>>>
>>> [ 3161.300073]
>>> ============================================================================= 
>>>
>>>
>>> [ 3161.300147] BUG kmalloc-512: Freechain corrupt
>>>
>>> The qemu process is then frozen, unkillable but reported in state "R"
>>>
>>> 13881 ? R 3:27 /usr/bin/qemu -S -M pc-0.13 -enable-kvm -m 1024 -smp
>>> 2,sockets=2,cores=1,threads=1 -nam
>>>
>>> The machine then progressively dies until it's frozen solid with no
>>> further error messages.
>>>
>>> I stupidly forgot to do an alt-sysrq-t prior to doing an alt-sysrq-b,
>>> but at least it responded to that.
>>>
>>> On the bright side I can reproduce it at will.
>>
>> Please try slub_debug=FZPU; that should point the finger (hopefully at
>> somebody else).
>>
>
> Well the first attempt locked the machine solid. No network, no console..
>
> I saw 
> "=========================================================================="
>
> on the console.. nothing after that. Would not respond to sysrq-t or 
> any other sysrq combination other than -b, which rebooted the box.
>
>
> No output on netconsole at all, I had to walk to the other building to 
> look at the monitor and reboot it.
>
> The second attempt jammed netconsole again, but I managed to get this 
> from an ssh session I already had established. The machine died a slow 
> and horrible death, but remained interactive enough for me to reboot 
> it with
>
> echo b > /proc/sysrq-trigger
>
> Nothing else worked.
>
>
> [  413.756416]  [<ffffffff81318f1c>] ? pskb_expand_head+0x15c/0x250
> [  413.756424]  [<ffffffff813a6c45>] ? nf_bridge_copy_header+0x145/0x160
> [  413.756431]  [<ffffffff8139f78d>] ? br_dev_queue_push_xmit+0x6d/0x80
> [  413.756439]  [<ffffffff813a55a0>] ? br_nf_post_routing+0x2a0/0x2f0
> [  413.756447]  [<ffffffff81346bc4>] ? nf_iterate+0x84/0xb0
> [  413.756453]  [<ffffffff8139f720>] ? br_flood_deliver+0x20/0x20
> [  413.756459]  [<ffffffff81346c64>] ? nf_hook_slow+0x74/0x120
> [  413.756465]  [<ffffffff8139f720>] ? br_flood_deliver+0x20/0x20
> [  413.756472]  [<ffffffff8139f7da>] ? br_forward_finish+0x3a/0x60
> [  413.756479]  [<ffffffff813a5758>] ? br_nf_forward_finish+0x168/0x170
> [  413.756487]  [<ffffffff813a5c90>] ? br_nf_forward_ip+0x360/0x3a0
> [  413.756492]  [<ffffffff81346bc4>] ? nf_iterate+0x84/0xb0
> [  413.756498]  [<ffffffff8139f7a0>] ? br_dev_queue_push_xmit+0x80/0x80
> [  413.756504]  [<ffffffff81346c64>] ? nf_hook_slow+0x74/0x120
> [  413.756510]  [<ffffffff8139f7a0>] ? br_dev_queue_push_xmit+0x80/0x80
> [  413.756516]  [<ffffffff8139f800>] ? br_forward_finish+0x60/0x60
> [  413.756522]  [<ffffffff8139f800>] ? br_forward_finish+0x60/0x60
> [  413.756528]  [<ffffffff8139f875>] ? __br_forward+0x75/0xc0
> [  413.756534]  [<ffffffff8139f426>] ? deliver_clone+0x36/0x60
> [  413.756540]  [<ffffffff8139f69d>] ? br_flood+0xbd/0x100
> [  413.756546]  [<ffffffff813a05b0>] ? br_handle_local_finish+0x40/0x40
> [  413.756552]  [<ffffffff813a080e>] ? br_handle_frame_finish+0x25e/0x280
> [  413.756560]  [<ffffffff813a60f0>] ? 
> br_nf_pre_routing_finish+0x1a0/0x330
> [  413.756568]  [<ffffffff813a6958>] ? br_nf_pre_routing+0x6d8/0x800
> [  413.756577]  [<ffffffff8102d46a>] ? enqueue_task+0x3a/0x90
> [  413.756582]  [<ffffffff81346bc4>] ? nf_iterate+0x84/0xb0
> [  413.756589]  [<ffffffff813a05b0>] ? br_handle_local_finish+0x40/0x40
> [  413.756594]  [<ffffffff81346c64>] ? nf_hook_slow+0x74/0x120
> [  413.756600]  [<ffffffff813a05b0>] ? br_handle_local_finish+0x40/0x40
> [  413.756607]  [<ffffffff810339b0>] ? try_to_wake_up+0x2c0/0x2c0
> [  413.756613]  [<ffffffff813a09d9>] ? br_handle_frame+0x1a9/0x280
> [  413.756620]  [<ffffffff813a0830>] ? br_handle_frame_finish+0x280/0x280
> [  413.756627]  [<ffffffff81320ef7>] ? __netif_receive_skb+0x157/0x5c0
> [  413.756634]  [<ffffffff81321443>] ? process_backlog+0xe3/0x1d0
> [  413.756641]  [<ffffffff81321da5>] ? net_rx_action+0xc5/0x1d0
> [  413.756650]  [<ffffffff8103df11>] ? __do_softirq+0x91/0x120
> [  413.756657]  [<ffffffff813d838c>] ? call_softirq+0x1c/0x30
> [  413.756660] <EOI>  [<ffffffff81003cbd>] ? do_softirq+0x4d/0x80
> [  413.756673]  [<ffffffff81321ece>] ? netif_rx_ni+0x1e/0x30
> [  413.756681]  [<ffffffff812b3ae2>] ? tun_chr_aio_write+0x332/0x4e0
> [  413.756688]  [<ffffffff812b37b0>] ? tun_sendmsg+0x4d0/0x4d0
> [  413.756697]  [<ffffffff810c24e9>] ? do_sync_readv_writev+0xa9/0xf0
> [  413.756704]  [<ffffffff81063f9c>] ? do_futex+0x13c/0xa70
> [  413.756711]  [<ffffffff811d6730>] ? timerqueue_add+0x60/0xb0
> [  413.756719]  [<ffffffff81056ab7>] ? 
> __hrtimer_start_range_ns+0x1e7/0x410
> [  413.756726]  [<ffffffff810c231b>] ? rw_copy_check_uvector+0x7b/0x140
> [  413.756734]  [<ffffffff810c2bcf>] ? do_readv_writev+0xdf/0x210
> [  413.756742]  [<ffffffff810c2e7e>] ? sys_writev+0x4e/0xc0
> [  413.756750]  [<ffffffff813d753b>] ? system_call_fastpath+0x16/0x1b
> [  413.756756] FIX kmalloc-1024: Restoring 
> 0xffff880417179566-0xffff880417179567=0x5a

bridge and netfilter, IIRC this was also the problem last time.

Do you have any ebtables loaded?

Can you try building a kernel without ebtables?  Without netfilter at all?

Please run all tests with slub_debug=FZPU.

-- 
error compiling committee.c: too many arguments to function

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
  2011-06-01  9:40                         ` Avi Kivity
@ 2011-06-01  9:41                           ` Avi Kivity
  -1 siblings, 0 replies; 111+ messages in thread
From: Avi Kivity @ 2011-06-01  9:41 UTC (permalink / raw)
  To: Brad Campbell
  Cc: Hugh Dickins, Andrea Arcangeli, Borislav Petkov, linux-kernel,
	kvm, linux-mm

On 06/01/2011 12:40 PM, Avi Kivity wrote:
>
> bridge and netfilter, IIRC this was also the problem last time.
>
> Do you have any ebtables loaded?
>
> Can you try building a kernel without ebtables?  Without netfilter at 
> all?
>
> Please run all tests with slub_debug=FZPU.
>

-- 
error compiling committee.c: too many arguments to function


^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
@ 2011-06-01  9:41                           ` Avi Kivity
  0 siblings, 0 replies; 111+ messages in thread
From: Avi Kivity @ 2011-06-01  9:41 UTC (permalink / raw)
  To: Brad Campbell
  Cc: Hugh Dickins, Andrea Arcangeli, Borislav Petkov, linux-kernel,
	kvm, linux-mm

On 06/01/2011 12:40 PM, Avi Kivity wrote:
>
> bridge and netfilter, IIRC this was also the problem last time.
>
> Do you have any ebtables loaded?
>
> Can you try building a kernel without ebtables?  Without netfilter at 
> all?
>
> Please run all tests with slub_debug=FZPU.
>

-- 
error compiling committee.c: too many arguments to function

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
  2011-06-01  9:41                           ` Avi Kivity
@ 2011-06-01 10:53                             ` Brad Campbell
  -1 siblings, 0 replies; 111+ messages in thread
From: Brad Campbell @ 2011-06-01 10:53 UTC (permalink / raw)
  To: Avi Kivity
  Cc: Hugh Dickins, Andrea Arcangeli, Borislav Petkov, linux-kernel,
	kvm, linux-mm

On 01/06/11 17:41, Avi Kivity wrote:
> On 06/01/2011 12:40 PM, Avi Kivity wrote:
>>
>> bridge and netfilter, IIRC this was also the problem last time.
>>
>> Do you have any ebtables loaded?

Never heard of them, but making a cursory check just in case..

brad@srv:/raid10/src/linux-2.6.39$ grep EBTABLE .config
# CONFIG_BRIDGE_NF_EBTABLES is not set

>> Can you try building a kernel without ebtables? Without netfilter at all?

Well, without netfilter I can't get it to crash. The problem is without 
netfilter I can't actually use it the way I use it to get it to crash.

I rebooted into a netfilter kernel, and did all the steps I'd used on 
the no-netfilter kernel and it ticked along happily.

So the result of the experiment is inconclusive. Having said that, the 
backtraces certainly smell networky.

To get it to crash, I have to start IE in the VM and https to the public 
address of the machine, which is then redirected by netfilter back into 
another of the VM's.

I can https directly to the other VM's address, but that does not cause 
it to crash, however without netfilter loaded I can't bounce off the 
public IP. It's all rather confusing really.

What next Sherlock?


^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
@ 2011-06-01 10:53                             ` Brad Campbell
  0 siblings, 0 replies; 111+ messages in thread
From: Brad Campbell @ 2011-06-01 10:53 UTC (permalink / raw)
  To: Avi Kivity
  Cc: Hugh Dickins, Andrea Arcangeli, Borislav Petkov, linux-kernel,
	kvm, linux-mm

On 01/06/11 17:41, Avi Kivity wrote:
> On 06/01/2011 12:40 PM, Avi Kivity wrote:
>>
>> bridge and netfilter, IIRC this was also the problem last time.
>>
>> Do you have any ebtables loaded?

Never heard of them, but making a cursory check just in case..

brad@srv:/raid10/src/linux-2.6.39$ grep EBTABLE .config
# CONFIG_BRIDGE_NF_EBTABLES is not set

>> Can you try building a kernel without ebtables? Without netfilter at all?

Well, without netfilter I can't get it to crash. The problem is without 
netfilter I can't actually use it the way I use it to get it to crash.

I rebooted into a netfilter kernel, and did all the steps I'd used on 
the no-netfilter kernel and it ticked along happily.

So the result of the experiment is inconclusive. Having said that, the 
backtraces certainly smell networky.

To get it to crash, I have to start IE in the VM and https to the public 
address of the machine, which is then redirected by netfilter back into 
another of the VM's.

I can https directly to the other VM's address, but that does not cause 
it to crash, however without netfilter loaded I can't bounce off the 
public IP. It's all rather confusing really.

What next Sherlock?

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
  2011-06-01 10:53                             ` Brad Campbell
@ 2011-06-01 11:09                               ` Avi Kivity
  -1 siblings, 0 replies; 111+ messages in thread
From: Avi Kivity @ 2011-06-01 11:09 UTC (permalink / raw)
  To: Brad Campbell
  Cc: Hugh Dickins, Andrea Arcangeli, Borislav Petkov, linux-kernel,
	kvm, linux-mm, netdev

On 06/01/2011 01:53 PM, Brad Campbell wrote:
> On 01/06/11 17:41, Avi Kivity wrote:
>> On 06/01/2011 12:40 PM, Avi Kivity wrote:
>>>
>>> bridge and netfilter, IIRC this was also the problem last time.
>>>
>>> Do you have any ebtables loaded?
>
> Never heard of them, but making a cursory check just in case..
>
> brad@srv:/raid10/src/linux-2.6.39$ grep EBTABLE .config
> # CONFIG_BRIDGE_NF_EBTABLES is not set
>
>>> Can you try building a kernel without ebtables? Without netfilter at 
>>> all?
>
> Well, without netfilter I can't get it to crash. The problem is 
> without netfilter I can't actually use it the way I use it to get it 
> to crash.
>
> I rebooted into a netfilter kernel, and did all the steps I'd used on 
> the no-netfilter kernel and it ticked along happily.
>
> So the result of the experiment is inconclusive. Having said that, the 
> backtraces certainly smell networky.
>
> To get it to crash, I have to start IE in the VM and https to the 
> public address of the machine, which is then redirected by netfilter 
> back into another of the VM's.
>
> I can https directly to the other VM's address, but that does not 
> cause it to crash, however without netfilter loaded I can't bounce off 
> the public IP. It's all rather confusing really.
>
> What next Sherlock?
>

Maybe the Sherlocks at netdev@ can tell.

-- 
error compiling committee.c: too many arguments to function


^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
@ 2011-06-01 11:09                               ` Avi Kivity
  0 siblings, 0 replies; 111+ messages in thread
From: Avi Kivity @ 2011-06-01 11:09 UTC (permalink / raw)
  To: Brad Campbell
  Cc: Hugh Dickins, Andrea Arcangeli, Borislav Petkov, linux-kernel,
	kvm, linux-mm, netdev

On 06/01/2011 01:53 PM, Brad Campbell wrote:
> On 01/06/11 17:41, Avi Kivity wrote:
>> On 06/01/2011 12:40 PM, Avi Kivity wrote:
>>>
>>> bridge and netfilter, IIRC this was also the problem last time.
>>>
>>> Do you have any ebtables loaded?
>
> Never heard of them, but making a cursory check just in case..
>
> brad@srv:/raid10/src/linux-2.6.39$ grep EBTABLE .config
> # CONFIG_BRIDGE_NF_EBTABLES is not set
>
>>> Can you try building a kernel without ebtables? Without netfilter at 
>>> all?
>
> Well, without netfilter I can't get it to crash. The problem is 
> without netfilter I can't actually use it the way I use it to get it 
> to crash.
>
> I rebooted into a netfilter kernel, and did all the steps I'd used on 
> the no-netfilter kernel and it ticked along happily.
>
> So the result of the experiment is inconclusive. Having said that, the 
> backtraces certainly smell networky.
>
> To get it to crash, I have to start IE in the VM and https to the 
> public address of the machine, which is then redirected by netfilter 
> back into another of the VM's.
>
> I can https directly to the other VM's address, but that does not 
> cause it to crash, however without netfilter loaded I can't bounce off 
> the public IP. It's all rather confusing really.
>
> What next Sherlock?
>

Maybe the Sherlocks at netdev@ can tell.

-- 
error compiling committee.c: too many arguments to function

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
  2011-06-01 10:53                             ` Brad Campbell
@ 2011-06-01 11:18                               ` CaT
  -1 siblings, 0 replies; 111+ messages in thread
From: CaT @ 2011-06-01 11:18 UTC (permalink / raw)
  To: Brad Campbell
  Cc: Avi Kivity, Hugh Dickins, Andrea Arcangeli, Borislav Petkov,
	linux-kernel, kvm, linux-mm, netdev

On Wed, Jun 01, 2011 at 06:53:31PM +0800, Brad Campbell wrote:
> I rebooted into a netfilter kernel, and did all the steps I'd used
> on the no-netfilter kernel and it ticked along happily.
> 
> So the result of the experiment is inconclusive. Having said that,
> the backtraces certainly smell networky.
> 
> To get it to crash, I have to start IE in the VM and https to the
> public address of the machine, which is then redirected by netfilter
> back into another of the VM's.
> 
> I can https directly to the other VM's address, but that does not
> cause it to crash, however without netfilter loaded I can't bounce
> off the public IP. It's all rather confusing really.
> 
> What next Sherlock?

I think you're hitting something I've seen. Can you try rewriting
your firewall rules so that it does not reference any bridge
interfaces at all. Instead, reference the real interface names
in their place. I'm betting it wont crash.

(netdev added to CC since we're aleady bouncing there)

-- 
  "A search of his car uncovered pornography, a homemade sex aid, women's 
  stockings and a Jack Russell terrier."
    - http://www.dailytelegraph.com.au/news/wacky/indeed/story-e6frev20-1111118083480

^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
@ 2011-06-01 11:18                               ` CaT
  0 siblings, 0 replies; 111+ messages in thread
From: CaT @ 2011-06-01 11:18 UTC (permalink / raw)
  To: Brad Campbell
  Cc: Avi Kivity, Hugh Dickins, Andrea Arcangeli, Borislav Petkov,
	linux-kernel, kvm, linux-mm, netdev

On Wed, Jun 01, 2011 at 06:53:31PM +0800, Brad Campbell wrote:
> I rebooted into a netfilter kernel, and did all the steps I'd used
> on the no-netfilter kernel and it ticked along happily.
> 
> So the result of the experiment is inconclusive. Having said that,
> the backtraces certainly smell networky.
> 
> To get it to crash, I have to start IE in the VM and https to the
> public address of the machine, which is then redirected by netfilter
> back into another of the VM's.
> 
> I can https directly to the other VM's address, but that does not
> cause it to crash, however without netfilter loaded I can't bounce
> off the public IP. It's all rather confusing really.
> 
> What next Sherlock?

I think you're hitting something I've seen. Can you try rewriting
your firewall rules so that it does not reference any bridge
interfaces at all. Instead, reference the real interface names
in their place. I'm betting it wont crash.

(netdev added to CC since we're aleady bouncing there)

-- 
  "A search of his car uncovered pornography, a homemade sex aid, women's 
  stockings and a Jack Russell terrier."
    - http://www.dailytelegraph.com.au/news/wacky/indeed/story-e6frev20-1111118083480

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
  2011-06-01 11:18                               ` CaT
@ 2011-06-01 11:52                                 ` Brad Campbell
  -1 siblings, 0 replies; 111+ messages in thread
From: Brad Campbell @ 2011-06-01 11:52 UTC (permalink / raw)
  To: CaT
  Cc: Avi Kivity, Hugh Dickins, Andrea Arcangeli, Borislav Petkov,
	linux-kernel, kvm, linux-mm, netdev

On 01/06/11 19:18, CaT wrote:
> On Wed, Jun 01, 2011 at 06:53:31PM +0800, Brad Campbell wrote:
>> I rebooted into a netfilter kernel, and did all the steps I'd used
>> on the no-netfilter kernel and it ticked along happily.
>>
>> So the result of the experiment is inconclusive. Having said that,
>> the backtraces certainly smell networky.
>>
>> To get it to crash, I have to start IE in the VM and https to the
>> public address of the machine, which is then redirected by netfilter
>> back into another of the VM's.
>>
>> I can https directly to the other VM's address, but that does not
>> cause it to crash, however without netfilter loaded I can't bounce
>> off the public IP. It's all rather confusing really.
>>
>> What next Sherlock?
>
> I think you're hitting something I've seen. Can you try rewriting
> your firewall rules so that it does not reference any bridge
> interfaces at all. Instead, reference the real interface names
> in their place. I'm betting it wont crash.
>

Unfortunately the only interface that is mentioned by name anywhere in 
my firewall is $DMZ (which is ppp0 and not part of any bridge).

All of the nat/dnat and other horrible hacks are based on IP addresses.

^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
@ 2011-06-01 11:52                                 ` Brad Campbell
  0 siblings, 0 replies; 111+ messages in thread
From: Brad Campbell @ 2011-06-01 11:52 UTC (permalink / raw)
  To: CaT
  Cc: Avi Kivity, Hugh Dickins, Andrea Arcangeli, Borislav Petkov,
	linux-kernel, kvm, linux-mm, netdev

On 01/06/11 19:18, CaT wrote:
> On Wed, Jun 01, 2011 at 06:53:31PM +0800, Brad Campbell wrote:
>> I rebooted into a netfilter kernel, and did all the steps I'd used
>> on the no-netfilter kernel and it ticked along happily.
>>
>> So the result of the experiment is inconclusive. Having said that,
>> the backtraces certainly smell networky.
>>
>> To get it to crash, I have to start IE in the VM and https to the
>> public address of the machine, which is then redirected by netfilter
>> back into another of the VM's.
>>
>> I can https directly to the other VM's address, but that does not
>> cause it to crash, however without netfilter loaded I can't bounce
>> off the public IP. It's all rather confusing really.
>>
>> What next Sherlock?
>
> I think you're hitting something I've seen. Can you try rewriting
> your firewall rules so that it does not reference any bridge
> interfaces at all. Instead, reference the real interface names
> in their place. I'm betting it wont crash.
>

Unfortunately the only interface that is mentioned by name anywhere in 
my firewall is $DMZ (which is ppp0 and not part of any bridge).

All of the nat/dnat and other horrible hacks are based on IP addresses.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
  2011-06-01 11:52                                 ` Brad Campbell
@ 2011-06-01 23:03                                   ` CaT
  -1 siblings, 0 replies; 111+ messages in thread
From: CaT @ 2011-06-01 23:03 UTC (permalink / raw)
  To: Brad Campbell
  Cc: Avi Kivity, Hugh Dickins, Andrea Arcangeli, Borislav Petkov,
	linux-kernel, kvm, linux-mm, netdev

On Wed, Jun 01, 2011 at 07:52:33PM +0800, Brad Campbell wrote:
> Unfortunately the only interface that is mentioned by name anywhere
> in my firewall is $DMZ (which is ppp0 and not part of any bridge).
> 
> All of the nat/dnat and other horrible hacks are based on IP addresses.

Damn. Not referencing the bridge interfaces at all stopped our host from
going down in flames when we passed it a few packets. These are two
of the oopses we got from it. Whilst the kernel here is .35 we got the
same issue from a range of kernels. Seems related.

The oopses may be a bit weird. Copy and paste from an ipmi terminal.


slab error in cache_alloc_debugcheck_after(): cache `size-64': double 
free, or n
Pid: 2431, comm: kvm Tainted: G      D     
2.6.35.9-local.20110314-141930 #1
Call Trace:
<IRQ>  [<ffffffff810fb8bf>] ? __slab_error+0x1f/0x30
  [<ffffffff810fc22b>] ? cache_alloc_debugcheck_after+0x6b/0x1f0
  [<ffffffff81530a00>] ? br_nf_pre_routing_finish+0x0/0x370
  [<ffffffff8153106b>] ? br_nf_pre_routing+0x2fb/0x980
  [<ffffffff810fdd3d>] ? kmem_cache_alloc_notrace+0x7d/0xf0

  [<ffffffff8153106b>] ? br_nf_pre_routing+0x2fb/0x980
  [<ffffffff81466e66>] ? nf_iterate+0x66/0xb0
  [<ffffffff8152b9f0>] ? br_handle_frame_finish+0x0/0x1c0
  [<ffffffff81466f14>] ? nf_hook_slow+0x64/0xf0
  [<ffffffff8152b9f0>] ? br_handle_frame_finish+0x0/0x1c0
  [<ffffffff8152bd3c>] ? br_handle_frame+0x18c/0x250
  [<ffffffff81445459>] ? __netif_receive_skb+0x169/0x2a0
  [<ffffffff81445673>] ? process_backlog+0xe3/0x1d0
  [<ffffffff81446347>] ? net_rx_action+0x87/0x1c0
  [<ffffffff810793f7>] ? __do_softirq+0xa7/0x1d0
  [<ffffffff81035b8c>] ? call_softirq+0x1c/0x30
<EOI>  [<ffffffff81037c6d>] ? do_softirq+0x4d/0x80
  [<ffffffff81446b4e>] ? netif_rx_ni+0x1e/0x30
  [<ffffffff8139541a>] ? tun_chr_aio_write+0x36a/0x510
  [<ffffffff813950b0>] ? tun_chr_aio_write+0x0/0x510
  [<ffffffff81102859>] ? do_sync_readv_writev+0xa9/0xf0
  [<ffffffff810973fb>] ? ktime_get+0x5b/0xe0
  [<ffffffff8104f958>] ? lapic_next_event+0x18/0x20
  [<ffffffff8109be18>] ? tick_dev_program_event+0x38/0x100
  [<ffffffff81102697>] ? rw_copy_check_uvector+0x77/0x130
  [<ffffffff81102f0c>] ? do_readv_writev+0xdc/0x200
  [<ffffffff8108dfec>] ? sys_timer_settime+0x13c/0x2e0
  [<ffffffff8110317e>] ? sys_writev+0x4e/0x90
  [<ffffffff81034d6b>] ? system_call_fastpath+0x16/0x1b
ffff8801e7621500: redzone 1:0xbf05bd0100000006, redzone 2:0x9f911029d74e35b

----------

Code: 40 01 00 00 4c 8b a4 24 48 01 00 00 4c 8b ac 24 50 01 00 00 4c 8b 
b4 24 5
RIP  [<ffffffff81652c67>] icmp_send+0x297/0x650
  RSP <ffff880001c036b8>
---[ end trace 9d3f7be7684ac91e ]---
Kernel panic - not syncing: Fatal exception in interrupt
Pid: 0, comm: swapper Tainted: G      D     
2.6.35.9-local.20110314-144920 #2
Call Trace:
<IRQ>  [<ffffffff8170eada>] ? panic+0x94/0x116
  [<ffffffff81711326>] ? _raw_spin_lock_irqsave+0x26/0x40
  [<ffffffff8103a05f>] ? oops_end+0xef/0xf0
  [<ffffffff81711a15>] ? general_protection+0x25/0x30
  [<ffffffff81652c2f>] ? icmp_send+0x25f/0x650
  [<ffffffff81652c67>] ? icmp_send+0x297/0x650
  [<ffffffff815fd8e6>] ? nf_iterate+0x66/0xb0
  [<ffffffff816dbfa0>] ? br_nf_forward_finish+0x0/0x170
  [<ffffffff815fd994>] ? nf_hook_slow+0x64/0xf0
  [<ffffffff816dbfa0>] ? br_nf_forward_finish+0x0/0x170
  [<ffffffff816dc461>] ? br_nf_forward_ip+0x201/0x3e0
  [<ffffffff815fd8e6>] ? nf_iterate+0x66/0xb0
  [<ffffffff816d6620>] ? br_forward_finish+0x0/0x60
  [<ffffffff815fd994>] ? nf_hook_slow+0x64/0xf0
  [<ffffffff816d6620>] ? br_forward_finish+0x0/0x60
  [<ffffffff816d66e9>] ? __br_forward+0x69/0xb0
  [<ffffffff816d741a>] ? br_handle_frame_finish+0x12a/0x280
  [<ffffffff816dcac8>] ? br_nf_pre_routing_finish+0x208/0x370
  [<ffffffff815fd994>] ? nf_hook_slow+0x64/0xf0
  [<ffffffff816dc8c0>] ? br_nf_pre_routing_finish+0x0/0x370
  [<ffffffff816dc538>] ? br_nf_forward_ip+0x2d8/0x3e0
  [<ffffffff816dd3b5>] ? br_nf_pre_routing+0x785/0x980
  [<ffffffff815fd8e6>] ? nf_iterate+0x66/0xb0
  [<ffffffff815fd994>] ? nf_hook_slow+0x64/0xf0
  [<ffffffff816d72f0>] ? br_handle_frame_finish+0x0/0x280
  [<ffffffff815fd994>] ? nf_hook_slow+0x64/0xf0
  [<ffffffff816d72f0>] ? br_handle_frame_finish+0x0/0x280
  [<ffffffff816d76fc>] ? br_handle_frame+0x18c/0x250
  [<ffffffff815dec5b>] ? __netif_receive_skb+0x1cb/0x350
  [<ffffffff8103d115>] ? read_tsc+0x5/0x20
  [<ffffffff815dfa18>] ? netif_receive_skb+0x78/0x80
  [<ffffffff815e0217>] ? napi_gro_receive+0x27/0x40
  [<ffffffff815e01d8>] ? napi_skb_finish+0x38/0x50
  [<ffffffff8152586d>] ? bnx2_poll_work+0xd0d/0x13d0
  [<ffffffff8160c950>] ? ctnetlink_conntrack_event+0x210/0x7d0
  [<ffffffff81092029>] ? autoremove_wake_function+0x9/0x30
  [<ffffffff8109a71b>] ? ktime_get+0x5b/0xe0
  [<ffffffff81526051>] ? bnx2_poll+0x61/0x230
  [<ffffffff81051db8>] ? lapic_next_event+0x18/0x20
  [<ffffffff815dfbef>] ? net_rx_action+0x9f/0x200
  [<ffffffff8109636f>] ? __hrtimer_start_range_ns+0x22f/0x410
  [<ffffffff8107c35f>] ? __do_softirq+0xaf/0x1e0
  [<ffffffff810ab547>] ? handle_IRQ_event+0x47/0x160
  [<ffffffff81036d5c>] ? call_softirq+0x1c/0x30
  [<ffffffff81038c85>] ? do_softirq+0x65/0xa0
  [<ffffffff8107c235>] ? irq_exit+0x85/0x90
  [<ffffffff8103820b>] ? do_IRQ+0x6b/0xe0
  [<ffffffff817117d3>] ? ret_from_intr+0x0/0x11
<EOI>  [<ffffffff81269340>] ? intel_idle+0xf0/0x180
  [<ffffffff81269320>] ? intel_idle+0xd0/0x180
  [<ffffffff815b0b0f>] ? cpuidle_idle_call+0x9f/0x140
  [<ffffffff81035032>] ? cpu_idle+0x62/0xb0
  [<ffffffff81a40c77>] ? start_kernel+0x2ef/0x2fa
  [<ffffffff81a403e3>] ? x86_64_start_kernel+0xfb/0x10a



-- 
  "A search of his car uncovered pornography, a homemade sex aid, women's 
  stockings and a Jack Russell terrier."
    - http://www.dailytelegraph.com.au/news/wacky/indeed/story-e6frev20-1111118083480

^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
@ 2011-06-01 23:03                                   ` CaT
  0 siblings, 0 replies; 111+ messages in thread
From: CaT @ 2011-06-01 23:03 UTC (permalink / raw)
  To: Brad Campbell
  Cc: Avi Kivity, Hugh Dickins, Andrea Arcangeli, Borislav Petkov,
	linux-kernel, kvm, linux-mm, netdev

On Wed, Jun 01, 2011 at 07:52:33PM +0800, Brad Campbell wrote:
> Unfortunately the only interface that is mentioned by name anywhere
> in my firewall is $DMZ (which is ppp0 and not part of any bridge).
> 
> All of the nat/dnat and other horrible hacks are based on IP addresses.

Damn. Not referencing the bridge interfaces at all stopped our host from
going down in flames when we passed it a few packets. These are two
of the oopses we got from it. Whilst the kernel here is .35 we got the
same issue from a range of kernels. Seems related.

The oopses may be a bit weird. Copy and paste from an ipmi terminal.


slab error in cache_alloc_debugcheck_after(): cache `size-64': double 
free, or n
Pid: 2431, comm: kvm Tainted: G      D     
2.6.35.9-local.20110314-141930 #1
Call Trace:
<IRQ>  [<ffffffff810fb8bf>] ? __slab_error+0x1f/0x30
  [<ffffffff810fc22b>] ? cache_alloc_debugcheck_after+0x6b/0x1f0
  [<ffffffff81530a00>] ? br_nf_pre_routing_finish+0x0/0x370
  [<ffffffff8153106b>] ? br_nf_pre_routing+0x2fb/0x980
  [<ffffffff810fdd3d>] ? kmem_cache_alloc_notrace+0x7d/0xf0

  [<ffffffff8153106b>] ? br_nf_pre_routing+0x2fb/0x980
  [<ffffffff81466e66>] ? nf_iterate+0x66/0xb0
  [<ffffffff8152b9f0>] ? br_handle_frame_finish+0x0/0x1c0
  [<ffffffff81466f14>] ? nf_hook_slow+0x64/0xf0
  [<ffffffff8152b9f0>] ? br_handle_frame_finish+0x0/0x1c0
  [<ffffffff8152bd3c>] ? br_handle_frame+0x18c/0x250
  [<ffffffff81445459>] ? __netif_receive_skb+0x169/0x2a0
  [<ffffffff81445673>] ? process_backlog+0xe3/0x1d0
  [<ffffffff81446347>] ? net_rx_action+0x87/0x1c0
  [<ffffffff810793f7>] ? __do_softirq+0xa7/0x1d0
  [<ffffffff81035b8c>] ? call_softirq+0x1c/0x30
<EOI>  [<ffffffff81037c6d>] ? do_softirq+0x4d/0x80
  [<ffffffff81446b4e>] ? netif_rx_ni+0x1e/0x30
  [<ffffffff8139541a>] ? tun_chr_aio_write+0x36a/0x510
  [<ffffffff813950b0>] ? tun_chr_aio_write+0x0/0x510
  [<ffffffff81102859>] ? do_sync_readv_writev+0xa9/0xf0
  [<ffffffff810973fb>] ? ktime_get+0x5b/0xe0
  [<ffffffff8104f958>] ? lapic_next_event+0x18/0x20
  [<ffffffff8109be18>] ? tick_dev_program_event+0x38/0x100
  [<ffffffff81102697>] ? rw_copy_check_uvector+0x77/0x130
  [<ffffffff81102f0c>] ? do_readv_writev+0xdc/0x200
  [<ffffffff8108dfec>] ? sys_timer_settime+0x13c/0x2e0
  [<ffffffff8110317e>] ? sys_writev+0x4e/0x90
  [<ffffffff81034d6b>] ? system_call_fastpath+0x16/0x1b
ffff8801e7621500: redzone 1:0xbf05bd0100000006, redzone 2:0x9f911029d74e35b

----------

Code: 40 01 00 00 4c 8b a4 24 48 01 00 00 4c 8b ac 24 50 01 00 00 4c 8b 
b4 24 5
RIP  [<ffffffff81652c67>] icmp_send+0x297/0x650
  RSP <ffff880001c036b8>
---[ end trace 9d3f7be7684ac91e ]---
Kernel panic - not syncing: Fatal exception in interrupt
Pid: 0, comm: swapper Tainted: G      D     
2.6.35.9-local.20110314-144920 #2
Call Trace:
<IRQ>  [<ffffffff8170eada>] ? panic+0x94/0x116
  [<ffffffff81711326>] ? _raw_spin_lock_irqsave+0x26/0x40
  [<ffffffff8103a05f>] ? oops_end+0xef/0xf0
  [<ffffffff81711a15>] ? general_protection+0x25/0x30
  [<ffffffff81652c2f>] ? icmp_send+0x25f/0x650
  [<ffffffff81652c67>] ? icmp_send+0x297/0x650
  [<ffffffff815fd8e6>] ? nf_iterate+0x66/0xb0
  [<ffffffff816dbfa0>] ? br_nf_forward_finish+0x0/0x170
  [<ffffffff815fd994>] ? nf_hook_slow+0x64/0xf0
  [<ffffffff816dbfa0>] ? br_nf_forward_finish+0x0/0x170
  [<ffffffff816dc461>] ? br_nf_forward_ip+0x201/0x3e0
  [<ffffffff815fd8e6>] ? nf_iterate+0x66/0xb0
  [<ffffffff816d6620>] ? br_forward_finish+0x0/0x60
  [<ffffffff815fd994>] ? nf_hook_slow+0x64/0xf0
  [<ffffffff816d6620>] ? br_forward_finish+0x0/0x60
  [<ffffffff816d66e9>] ? __br_forward+0x69/0xb0
  [<ffffffff816d741a>] ? br_handle_frame_finish+0x12a/0x280
  [<ffffffff816dcac8>] ? br_nf_pre_routing_finish+0x208/0x370
  [<ffffffff815fd994>] ? nf_hook_slow+0x64/0xf0
  [<ffffffff816dc8c0>] ? br_nf_pre_routing_finish+0x0/0x370
  [<ffffffff816dc538>] ? br_nf_forward_ip+0x2d8/0x3e0
  [<ffffffff816dd3b5>] ? br_nf_pre_routing+0x785/0x980
  [<ffffffff815fd8e6>] ? nf_iterate+0x66/0xb0
  [<ffffffff815fd994>] ? nf_hook_slow+0x64/0xf0
  [<ffffffff816d72f0>] ? br_handle_frame_finish+0x0/0x280
  [<ffffffff815fd994>] ? nf_hook_slow+0x64/0xf0
  [<ffffffff816d72f0>] ? br_handle_frame_finish+0x0/0x280
  [<ffffffff816d76fc>] ? br_handle_frame+0x18c/0x250
  [<ffffffff815dec5b>] ? __netif_receive_skb+0x1cb/0x350
  [<ffffffff8103d115>] ? read_tsc+0x5/0x20
  [<ffffffff815dfa18>] ? netif_receive_skb+0x78/0x80
  [<ffffffff815e0217>] ? napi_gro_receive+0x27/0x40
  [<ffffffff815e01d8>] ? napi_skb_finish+0x38/0x50
  [<ffffffff8152586d>] ? bnx2_poll_work+0xd0d/0x13d0
  [<ffffffff8160c950>] ? ctnetlink_conntrack_event+0x210/0x7d0
  [<ffffffff81092029>] ? autoremove_wake_function+0x9/0x30
  [<ffffffff8109a71b>] ? ktime_get+0x5b/0xe0
  [<ffffffff81526051>] ? bnx2_poll+0x61/0x230
  [<ffffffff81051db8>] ? lapic_next_event+0x18/0x20
  [<ffffffff815dfbef>] ? net_rx_action+0x9f/0x200
  [<ffffffff8109636f>] ? __hrtimer_start_range_ns+0x22f/0x410
  [<ffffffff8107c35f>] ? __do_softirq+0xaf/0x1e0
  [<ffffffff810ab547>] ? handle_IRQ_event+0x47/0x160
  [<ffffffff81036d5c>] ? call_softirq+0x1c/0x30
  [<ffffffff81038c85>] ? do_softirq+0x65/0xa0
  [<ffffffff8107c235>] ? irq_exit+0x85/0x90
  [<ffffffff8103820b>] ? do_IRQ+0x6b/0xe0
  [<ffffffff817117d3>] ? ret_from_intr+0x0/0x11
<EOI>  [<ffffffff81269340>] ? intel_idle+0xf0/0x180
  [<ffffffff81269320>] ? intel_idle+0xd0/0x180
  [<ffffffff815b0b0f>] ? cpuidle_idle_call+0x9f/0x140
  [<ffffffff81035032>] ? cpu_idle+0x62/0xb0
  [<ffffffff81a40c77>] ? start_kernel+0x2ef/0x2fa
  [<ffffffff81a403e3>] ? x86_64_start_kernel+0xfb/0x10a



-- 
  "A search of his car uncovered pornography, a homemade sex aid, women's 
  stockings and a Jack Russell terrier."
    - http://www.dailytelegraph.com.au/news/wacky/indeed/story-e6frev20-1111118083480

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
  2011-06-01 23:03                                   ` CaT
@ 2011-06-03 13:38                                     ` Brad Campbell
  -1 siblings, 0 replies; 111+ messages in thread
From: Brad Campbell @ 2011-06-03 13:38 UTC (permalink / raw)
  To: CaT
  Cc: Avi Kivity, Hugh Dickins, Andrea Arcangeli, Borislav Petkov,
	linux-kernel, kvm, linux-mm, netdev

On 02/06/11 07:03, CaT wrote:
> On Wed, Jun 01, 2011 at 07:52:33PM +0800, Brad Campbell wrote:
>> Unfortunately the only interface that is mentioned by name anywhere
>> in my firewall is $DMZ (which is ppp0 and not part of any bridge).
>>
>> All of the nat/dnat and other horrible hacks are based on IP addresses.
>
> Damn. Not referencing the bridge interfaces at all stopped our host from
> going down in flames when we passed it a few packets. These are two
> of the oopses we got from it. Whilst the kernel here is .35 we got the
> same issue from a range of kernels. Seems related.

Well, I tried sending an explanatory message to netdev, netfilter & cc'd 
to kvm, but it appears not to have made it to kvm or netfilter, and the 
cc to netdev has not elicited a response. My resend to netfilter seems 
to have dropped into the bit bucket also.

Is there anyone who can point me at the appropriate cage to rattle? I 
know it appears to be a netfilter issue, but I don't seem to be able to 
get a message to the list (and I am subscribed to it and have been 
getting mail for months) and I'm not sure who to pester. The other 
alternative is I just stop doing "that" and wait for it to bite someone 
else.

Cheers.
Brad

^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
@ 2011-06-03 13:38                                     ` Brad Campbell
  0 siblings, 0 replies; 111+ messages in thread
From: Brad Campbell @ 2011-06-03 13:38 UTC (permalink / raw)
  To: CaT
  Cc: Avi Kivity, Hugh Dickins, Andrea Arcangeli, Borislav Petkov,
	linux-kernel, kvm, linux-mm, netdev

On 02/06/11 07:03, CaT wrote:
> On Wed, Jun 01, 2011 at 07:52:33PM +0800, Brad Campbell wrote:
>> Unfortunately the only interface that is mentioned by name anywhere
>> in my firewall is $DMZ (which is ppp0 and not part of any bridge).
>>
>> All of the nat/dnat and other horrible hacks are based on IP addresses.
>
> Damn. Not referencing the bridge interfaces at all stopped our host from
> going down in flames when we passed it a few packets. These are two
> of the oopses we got from it. Whilst the kernel here is .35 we got the
> same issue from a range of kernels. Seems related.

Well, I tried sending an explanatory message to netdev, netfilter & cc'd 
to kvm, but it appears not to have made it to kvm or netfilter, and the 
cc to netdev has not elicited a response. My resend to netfilter seems 
to have dropped into the bit bucket also.

Is there anyone who can point me at the appropriate cage to rattle? I 
know it appears to be a netfilter issue, but I don't seem to be able to 
get a message to the list (and I am subscribed to it and have been 
getting mail for months) and I'm not sure who to pester. The other 
alternative is I just stop doing "that" and wait for it to bite someone 
else.

Cheers.
Brad

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
  2011-06-03 13:38                                     ` Brad Campbell
  (?)
@ 2011-06-03 15:50                                       ` Bernhard Held
  -1 siblings, 0 replies; 111+ messages in thread
From: Bernhard Held @ 2011-06-03 15:50 UTC (permalink / raw)
  To: linux-kernel; +Cc: linux-mm, kvm, netdev

Am 03.06.2011 15:38, schrieb Brad Campbell:
> On 02/06/11 07:03, CaT wrote:
>> On Wed, Jun 01, 2011 at 07:52:33PM +0800, Brad Campbell wrote:
>>> Unfortunately the only interface that is mentioned by name anywhere
>>> in my firewall is $DMZ (which is ppp0 and not part of any bridge).
>>>
>>> All of the nat/dnat and other horrible hacks are based on IP addresses.
>>
>> Damn. Not referencing the bridge interfaces at all stopped our host from
>> going down in flames when we passed it a few packets. These are two
>> of the oopses we got from it. Whilst the kernel here is .35 we got the
>> same issue from a range of kernels. Seems related.
>
> Well, I tried sending an explanatory message to netdev, netfilter & cc'd to kvm,
> but it appears not to have made it to kvm or netfilter, and the cc to netdev has
> not elicited a response. My resend to netfilter seems to have dropped into the
> bit bucket also.
Just another reference 3.5 months ago:
http://www.spinics.net/lists/netfilter-devel/msg17239.html

Bernhard


^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
@ 2011-06-03 15:50                                       ` Bernhard Held
  0 siblings, 0 replies; 111+ messages in thread
From: Bernhard Held @ 2011-06-03 15:50 UTC (permalink / raw)
  To: kvm; +Cc: linux-mm, linux-kernel, netdev, linux-mm, kvm, netdev

Am 03.06.2011 15:38, schrieb Brad Campbell:
> On 02/06/11 07:03, CaT wrote:
>> On Wed, Jun 01, 2011 at 07:52:33PM +0800, Brad Campbell wrote:
>>> Unfortunately the only interface that is mentioned by name anywhere
>>> in my firewall is $DMZ (which is ppp0 and not part of any bridge).
>>>
>>> All of the nat/dnat and other horrible hacks are based on IP addresses.
>>
>> Damn. Not referencing the bridge interfaces at all stopped our host from
>> going down in flames when we passed it a few packets. These are two
>> of the oopses we got from it. Whilst the kernel here is .35 we got the
>> same issue from a range of kernels. Seems related.
>
> Well, I tried sending an explanatory message to netdev, netfilter & cc'd to kvm,
> but it appears not to have made it to kvm or netfilter, and the cc to netdev has
> not elicited a response. My resend to netfilter seems to have dropped into the
> bit bucket also.
Just another reference 3.5 months ago:
http://www.spinics.net/lists/netfilter-devel/msg17239.html

Bernhard


^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
@ 2011-06-03 15:50                                       ` Bernhard Held
  0 siblings, 0 replies; 111+ messages in thread
From: Bernhard Held @ 2011-06-03 15:50 UTC (permalink / raw)
  To: linux-mm; +Cc: linux-kernel, kvm, netdev

Am 03.06.2011 15:38, schrieb Brad Campbell:
> On 02/06/11 07:03, CaT wrote:
>> On Wed, Jun 01, 2011 at 07:52:33PM +0800, Brad Campbell wrote:
>>> Unfortunately the only interface that is mentioned by name anywhere
>>> in my firewall is $DMZ (which is ppp0 and not part of any bridge).
>>>
>>> All of the nat/dnat and other horrible hacks are based on IP addresses.
>>
>> Damn. Not referencing the bridge interfaces at all stopped our host from
>> going down in flames when we passed it a few packets. These are two
>> of the oopses we got from it. Whilst the kernel here is .35 we got the
>> same issue from a range of kernels. Seems related.
>
> Well, I tried sending an explanatory message to netdev, netfilter & cc'd to kvm,
> but it appears not to have made it to kvm or netfilter, and the cc to netdev has
> not elicited a response. My resend to netfilter seems to have dropped into the
> bit bucket also.
Just another reference 3.5 months ago:
http://www.spinics.net/lists/netfilter-devel/msg17239.html

Bernhard

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
  2011-06-03 15:50                                       ` Bernhard Held
@ 2011-06-03 16:07                                         ` Brad Campbell
  -1 siblings, 0 replies; 111+ messages in thread
From: Brad Campbell @ 2011-06-03 16:07 UTC (permalink / raw)
  To: kvm; +Cc: linux-mm, linux-kernel, netdev, netfilter-devel

On 03/06/11 23:50, Bernhard Held wrote:
> Am 03.06.2011 15:38, schrieb Brad Campbell:
>> On 02/06/11 07:03, CaT wrote:
>>> On Wed, Jun 01, 2011 at 07:52:33PM +0800, Brad Campbell wrote:
>>>> Unfortunately the only interface that is mentioned by name anywhere
>>>> in my firewall is $DMZ (which is ppp0 and not part of any bridge).
>>>>
>>>> All of the nat/dnat and other horrible hacks are based on IP addresses.
>>>
>>> Damn. Not referencing the bridge interfaces at all stopped our host from
>>> going down in flames when we passed it a few packets. These are two
>>> of the oopses we got from it. Whilst the kernel here is .35 we got the
>>> same issue from a range of kernels. Seems related.
>>
>> Well, I tried sending an explanatory message to netdev, netfilter &
>> cc'd to kvm,
>> but it appears not to have made it to kvm or netfilter, and the cc to
>> netdev has
>> not elicited a response. My resend to netfilter seems to have dropped
>> into the
>> bit bucket also.
> Just another reference 3.5 months ago:
> http://www.spinics.net/lists/netfilter-devel/msg17239.html

<waves hands around shouting "I have a reproducible test case for this 
and don't mind patching and crashing the machine to get it fixed">

Attempted to add netfilter-devel to the cc this time.

^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
@ 2011-06-03 16:07                                         ` Brad Campbell
  0 siblings, 0 replies; 111+ messages in thread
From: Brad Campbell @ 2011-06-03 16:07 UTC (permalink / raw)
  To: kvm; +Cc: linux-mm, linux-kernel, netdev, netfilter-devel

On 03/06/11 23:50, Bernhard Held wrote:
>Am 03.06.2011 15:38, schrieb Brad Campbell:
>>On 02/06/11 07:03, CaT wrote:
>>>On Wed, Jun 01, 2011 at 07:52:33PM +0800, Brad Campbell wrote:
>>>>Unfortunately the only interface that is mentioned by name anywhere
>>>>in my firewall is $DMZ (which is ppp0 and not part of any bridge).
>>>>
>>>>All of the nat/dnat and other horrible hacks are based on IP addresses.
>>>
>>>Damn. Not referencing the bridge interfaces at all stopped our host from
>>>going down in flames when we passed it a few packets. These are two
>>>of the oopses we got from it. Whilst the kernel here is .35 we got the
>>>same issue from a range of kernels. Seems related.
>>
>>Well, I tried sending an explanatory message to netdev, netfilter &
>>cc'd to kvm,
>>but it appears not to have made it to kvm or netfilter, and the cc to
>>netdev has
>>not elicited a response. My resend to netfilter seems to have dropped
>>into the
>>bit bucket also.
>Just another reference 3.5 months ago:
>http://www.spinics.net/lists/netfilter-devel/msg17239.html

<waves hands around shouting "I have a reproducible test case for this 
and don't mind patching and crashing the machine to get it fixed">

Attempted to add netfilter-devel to the cc this time.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
  2011-06-03 13:38                                     ` Brad Campbell
@ 2011-06-05  8:14                                       ` Avi Kivity
  -1 siblings, 0 replies; 111+ messages in thread
From: Avi Kivity @ 2011-06-05  8:14 UTC (permalink / raw)
  To: Brad Campbell
  Cc: CaT, Hugh Dickins, Andrea Arcangeli, Borislav Petkov,
	linux-kernel, kvm, linux-mm, netdev

On 06/03/2011 04:38 PM, Brad Campbell wrote:
>
> Is there anyone who can point me at the appropriate cage to rattle? I 
> know it appears to be a netfilter issue, but I don't seem to be able 
> to get a message to the list (and I am subscribed to it and have been 
> getting mail for months) and I'm not sure who to pester. The other 
> alternative is I just stop doing "that" and wait for it to bite 
> someone else.

The mailing list might be set not to send your own mails back to you.  
Check the list archive.

-- 
error compiling committee.c: too many arguments to function


^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
@ 2011-06-05  8:14                                       ` Avi Kivity
  0 siblings, 0 replies; 111+ messages in thread
From: Avi Kivity @ 2011-06-05  8:14 UTC (permalink / raw)
  To: Brad Campbell
  Cc: CaT, Hugh Dickins, Andrea Arcangeli, Borislav Petkov,
	linux-kernel, kvm, linux-mm, netdev

On 06/03/2011 04:38 PM, Brad Campbell wrote:
>
> Is there anyone who can point me at the appropriate cage to rattle? I 
> know it appears to be a netfilter issue, but I don't seem to be able 
> to get a message to the list (and I am subscribed to it and have been 
> getting mail for months) and I'm not sure who to pester. The other 
> alternative is I just stop doing "that" and wait for it to bite 
> someone else.

The mailing list might be set not to send your own mails back to you.  
Check the list archive.

-- 
error compiling committee.c: too many arguments to function

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
  2011-06-05  8:14                                       ` Avi Kivity
@ 2011-06-05 13:45                                         ` Brad Campbell
  -1 siblings, 0 replies; 111+ messages in thread
From: Brad Campbell @ 2011-06-05 13:45 UTC (permalink / raw)
  To: Avi Kivity
  Cc: CaT, Hugh Dickins, Andrea Arcangeli, Borislav Petkov,
	linux-kernel, kvm, linux-mm, netdev

On 05/06/11 16:14, Avi Kivity wrote:
> On 06/03/2011 04:38 PM, Brad Campbell wrote:
>>
>> Is there anyone who can point me at the appropriate cage to rattle? I
>> know it appears to be a netfilter issue, but I don't seem to be able
>> to get a message to the list (and I am subscribed to it and have been
>> getting mail for months) and I'm not sure who to pester. The other
>> alternative is I just stop doing "that" and wait for it to bite
>> someone else.
>
> The mailing list might be set not to send your own mails back to you.
> Check the list archive.

Yep, I did that first..

Given the response to previous issues along the same line, it looks a 
bit like I just remember not to actually use the system in the way that 
triggers the bug and be happy that 99% of the time the kernel does not 
panic, but have that lovely feeling in the back of the skull that says 
"any time now, and without obvious reason the whole machine might just 
come crashing down"..

I guess it's still better than running Xen or Windows..

^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
@ 2011-06-05 13:45                                         ` Brad Campbell
  0 siblings, 0 replies; 111+ messages in thread
From: Brad Campbell @ 2011-06-05 13:45 UTC (permalink / raw)
  To: Avi Kivity
  Cc: CaT, Hugh Dickins, Andrea Arcangeli, Borislav Petkov,
	linux-kernel, kvm, linux-mm, netdev

On 05/06/11 16:14, Avi Kivity wrote:
> On 06/03/2011 04:38 PM, Brad Campbell wrote:
>>
>> Is there anyone who can point me at the appropriate cage to rattle? I
>> know it appears to be a netfilter issue, but I don't seem to be able
>> to get a message to the list (and I am subscribed to it and have been
>> getting mail for months) and I'm not sure who to pester. The other
>> alternative is I just stop doing "that" and wait for it to bite
>> someone else.
>
> The mailing list might be set not to send your own mails back to you.
> Check the list archive.

Yep, I did that first..

Given the response to previous issues along the same line, it looks a 
bit like I just remember not to actually use the system in the way that 
triggers the bug and be happy that 99% of the time the kernel does not 
panic, but have that lovely feeling in the back of the skull that says 
"any time now, and without obvious reason the whole machine might just 
come crashing down"..

I guess it's still better than running Xen or Windows..

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
  2011-06-05 13:45                                         ` Brad Campbell
@ 2011-06-05 13:58                                           ` Avi Kivity
  -1 siblings, 0 replies; 111+ messages in thread
From: Avi Kivity @ 2011-06-05 13:58 UTC (permalink / raw)
  To: Brad Campbell
  Cc: CaT, Hugh Dickins, Andrea Arcangeli, Borislav Petkov,
	linux-kernel, kvm, linux-mm, netdev, netfilter-devel

On 06/05/2011 04:45 PM, Brad Campbell wrote:
>> The mailing list might be set not to send your own mails back to you.
>> Check the list archive.
>
>
> Yep, I did that first..
>
> Given the response to previous issues along the same line, it looks a 
> bit like I just remember not to actually use the system in the way 
> that triggers the bug and be happy that 99% of the time the kernel 
> does not panic, but have that lovely feeling in the back of the skull 
> that says "any time now, and without obvious reason the whole machine 
> might just come crashing down"..
>
> I guess it's still better than running Xen or Windows..

Not at all.  Can some networking/netfilter expert look at this?

Please file a bug with all the relevant information in this thread.  If 
you can look for a previous version that worked, that might increase the 
chances of the bug being resolved faster.

-- 
error compiling committee.c: too many arguments to function


^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
@ 2011-06-05 13:58                                           ` Avi Kivity
  0 siblings, 0 replies; 111+ messages in thread
From: Avi Kivity @ 2011-06-05 13:58 UTC (permalink / raw)
  To: Brad Campbell
  Cc: CaT, Hugh Dickins, Andrea Arcangeli, Borislav Petkov,
	linux-kernel, kvm, linux-mm, netdev, netfilter-devel

On 06/05/2011 04:45 PM, Brad Campbell wrote:
>> The mailing list might be set not to send your own mails back to you.
>> Check the list archive.
>
>
> Yep, I did that first..
>
> Given the response to previous issues along the same line, it looks a 
> bit like I just remember not to actually use the system in the way 
> that triggers the bug and be happy that 99% of the time the kernel 
> does not panic, but have that lovely feeling in the back of the skull 
> that says "any time now, and without obvious reason the whole machine 
> might just come crashing down"..
>
> I guess it's still better than running Xen or Windows..

Not at all.  Can some networking/netfilter expert look at this?

Please file a bug with all the relevant information in this thread.  If 
you can look for a previous version that worked, that might increase the 
chances of the bug being resolved faster.

-- 
error compiling committee.c: too many arguments to function

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
  2011-06-03 16:07                                         ` Brad Campbell
@ 2011-06-06 20:10                                           ` Bart De Schuymer
  -1 siblings, 0 replies; 111+ messages in thread
From: Bart De Schuymer @ 2011-06-06 20:10 UTC (permalink / raw)
  To: Brad Campbell; +Cc: kvm, linux-mm, linux-kernel, netdev, netfilter-devel

Hi Brad,

This has probably nothing to do with ebtables, so please rmmod in case 
it's loaded.
A few questions I didn't directly see an answer to in the threads I 
scanned...
I'm assuming you actually use the bridging firewall functionality. So, 
what iptables modules do you use? Can you reduce your iptables rules to 
a core that triggers the bug?
Or does it get triggered even with an empty set of firewall rules?
Are you using a stock .35 kernel or is it patched?
Is this something I can trigger on a poor guy's laptop or does it 
require specialized hardware (I'm catching up on qemu/kvm...)?

cheers,
Bart

PS: I'm not sure if we should keep CC-ing everybody, netfilter-devel 
together with kvm should probably do fine.

Op 3/06/2011 18:07, Brad Campbell schreef:
> On 03/06/11 23:50, Bernhard Held wrote:
>> Am 03.06.2011 15:38, schrieb Brad Campbell:
>>> On 02/06/11 07:03, CaT wrote:
>>>> On Wed, Jun 01, 2011 at 07:52:33PM +0800, Brad Campbell wrote:
>>>>> Unfortunately the only interface that is mentioned by name anywhere
>>>>> in my firewall is $DMZ (which is ppp0 and not part of any bridge).
>>>>>
>>>>> All of the nat/dnat and other horrible hacks are based on IP 
>>>>> addresses.
>>>>
>>>> Damn. Not referencing the bridge interfaces at all stopped our host 
>>>> from
>>>> going down in flames when we passed it a few packets. These are two
>>>> of the oopses we got from it. Whilst the kernel here is .35 we got the
>>>> same issue from a range of kernels. Seems related.
>>>
>>> Well, I tried sending an explanatory message to netdev, netfilter &
>>> cc'd to kvm,
>>> but it appears not to have made it to kvm or netfilter, and the cc to
>>> netdev has
>>> not elicited a response. My resend to netfilter seems to have dropped
>>> into the
>>> bit bucket also.
>> Just another reference 3.5 months ago:
>> http://www.spinics.net/lists/netfilter-devel/msg17239.html
>
> <waves hands around shouting "I have a reproducible test case for this 
> and don't mind patching and crashing the machine to get it fixed">
>
> Attempted to add netfilter-devel to the cc this time.
> -- 
> To unsubscribe from this list: send the line "unsubscribe 
> netfilter-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>


-- 
Bart De Schuymer
www.artinalgorithms.be


^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
@ 2011-06-06 20:10                                           ` Bart De Schuymer
  0 siblings, 0 replies; 111+ messages in thread
From: Bart De Schuymer @ 2011-06-06 20:10 UTC (permalink / raw)
  To: Brad Campbell; +Cc: kvm, linux-mm, linux-kernel, netdev, netfilter-devel

Hi Brad,

This has probably nothing to do with ebtables, so please rmmod in case 
it's loaded.
A few questions I didn't directly see an answer to in the threads I 
scanned...
I'm assuming you actually use the bridging firewall functionality. So, 
what iptables modules do you use? Can you reduce your iptables rules to 
a core that triggers the bug?
Or does it get triggered even with an empty set of firewall rules?
Are you using a stock .35 kernel or is it patched?
Is this something I can trigger on a poor guy's laptop or does it 
require specialized hardware (I'm catching up on qemu/kvm...)?

cheers,
Bart

PS: I'm not sure if we should keep CC-ing everybody, netfilter-devel 
together with kvm should probably do fine.

Op 3/06/2011 18:07, Brad Campbell schreef:
> On 03/06/11 23:50, Bernhard Held wrote:
>> Am 03.06.2011 15:38, schrieb Brad Campbell:
>>> On 02/06/11 07:03, CaT wrote:
>>>> On Wed, Jun 01, 2011 at 07:52:33PM +0800, Brad Campbell wrote:
>>>>> Unfortunately the only interface that is mentioned by name anywhere
>>>>> in my firewall is $DMZ (which is ppp0 and not part of any bridge).
>>>>>
>>>>> All of the nat/dnat and other horrible hacks are based on IP 
>>>>> addresses.
>>>>
>>>> Damn. Not referencing the bridge interfaces at all stopped our host 
>>>> from
>>>> going down in flames when we passed it a few packets. These are two
>>>> of the oopses we got from it. Whilst the kernel here is .35 we got the
>>>> same issue from a range of kernels. Seems related.
>>>
>>> Well, I tried sending an explanatory message to netdev, netfilter &
>>> cc'd to kvm,
>>> but it appears not to have made it to kvm or netfilter, and the cc to
>>> netdev has
>>> not elicited a response. My resend to netfilter seems to have dropped
>>> into the
>>> bit bucket also.
>> Just another reference 3.5 months ago:
>> http://www.spinics.net/lists/netfilter-devel/msg17239.html
>
> <waves hands around shouting "I have a reproducible test case for this 
> and don't mind patching and crashing the machine to get it fixed">
>
> Attempted to add netfilter-devel to the cc this time.
> -- 
> To unsubscribe from this list: send the line "unsubscribe 
> netfilter-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>


-- 
Bart De Schuymer
www.artinalgorithms.be

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
  2011-06-05 13:45                                         ` Brad Campbell
  (?)
@ 2011-06-06 20:22                                           ` Eric Dumazet
  -1 siblings, 0 replies; 111+ messages in thread
From: Eric Dumazet @ 2011-06-06 20:22 UTC (permalink / raw)
  To: Brad Campbell
  Cc: Avi Kivity, CaT, Hugh Dickins, Andrea Arcangeli, Borislav Petkov,
	linux-kernel, kvm, linux-mm, netdev

Le dimanche 05 juin 2011 à 21:45 +0800, Brad Campbell a écrit :
> On 05/06/11 16:14, Avi Kivity wrote:
> > On 06/03/2011 04:38 PM, Brad Campbell wrote:
> >>
> >> Is there anyone who can point me at the appropriate cage to rattle? I
> >> know it appears to be a netfilter issue, but I don't seem to be able
> >> to get a message to the list (and I am subscribed to it and have been
> >> getting mail for months) and I'm not sure who to pester. The other
> >> alternative is I just stop doing "that" and wait for it to bite
> >> someone else.
> >
> > The mailing list might be set not to send your own mails back to you.
> > Check the list archive.
> 
> Yep, I did that first..
> 
> Given the response to previous issues along the same line, it looks a 
> bit like I just remember not to actually use the system in the way that 
> triggers the bug and be happy that 99% of the time the kernel does not 
> panic, but have that lovely feeling in the back of the skull that says 
> "any time now, and without obvious reason the whole machine might just 
> come crashing down"..
> 
> I guess it's still better than running Xen or Windows..

Could you please try latest linux-2.6 tree ?

We fixed many networking bugs that could explain your crash.




^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
@ 2011-06-06 20:22                                           ` Eric Dumazet
  0 siblings, 0 replies; 111+ messages in thread
From: Eric Dumazet @ 2011-06-06 20:22 UTC (permalink / raw)
  To: Brad Campbell
  Cc: Avi Kivity, CaT, Hugh Dickins, Andrea Arcangeli, Borislav Petkov,
	linux-kernel, kvm, linux-mm, netdev

Le dimanche 05 juin 2011 à 21:45 +0800, Brad Campbell a écrit :
> On 05/06/11 16:14, Avi Kivity wrote:
> > On 06/03/2011 04:38 PM, Brad Campbell wrote:
> >>
> >> Is there anyone who can point me at the appropriate cage to rattle? I
> >> know it appears to be a netfilter issue, but I don't seem to be able
> >> to get a message to the list (and I am subscribed to it and have been
> >> getting mail for months) and I'm not sure who to pester. The other
> >> alternative is I just stop doing "that" and wait for it to bite
> >> someone else.
> >
> > The mailing list might be set not to send your own mails back to you.
> > Check the list archive.
> 
> Yep, I did that first..
> 
> Given the response to previous issues along the same line, it looks a 
> bit like I just remember not to actually use the system in the way that 
> triggers the bug and be happy that 99% of the time the kernel does not 
> panic, but have that lovely feeling in the back of the skull that says 
> "any time now, and without obvious reason the whole machine might just 
> come crashing down"..
> 
> I guess it's still better than running Xen or Windows..

Could you please try latest linux-2.6 tree ?

We fixed many networking bugs that could explain your crash.



--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
@ 2011-06-06 20:22                                           ` Eric Dumazet
  0 siblings, 0 replies; 111+ messages in thread
From: Eric Dumazet @ 2011-06-06 20:22 UTC (permalink / raw)
  To: Brad Campbell
  Cc: Avi Kivity, CaT, Hugh Dickins, Andrea Arcangeli, Borislav Petkov,
	linux-kernel, kvm, linux-mm, netdev

Le dimanche 05 juin 2011 A  21:45 +0800, Brad Campbell a A(C)crit :
> On 05/06/11 16:14, Avi Kivity wrote:
> > On 06/03/2011 04:38 PM, Brad Campbell wrote:
> >>
> >> Is there anyone who can point me at the appropriate cage to rattle? I
> >> know it appears to be a netfilter issue, but I don't seem to be able
> >> to get a message to the list (and I am subscribed to it and have been
> >> getting mail for months) and I'm not sure who to pester. The other
> >> alternative is I just stop doing "that" and wait for it to bite
> >> someone else.
> >
> > The mailing list might be set not to send your own mails back to you.
> > Check the list archive.
> 
> Yep, I did that first..
> 
> Given the response to previous issues along the same line, it looks a 
> bit like I just remember not to actually use the system in the way that 
> triggers the bug and be happy that 99% of the time the kernel does not 
> panic, but have that lovely feeling in the back of the skull that says 
> "any time now, and without obvious reason the whole machine might just 
> come crashing down"..
> 
> I guess it's still better than running Xen or Windows..

Could you please try latest linux-2.6 tree ?

We fixed many networking bugs that could explain your crash.



--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
  2011-06-06 20:10                                           ` Bart De Schuymer
  (?)
@ 2011-06-06 20:23                                             ` Eric Dumazet
  -1 siblings, 0 replies; 111+ messages in thread
From: Eric Dumazet @ 2011-06-06 20:23 UTC (permalink / raw)
  To: Bart De Schuymer
  Cc: Brad Campbell, kvm, linux-mm, linux-kernel, netdev, netfilter-devel

Le lundi 06 juin 2011 à 22:10 +0200, Bart De Schuymer a écrit :
> Hi Brad,
> 
> This has probably nothing to do with ebtables, so please rmmod in case 
> it's loaded.
> A few questions I didn't directly see an answer to in the threads I 
> scanned...
> I'm assuming you actually use the bridging firewall functionality. So, 
> what iptables modules do you use? Can you reduce your iptables rules to 
> a core that triggers the bug?
> Or does it get triggered even with an empty set of firewall rules?
> Are you using a stock .35 kernel or is it patched?
> Is this something I can trigger on a poor guy's laptop or does it 
> require specialized hardware (I'm catching up on qemu/kvm...)?
> 

Keep netdev, as this most probably is a networking bug.




^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
@ 2011-06-06 20:23                                             ` Eric Dumazet
  0 siblings, 0 replies; 111+ messages in thread
From: Eric Dumazet @ 2011-06-06 20:23 UTC (permalink / raw)
  To: Bart De Schuymer
  Cc: Brad Campbell, kvm, linux-mm, linux-kernel, netdev, netfilter-devel

Le lundi 06 juin 2011 à 22:10 +0200, Bart De Schuymer a écrit :
> Hi Brad,
> 
> This has probably nothing to do with ebtables, so please rmmod in case 
> it's loaded.
> A few questions I didn't directly see an answer to in the threads I 
> scanned...
> I'm assuming you actually use the bridging firewall functionality. So, 
> what iptables modules do you use? Can you reduce your iptables rules to 
> a core that triggers the bug?
> Or does it get triggered even with an empty set of firewall rules?
> Are you using a stock .35 kernel or is it patched?
> Is this something I can trigger on a poor guy's laptop or does it 
> require specialized hardware (I'm catching up on qemu/kvm...)?
> 

Keep netdev, as this most probably is a networking bug.



--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
@ 2011-06-06 20:23                                             ` Eric Dumazet
  0 siblings, 0 replies; 111+ messages in thread
From: Eric Dumazet @ 2011-06-06 20:23 UTC (permalink / raw)
  To: Bart De Schuymer
  Cc: Brad Campbell, kvm, linux-mm, linux-kernel, netdev, netfilter-devel

Le lundi 06 juin 2011 A  22:10 +0200, Bart De Schuymer a A(C)crit :
> Hi Brad,
> 
> This has probably nothing to do with ebtables, so please rmmod in case 
> it's loaded.
> A few questions I didn't directly see an answer to in the threads I 
> scanned...
> I'm assuming you actually use the bridging firewall functionality. So, 
> what iptables modules do you use? Can you reduce your iptables rules to 
> a core that triggers the bug?
> Or does it get triggered even with an empty set of firewall rules?
> Are you using a stock .35 kernel or is it patched?
> Is this something I can trigger on a poor guy's laptop or does it 
> require specialized hardware (I'm catching up on qemu/kvm...)?
> 

Keep netdev, as this most probably is a networking bug.



--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
  2011-06-06 20:10                                           ` Bart De Schuymer
@ 2011-06-07  3:33                                             ` Brad Campbell
  -1 siblings, 0 replies; 111+ messages in thread
From: Brad Campbell @ 2011-06-07  3:33 UTC (permalink / raw)
  To: Bart De Schuymer; +Cc: kvm, linux-mm, linux-kernel, netdev, netfilter-devel

On 07/06/11 04:10, Bart De Schuymer wrote:
> Hi Brad,
>
> This has probably nothing to do with ebtables, so please rmmod in case
> it's loaded.
> A few questions I didn't directly see an answer to in the threads I
> scanned...
> I'm assuming you actually use the bridging firewall functionality. So,
> what iptables modules do you use? Can you reduce your iptables rules to
> a core that triggers the bug?
> Or does it get triggered even with an empty set of firewall rules?
> Are you using a stock .35 kernel or is it patched?
> Is this something I can trigger on a poor guy's laptop or does it
> require specialized hardware (I'm catching up on qemu/kvm...)?

Not specialised hardware as such, I've just not been able to reproduce 
it outside of this specific operating scenario.

I can't trigger it with empty firewall rules as it relies on a DNAT to 
occur. If I try it directly to the internal IP address (as I have to 
without netfilter loaded) then of course nothing fails.

It's a pain in the bum as a fault, but it's one I can easily reproduce 
as long as I use the same set of circumstances.

I'll try using 3.0-rc2 (current git) tonight, and if I can reproduce it 
on that then I'll attempt to pare down the IPTABLES rules to a bare minimum.

It is nothing to do with ebtables as I don't compile it. I'm not really 
sure about "bridging firewall" functionality. I just use a couple of 
hand coded bash scripts to set the tables up.

brad@srv:~$ lsmod
Module                  Size  Used by
xt_iprange              1637  1
xt_DSCP                 2077  2
xt_length               1216  1
xt_CLASSIFY             1091  26
sch_sfq                 6681  4
xt_CHECKSUM             1229  2 brad@srv:~$ lsmod
Module                  Size  Used by
xt_iprange              1637  1
xt_DSCP                 2077  2
xt_length               1216  1
xt_CLASSIFY             1091  26
sch_sfq                 6681  4
xt_CHECKSUM             1229  2
ipt_REJECT              2277  1
ipt_MASQUERADE          1759  7
ipt_REDIRECT            1133  1
xt_recent               8223  2
xt_state                1226  5
iptable_nat             3993  1
nf_nat                 16773  3 ipt_MASQUERADE,ipt_REDIRECT,iptable_nat
nf_conntrack_ipv4      11868  8 iptable_nat,nf_nat
nf_conntrack           60962  5 
ipt_MASQUERADE,xt_state,iptable_nat,nf_nat,nf_conntrack_ipv4
nf_defrag_ipv4          1417  1 nf_conntrack_ipv4
xt_TCPMSS               2567  2
xt_tcpmss               1469  0
xt_tcpudp               2467  56
iptable_mangle          1487  1
pppoe                   9574  2
pppox                   2188  1 pppoe
iptable_filter          1442  1
ip_tables              16762  3 iptable_nat,iptable_mangle,iptable_filter
x_tables               20462  17 
xt_iprange,xt_DSCP,xt_length,xt_CLASSIFY,xt_CHECKSUM,ipt_REJECT,ipt_MASQUERADE,ipt_REDIRECT,xt_recent,xt_state,iptable_nat,xt_TCPMSS,xt_tcpmss,xt_tcpudp,iptable_mangle,iptable_filter,ip_tables
ppp_generic            24243  6 pppoe,pppox
slhc                    5293  1 ppp_generic
cls_u32                 6468  6
sch_htb                14432  2
deflate                 1937  0
zlib_deflate           21228  1 deflate
des_generic            16135  0
cbc                     2721  0
ecb                     1975  0
crypto_blkcipher       13645  2 cbc,ecb
sha1_generic            2095  0
md5                     4001  0
hmac                    2977  0
crypto_hash            14519  3 sha1_generic,md5,hmac
cryptomgr               2636  0
aead                    6137  1 cryptomgr
crypto_algapi          15289  9 
deflate,des_generic,cbc,ecb,crypto_blkcipher,hmac,crypto_hash,cryptomgr,aead
af_key                 27372  0
fuse                   66747  1
w83627ehf              32052  0
hwmon_vid               2867  1 w83627ehf
vhost_net              16802  6
powernow_k8            12932  0
mperf                   1263  1 powernow_k8
kvm_amd                53431  24
kvm                   235155  1 kvm_amd
pl2303                 12732  1
xhci_hcd               62865  0
i2c_piix4               8391  0
k10temp                 3183  0
usbserial              34452  3 pl2303
usb_storage            37887  1
usb_libusual           10999  1 usb_storage
ohci_hcd               18105  0
ehci_hcd               33641  0
ahci                   20748  4
usbcore               130936  7 
pl2303,xhci_hcd,usbserial,usb_storage,usb_libusual,ohci_hcd,ehci_hcd
libahci                21202  1 ahci
sata_mv                26939  0
megaraid_sas           71659  14

Nat Table (external ip substituted for xxx.xxx.xxx.xxx)

Chain PREROUTING (policy ACCEPT 1761K packets, 152M bytes)
  pkts bytes target     prot opt in     out     source 
destination
     5   210 DNAT       udp  --  ppp0   *       0.0.0.0/0 
0.0.0.0/0           udp dpt:1195 to:192.168.253.199
     6   252 DNAT       udp  --  !ppp0  *       0.0.0.0/0 
xxx.xxx.xxx.xxx       udp dpt:1195 to:192.168.253.199
     0     0 DNAT       tcp  --  ppp0   *       0.0.0.0/0 
0.0.0.0/0           tcp dpt:25001 to:192.168.253.199:465
     0     0 DNAT       tcp  --  ppp0   *       0.0.0.0/0 
0.0.0.0/0           tcp dpt:25000 to:192.168.253.199:993
     0     0 DNAT       tcp  --  !ppp0  *       0.0.0.0/0 
xxx.xxx.xxx.xxx       tcp dpt:25001 to:192.168.253.199:465
     0     0 DNAT       tcp  --  !ppp0  *       0.0.0.0/0 
xxx.xxx.xxx.xxx       tcp dpt:25000 to:192.168.253.199:993
     2   142 DNAT       47   --  ppp0   *       0.0.0.0/0 
0.0.0.0/0           to:192.168.253.199
    18   880 DNAT       tcp  --  ppp0   *       0.0.0.0/0 
0.0.0.0/0           tcp dpt:1723 to:192.168.253.199
     0     0 DNAT       47   --  !ppp0  *       0.0.0.0/0 
xxx.xxx.xxx.xxx       to:192.168.253.199
     0     0 DNAT       tcp  --  !ppp0  *       0.0.0.0/0 
xxx.xxx.xxx.xxx       tcp dpt:1723 to:192.168.253.199
  2969  149K DNAT       tcp  --  ppp0   *       0.0.0.0/0 
0.0.0.0/0           tcp dpt:443 to:192.168.253.198
    20  1280 DNAT       tcp  --  !ppp0  *       0.0.0.0/0 
xxx.xxx.xxx.xxx       tcp dpt:443 to:192.168.253.198
     0     0 DNAT       tcp  --  ppp0   *       0.0.0.0/0 
0.0.0.0/0           tcp dpt:3101 to:192.168.253.197
     0     0 DNAT       tcp  --  !ppp0  *       0.0.0.0/0 
xxx.xxx.xxx.xxx       tcp dpt:3101 to:192.168.253.197
     0     0 DNAT       tcp  --  !ppp0  *       0.0.0.0/0 
xxx.xxx.xxx.xxx       tcp dpt:4101 to:192.168.253.197
44528 2718K REDIRECT   tcp  --  !ppp0  *       0.0.0.0/0 
!192.168.0.0/16      tcp dpt:80 redir ports 8080
     0     0 DNAT       tcp  --  *      *       0.0.0.0/0 
0.0.0.0/0           tcp dpt:3724 to:192.168.2.107
  596K   33M DNAT       tcp  --  ppp0   *       0.0.0.0/0 
0.0.0.0/0           tcp dpts:2001:2030 to:10.99.99.2
1420K  119M DNAT       udp  --  ppp0   *       0.0.0.0/0 
0.0.0.0/0           udp dpts:2001:2030 to:10.99.99.2
  7483  449K DNAT       all  --  !ppp0  *       0.0.0.0/0 
xxx.xxx.xxx.xxx       to:192.168.2.1


Mangle Table

Chain INPUT (policy ACCEPT 270K packets, 17M bytes)
  pkts bytes target     prot opt in     out     source 
destination

Chain OUTPUT (policy ACCEPT 170K packets, 12M bytes)
  pkts bytes target     prot opt in     out     source 
destination

Chain POSTROUTING (policy ACCEPT 2205K packets, 166M bytes)
  pkts bytes target     prot opt in     out     source 
destination
     0     0 MASQUERADE  all  --  *      *       0.0.0.0/0 
192.168.254.3
     6   360 ACCEPT     all  --  *      *       0.0.0.0/0 
xxx.xxx.xxx.xxx
20424 2120K MASQUERADE  all  --  *      ppp0    192.168.0.0/16 
!192.168.0.0/16
     0     0 MASQUERADE  all  --  *      ppp0    10.0.0.0/24 
0.0.0.0/0
     3   204 MASQUERADE  all  --  *      *       192.168.2.0/24 
10.8.0.0/24
1418K  128M MASQUERADE  all  --  *      *       10.99.99.0/24 
0.0.0.0/0
68248 4095K MASQUERADE  all  --  *      *       192.168.253.0/24 
10.8.0.0/16
13305 2405K MASQUERADE  all  --  *      *       192.168.253.0/24 
!192.168.0.0/16

Chain PREROUTING (policy ACCEPT 278M packets, 293G bytes)
  pkts bytes target     prot opt in     out     source 
destination
   169 55528 CHECKSUM   udp  --  br1    *       0.0.0.0/0 
0.0.0.0/0           udp dpt:67 CHECKSUM fill

Chain INPUT (policy ACCEPT 180M packets, 250G bytes)
  pkts bytes target     prot opt in     out     source 
destination

Chain FORWARD (policy ACCEPT 98M packets, 44G bytes)
  pkts bytes target     prot opt in     out     source 
destination

Chain OUTPUT (policy ACCEPT 155M packets, 180G bytes)
  pkts bytes target     prot opt in     out     source 
destination

Chain POSTROUTING (policy ACCEPT 253M packets, 223G bytes)
  pkts bytes target     prot opt in     out     source 
destination
   165 54182 CHECKSUM   udp  --  *      br1     0.0.0.0/0 
0.0.0.0/0           udp spt:67 CHECKSUM fill
    51  3712 CLASSIFY   tcp  --  *      ppp0    0.0.0.0/0 
0.0.0.0/0           tcp dpt:53 CLASSIFY set 1:20
85274 6454K CLASSIFY   udp  --  *      ppp0    0.0.0.0/0 
0.0.0.0/0           udp dpt:53 CLASSIFY set 1:20
   187  257K CLASSIFY   tcp  --  *      ppp0    0.0.0.0/0 
0.0.0.0/0           tcp spt:81 CLASSIFY set 1:20
   25M 1180M CLASSIFY   tcp  --  *      ppp0    0.0.0.0/0 
0.0.0.0/0           tcp flags:0x3F/0x10 state ESTABLISHED length 40:100 
CLASSIFY set 1:15
  728K   67M CLASSIFY   icmp --  *      ppp0    0.0.0.0/0 
0.0.0.0/0           CLASSIFY set 1:15
   231 23484 CLASSIFY   tcp  --  *      ppp0    0.0.0.0/0 
0.0.0.0/0           tcp dpt:2401 CLASSIFY set 1:15
65636 5610K CLASSIFY   tcp  --  *      ppp0    0.0.0.0/0 
0.0.0.0/0           tcp dpt:22 CLASSIFY set 1:10
  2018  315K CLASSIFY   tcp  --  *      ppp0    0.0.0.0/0 
0.0.0.0/0           tcp spt:22 CLASSIFY set 1:10
    80 10092 CLASSIFY   tcp  --  *      ppp0    0.0.0.0/0 
0.0.0.0/0           tcp dpt:3389 CLASSIFY set 1:10
26063 8910K CLASSIFY   tcp  --  *      ppp0    0.0.0.0/0 
0.0.0.0/0           tcp dpt:8080 CLASSIFY set 1:15
  932K  131M CLASSIFY   tcp  --  *      ppp0    0.0.0.0/0 
0.0.0.0/0           tcp dpt:80 CLASSIFY set 1:15
  3511  267K CLASSIFY   udp  --  *      ppp0    0.0.0.0/0 
0.0.0.0/0           udp dpt:123 CLASSIFY set 1:10
     0     0 CLASSIFY   tcp  --  *      ppp0    0.0.0.0/0 
0.0.0.0/0           tcp spt:20 CLASSIFY set 1:15
     3   180 CLASSIFY   tcp  --  *      ppp0    0.0.0.0/0 
0.0.0.0/0           tcp dpt:20 CLASSIFY set 1:15
94058   38M CLASSIFY   47   --  *      ppp0    0.0.0.0/0 
0.0.0.0/0           CLASSIFY set 1:10
1086K  183M CLASSIFY   udp  --  *      ppp0    0.0.0.0/0 
0.0.0.0/0           udp spt:1194 CLASSIFY set 1:10
1086K  183M TOS        udp  --  *      ppp0    0.0.0.0/0 
0.0.0.0/0           udp spt:1194 TOS set 0x10/0x3f
48817   10M CLASSIFY   udp  --  *      ppp0    0.0.0.0/0 
0.0.0.0/0           udp spt:1195 CLASSIFY set 1:10
48817   10M TOS        udp  --  *      ppp0    0.0.0.0/0 
0.0.0.0/0           udp spt:1195 TOS set 0x10/0x3f
94058   38M CLASSIFY   47   --  *      ppp0    0.0.0.0/0 
0.0.0.0/0           CLASSIFY set 1:15
   106  7207 CLASSIFY   tcp  --  *      ppp0    0.0.0.0/0 
0.0.0.0/0           tcp dpt:1863 CLASSIFY set 1:15
  188K   34M CLASSIFY   tcp  --  *      ppp0    0.0.0.0/0 
0.0.0.0/0           tcp dpt:443 CLASSIFY set 1:15
51541 3327K CLASSIFY   tcp  --  *      ppp0    0.0.0.0/0 
0.0.0.0/0           tcp dpts:6660:6669 CLASSIFY set 1:15
     0     0 CLASSIFY   tcp  --  *      ppp0    0.0.0.0/0 
0.0.0.0/0           tcp spts:2021:2030 CLASSIFY set 1:15
    85  4944 CLASSIFY   tcp  --  *      ppp0    0.0.0.0/0 
0.0.0.0/0           tcp dpt:19999 CLASSIFY set 1:15
  208K   86M CLASSIFY   udp  --  *      *       0.0.0.0/0 
0.0.0.0/0           source IP range 192.168.2.80-192.168.2.120 CLASSIFY 
set 1:10
     0     0 CLASSIFY   tcp  --  *      ppp0    0.0.0.0/0 
0.0.0.0/0           tcp spt:12345 CLASSIFY set 1:15
     1    80 CLASSIFY   udp  --  *      ppp0    0.0.0.0/0 
0.0.0.0/0           udp spt:12345 CLASSIFY set 1:15


Default table

Chain INPUT (policy ACCEPT 176M packets, 247G bytes)
  pkts bytes target     prot opt in     out     source 
destination
     0     0 ACCEPT     udp  --  ppp0   *       0.0.0.0/0 
0.0.0.0/0           udp dpt:4569
1187K  582M ACCEPT     udp  --  ppp0   *       0.0.0.0/0 
0.0.0.0/0           udp dpt:1194
     2   577 ACCEPT     udp  --  ppp0   *       0.0.0.0/0 
0.0.0.0/0           udp dpt:1195
    28  1224 ACCEPT     tcp  --  ppp0   *       0.0.0.0/0 
0.0.0.0/0           tcp dpt:3389
   230 12372            tcp  --  ppp0   *       0.0.0.0/0 
0.0.0.0/0           tcp dpt:22 state NEW recent: SET name: DEFAULT side: 
source
     3   180 DROP       tcp  --  ppp0   *       0.0.0.0/0 
0.0.0.0/0           tcp dpt:22 state NEW recent: UPDATE seconds: 300 
hit_count: 4 name: DEFAULT side: source
  1750  143K ACCEPT     tcp  --  ppp0   *       0.0.0.0/0 
0.0.0.0/0           tcp dpt:22
     3   144 ACCEPT     tcp  --  ppp0   *       0.0.0.0/0 
0.0.0.0/0           tcp dpt:113
   120  6090 ACCEPT     tcp  --  ppp0   *       0.0.0.0/0 
0.0.0.0/0           tcp dpt:81
36094   29M ACCEPT     tcp  --  ppp0   *       0.0.0.0/0 
0.0.0.0/0           tcp dpt:25
1456K 1706M ACCEPT     all  --  ppp0   *       0.0.0.0/0 
0.0.0.0/0           state RELATED,ESTABLISHED
31047 2334K REJECT     tcp  --  ppp0   *       0.0.0.0/0 
0.0.0.0/0           tcp option=!2 reject-with tcp-reset
  552K   60M ACCEPT     all  --  !ppp0  *       0.0.0.0/0 
0.0.0.0/0           state NEW
13552 1207K ACCEPT     icmp --  ppp0   *       0.0.0.0/0 
0.0.0.0/0
  5712  392K DROP       all  --  ppp0   *       0.0.0.0/0 
0.0.0.0/0

Chain FORWARD (policy ACCEPT 98M packets, 44G bytes)
  pkts bytes target     prot opt in     out     source 
destination
1207K   68M TCPMSS     tcp  --  *      ppp0    0.0.0.0/0 
0.0.0.0/0           tcp flags:0x06/0x02 TCPMSS clamp to PMTU

Chain OUTPUT (policy ACCEPT 155M packets, 180G bytes)
  pkts bytes target     prot opt in     out     source 
destination
31675 1895K TCPMSS     tcp  --  *      ppp0    0.0.0.0/0 
0.0.0.0/0           tcp flags:0x06/0x02 TCPMSS clamp to PMTU

lsmod

ipt_REJECT              2277  1
ipt_MASQUERADE          1759  7
ipt_REDIRECT            1133  1
xt_recent               8223  2
xt_state                1226  5
iptable_nat             3993  1
nf_nat                 16773  3 ipt_MASQUERADE,ipt_REDIRECT,iptable_nat
nf_conntrack_ipv4      11868  8 iptable_nat,nf_nat
nf_conntrack           60962  5 
ipt_MASQUERADE,xt_state,iptable_nat,nf_nat,nf_conntrack_ipv4
nf_defrag_ipv4          1417  1 nf_conntrack_ipv4
xt_TCPMSS               2567  2
xt_tcpmss               1469  0
xt_tcpudp               2467  56
iptable_mangle          1487  1
pppoe                   9574  2
pppox                   2188  1 pppoe
iptable_filter          1442  1
ip_tables              16762  3 iptable_nat,iptable_mangle,iptable_filter
x_tables               20462  17 
xt_iprange,xt_DSCP,xt_length,xt_CLASSIFY,xt_CHECKSUM,ipt_REJECT,ipt_MASQUERADE,ipt_REDIRECT,xt_recent,xt_state,iptable_nat,xt_TCPMSS,xt_tcpmss,xt_tcpudp,iptable_mangle,iptable_filter,ip_tables
ppp_generic            24243  6 pppoe,pppox
slhc                    5293  1 ppp_generic
cls_u32                 6468  6
sch_htb                14432  2
deflate                 1937  0
zlib_deflate           21228  1 deflate
des_generic            16135  0
cbc                     2721  0
ecb                     1975  0
crypto_blkcipher       13645  2 cbc,ecb
sha1_generic            2095  0
md5                     4001  0
hmac                    2977  0
crypto_hash            14519  3 sha1_generic,md5,hmac
cryptomgr               2636  0
aead                    6137  1 cryptomgr
crypto_algapi          15289  9 
deflate,des_generic,cbc,ecb,crypto_blkcipher,hmac,crypto_hash,cryptomgr,aead
af_key                 27372  0
fuse                   66747  1
w83627ehf              32052  0
hwmon_vid               2867  1 w83627ehf
vhost_net              16802  6
powernow_k8            12932  0
mperf                   1263  1 powernow_k8
kvm_amd                53431  24
kvm                   235155  1 kvm_amd
pl2303                 12732  1
xhci_hcd               62865  0
i2c_piix4               8391  0
k10temp                 3183  0
usbserial              34452  3 pl2303
usb_storage            37887  1
usb_libusual           10999  1 usb_storage
ohci_hcd               18105  0
ehci_hcd               33641  0
ahci                   20748  4
usbcore               130936  7 
pl2303,xhci_hcd,usbserial,usb_storage,usb_libusual,ohci_hcd,ehci_hcd
libahci                21202  1 ahci
sata_mv                26939  0
megaraid_sas           71659  14

^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
@ 2011-06-07  3:33                                             ` Brad Campbell
  0 siblings, 0 replies; 111+ messages in thread
From: Brad Campbell @ 2011-06-07  3:33 UTC (permalink / raw)
  To: Bart De Schuymer; +Cc: kvm, linux-mm, linux-kernel, netdev, netfilter-devel

On 07/06/11 04:10, Bart De Schuymer wrote:
> Hi Brad,
>
> This has probably nothing to do with ebtables, so please rmmod in case
> it's loaded.
> A few questions I didn't directly see an answer to in the threads I
> scanned...
> I'm assuming you actually use the bridging firewall functionality. So,
> what iptables modules do you use? Can you reduce your iptables rules to
> a core that triggers the bug?
> Or does it get triggered even with an empty set of firewall rules?
> Are you using a stock .35 kernel or is it patched?
> Is this something I can trigger on a poor guy's laptop or does it
> require specialized hardware (I'm catching up on qemu/kvm...)?

Not specialised hardware as such, I've just not been able to reproduce 
it outside of this specific operating scenario.

I can't trigger it with empty firewall rules as it relies on a DNAT to 
occur. If I try it directly to the internal IP address (as I have to 
without netfilter loaded) then of course nothing fails.

It's a pain in the bum as a fault, but it's one I can easily reproduce 
as long as I use the same set of circumstances.

I'll try using 3.0-rc2 (current git) tonight, and if I can reproduce it 
on that then I'll attempt to pare down the IPTABLES rules to a bare minimum.

It is nothing to do with ebtables as I don't compile it. I'm not really 
sure about "bridging firewall" functionality. I just use a couple of 
hand coded bash scripts to set the tables up.

brad@srv:~$ lsmod
Module                  Size  Used by
xt_iprange              1637  1
xt_DSCP                 2077  2
xt_length               1216  1
xt_CLASSIFY             1091  26
sch_sfq                 6681  4
xt_CHECKSUM             1229  2 brad@srv:~$ lsmod
Module                  Size  Used by
xt_iprange              1637  1
xt_DSCP                 2077  2
xt_length               1216  1
xt_CLASSIFY             1091  26
sch_sfq                 6681  4
xt_CHECKSUM             1229  2
ipt_REJECT              2277  1
ipt_MASQUERADE          1759  7
ipt_REDIRECT            1133  1
xt_recent               8223  2
xt_state                1226  5
iptable_nat             3993  1
nf_nat                 16773  3 ipt_MASQUERADE,ipt_REDIRECT,iptable_nat
nf_conntrack_ipv4      11868  8 iptable_nat,nf_nat
nf_conntrack           60962  5 
ipt_MASQUERADE,xt_state,iptable_nat,nf_nat,nf_conntrack_ipv4
nf_defrag_ipv4          1417  1 nf_conntrack_ipv4
xt_TCPMSS               2567  2
xt_tcpmss               1469  0
xt_tcpudp               2467  56
iptable_mangle          1487  1
pppoe                   9574  2
pppox                   2188  1 pppoe
iptable_filter          1442  1
ip_tables              16762  3 iptable_nat,iptable_mangle,iptable_filter
x_tables               20462  17 
xt_iprange,xt_DSCP,xt_length,xt_CLASSIFY,xt_CHECKSUM,ipt_REJECT,ipt_MASQUERADE,ipt_REDIRECT,xt_recent,xt_state,iptable_nat,xt_TCPMSS,xt_tcpmss,xt_tcpudp,iptable_mangle,iptable_filter,ip_tables
ppp_generic            24243  6 pppoe,pppox
slhc                    5293  1 ppp_generic
cls_u32                 6468  6
sch_htb                14432  2
deflate                 1937  0
zlib_deflate           21228  1 deflate
des_generic            16135  0
cbc                     2721  0
ecb                     1975  0
crypto_blkcipher       13645  2 cbc,ecb
sha1_generic            2095  0
md5                     4001  0
hmac                    2977  0
crypto_hash            14519  3 sha1_generic,md5,hmac
cryptomgr               2636  0
aead                    6137  1 cryptomgr
crypto_algapi          15289  9 
deflate,des_generic,cbc,ecb,crypto_blkcipher,hmac,crypto_hash,cryptomgr,aead
af_key                 27372  0
fuse                   66747  1
w83627ehf              32052  0
hwmon_vid               2867  1 w83627ehf
vhost_net              16802  6
powernow_k8            12932  0
mperf                   1263  1 powernow_k8
kvm_amd                53431  24
kvm                   235155  1 kvm_amd
pl2303                 12732  1
xhci_hcd               62865  0
i2c_piix4               8391  0
k10temp                 3183  0
usbserial              34452  3 pl2303
usb_storage            37887  1
usb_libusual           10999  1 usb_storage
ohci_hcd               18105  0
ehci_hcd               33641  0
ahci                   20748  4
usbcore               130936  7 
pl2303,xhci_hcd,usbserial,usb_storage,usb_libusual,ohci_hcd,ehci_hcd
libahci                21202  1 ahci
sata_mv                26939  0
megaraid_sas           71659  14

Nat Table (external ip substituted for xxx.xxx.xxx.xxx)

Chain PREROUTING (policy ACCEPT 1761K packets, 152M bytes)
  pkts bytes target     prot opt in     out     source 
destination
     5   210 DNAT       udp  --  ppp0   *       0.0.0.0/0 
0.0.0.0/0           udp dpt:1195 to:192.168.253.199
     6   252 DNAT       udp  --  !ppp0  *       0.0.0.0/0 
xxx.xxx.xxx.xxx       udp dpt:1195 to:192.168.253.199
     0     0 DNAT       tcp  --  ppp0   *       0.0.0.0/0 
0.0.0.0/0           tcp dpt:25001 to:192.168.253.199:465
     0     0 DNAT       tcp  --  ppp0   *       0.0.0.0/0 
0.0.0.0/0           tcp dpt:25000 to:192.168.253.199:993
     0     0 DNAT       tcp  --  !ppp0  *       0.0.0.0/0 
xxx.xxx.xxx.xxx       tcp dpt:25001 to:192.168.253.199:465
     0     0 DNAT       tcp  --  !ppp0  *       0.0.0.0/0 
xxx.xxx.xxx.xxx       tcp dpt:25000 to:192.168.253.199:993
     2   142 DNAT       47   --  ppp0   *       0.0.0.0/0 
0.0.0.0/0           to:192.168.253.199
    18   880 DNAT       tcp  --  ppp0   *       0.0.0.0/0 
0.0.0.0/0           tcp dpt:1723 to:192.168.253.199
     0     0 DNAT       47   --  !ppp0  *       0.0.0.0/0 
xxx.xxx.xxx.xxx       to:192.168.253.199
     0     0 DNAT       tcp  --  !ppp0  *       0.0.0.0/0 
xxx.xxx.xxx.xxx       tcp dpt:1723 to:192.168.253.199
  2969  149K DNAT       tcp  --  ppp0   *       0.0.0.0/0 
0.0.0.0/0           tcp dpt:443 to:192.168.253.198
    20  1280 DNAT       tcp  --  !ppp0  *       0.0.0.0/0 
xxx.xxx.xxx.xxx       tcp dpt:443 to:192.168.253.198
     0     0 DNAT       tcp  --  ppp0   *       0.0.0.0/0 
0.0.0.0/0           tcp dpt:3101 to:192.168.253.197
     0     0 DNAT       tcp  --  !ppp0  *       0.0.0.0/0 
xxx.xxx.xxx.xxx       tcp dpt:3101 to:192.168.253.197
     0     0 DNAT       tcp  --  !ppp0  *       0.0.0.0/0 
xxx.xxx.xxx.xxx       tcp dpt:4101 to:192.168.253.197
44528 2718K REDIRECT   tcp  --  !ppp0  *       0.0.0.0/0 
!192.168.0.0/16      tcp dpt:80 redir ports 8080
     0     0 DNAT       tcp  --  *      *       0.0.0.0/0 
0.0.0.0/0           tcp dpt:3724 to:192.168.2.107
  596K   33M DNAT       tcp  --  ppp0   *       0.0.0.0/0 
0.0.0.0/0           tcp dpts:2001:2030 to:10.99.99.2
1420K  119M DNAT       udp  --  ppp0   *       0.0.0.0/0 
0.0.0.0/0           udp dpts:2001:2030 to:10.99.99.2
  7483  449K DNAT       all  --  !ppp0  *       0.0.0.0/0 
xxx.xxx.xxx.xxx       to:192.168.2.1


Mangle Table

Chain INPUT (policy ACCEPT 270K packets, 17M bytes)
  pkts bytes target     prot opt in     out     source 
destination

Chain OUTPUT (policy ACCEPT 170K packets, 12M bytes)
  pkts bytes target     prot opt in     out     source 
destination

Chain POSTROUTING (policy ACCEPT 2205K packets, 166M bytes)
  pkts bytes target     prot opt in     out     source 
destination
     0     0 MASQUERADE  all  --  *      *       0.0.0.0/0 
192.168.254.3
     6   360 ACCEPT     all  --  *      *       0.0.0.0/0 
xxx.xxx.xxx.xxx
20424 2120K MASQUERADE  all  --  *      ppp0    192.168.0.0/16 
!192.168.0.0/16
     0     0 MASQUERADE  all  --  *      ppp0    10.0.0.0/24 
0.0.0.0/0
     3   204 MASQUERADE  all  --  *      *       192.168.2.0/24 
10.8.0.0/24
1418K  128M MASQUERADE  all  --  *      *       10.99.99.0/24 
0.0.0.0/0
68248 4095K MASQUERADE  all  --  *      *       192.168.253.0/24 
10.8.0.0/16
13305 2405K MASQUERADE  all  --  *      *       192.168.253.0/24 
!192.168.0.0/16

Chain PREROUTING (policy ACCEPT 278M packets, 293G bytes)
  pkts bytes target     prot opt in     out     source 
destination
   169 55528 CHECKSUM   udp  --  br1    *       0.0.0.0/0 
0.0.0.0/0           udp dpt:67 CHECKSUM fill

Chain INPUT (policy ACCEPT 180M packets, 250G bytes)
  pkts bytes target     prot opt in     out     source 
destination

Chain FORWARD (policy ACCEPT 98M packets, 44G bytes)
  pkts bytes target     prot opt in     out     source 
destination

Chain OUTPUT (policy ACCEPT 155M packets, 180G bytes)
  pkts bytes target     prot opt in     out     source 
destination

Chain POSTROUTING (policy ACCEPT 253M packets, 223G bytes)
  pkts bytes target     prot opt in     out     source 
destination
   165 54182 CHECKSUM   udp  --  *      br1     0.0.0.0/0 
0.0.0.0/0           udp spt:67 CHECKSUM fill
    51  3712 CLASSIFY   tcp  --  *      ppp0    0.0.0.0/0 
0.0.0.0/0           tcp dpt:53 CLASSIFY set 1:20
85274 6454K CLASSIFY   udp  --  *      ppp0    0.0.0.0/0 
0.0.0.0/0           udp dpt:53 CLASSIFY set 1:20
   187  257K CLASSIFY   tcp  --  *      ppp0    0.0.0.0/0 
0.0.0.0/0           tcp spt:81 CLASSIFY set 1:20
   25M 1180M CLASSIFY   tcp  --  *      ppp0    0.0.0.0/0 
0.0.0.0/0           tcp flags:0x3F/0x10 state ESTABLISHED length 40:100 
CLASSIFY set 1:15
  728K   67M CLASSIFY   icmp --  *      ppp0    0.0.0.0/0 
0.0.0.0/0           CLASSIFY set 1:15
   231 23484 CLASSIFY   tcp  --  *      ppp0    0.0.0.0/0 
0.0.0.0/0           tcp dpt:2401 CLASSIFY set 1:15
65636 5610K CLASSIFY   tcp  --  *      ppp0    0.0.0.0/0 
0.0.0.0/0           tcp dpt:22 CLASSIFY set 1:10
  2018  315K CLASSIFY   tcp  --  *      ppp0    0.0.0.0/0 
0.0.0.0/0           tcp spt:22 CLASSIFY set 1:10
    80 10092 CLASSIFY   tcp  --  *      ppp0    0.0.0.0/0 
0.0.0.0/0           tcp dpt:3389 CLASSIFY set 1:10
26063 8910K CLASSIFY   tcp  --  *      ppp0    0.0.0.0/0 
0.0.0.0/0           tcp dpt:8080 CLASSIFY set 1:15
  932K  131M CLASSIFY   tcp  --  *      ppp0    0.0.0.0/0 
0.0.0.0/0           tcp dpt:80 CLASSIFY set 1:15
  3511  267K CLASSIFY   udp  --  *      ppp0    0.0.0.0/0 
0.0.0.0/0           udp dpt:123 CLASSIFY set 1:10
     0     0 CLASSIFY   tcp  --  *      ppp0    0.0.0.0/0 
0.0.0.0/0           tcp spt:20 CLASSIFY set 1:15
     3   180 CLASSIFY   tcp  --  *      ppp0    0.0.0.0/0 
0.0.0.0/0           tcp dpt:20 CLASSIFY set 1:15
94058   38M CLASSIFY   47   --  *      ppp0    0.0.0.0/0 
0.0.0.0/0           CLASSIFY set 1:10
1086K  183M CLASSIFY   udp  --  *      ppp0    0.0.0.0/0 
0.0.0.0/0           udp spt:1194 CLASSIFY set 1:10
1086K  183M TOS        udp  --  *      ppp0    0.0.0.0/0 
0.0.0.0/0           udp spt:1194 TOS set 0x10/0x3f
48817   10M CLASSIFY   udp  --  *      ppp0    0.0.0.0/0 
0.0.0.0/0           udp spt:1195 CLASSIFY set 1:10
48817   10M TOS        udp  --  *      ppp0    0.0.0.0/0 
0.0.0.0/0           udp spt:1195 TOS set 0x10/0x3f
94058   38M CLASSIFY   47   --  *      ppp0    0.0.0.0/0 
0.0.0.0/0           CLASSIFY set 1:15
   106  7207 CLASSIFY   tcp  --  *      ppp0    0.0.0.0/0 
0.0.0.0/0           tcp dpt:1863 CLASSIFY set 1:15
  188K   34M CLASSIFY   tcp  --  *      ppp0    0.0.0.0/0 
0.0.0.0/0           tcp dpt:443 CLASSIFY set 1:15
51541 3327K CLASSIFY   tcp  --  *      ppp0    0.0.0.0/0 
0.0.0.0/0           tcp dpts:6660:6669 CLASSIFY set 1:15
     0     0 CLASSIFY   tcp  --  *      ppp0    0.0.0.0/0 
0.0.0.0/0           tcp spts:2021:2030 CLASSIFY set 1:15
    85  4944 CLASSIFY   tcp  --  *      ppp0    0.0.0.0/0 
0.0.0.0/0           tcp dpt:19999 CLASSIFY set 1:15
  208K   86M CLASSIFY   udp  --  *      *       0.0.0.0/0 
0.0.0.0/0           source IP range 192.168.2.80-192.168.2.120 CLASSIFY 
set 1:10
     0     0 CLASSIFY   tcp  --  *      ppp0    0.0.0.0/0 
0.0.0.0/0           tcp spt:12345 CLASSIFY set 1:15
     1    80 CLASSIFY   udp  --  *      ppp0    0.0.0.0/0 
0.0.0.0/0           udp spt:12345 CLASSIFY set 1:15


Default table

Chain INPUT (policy ACCEPT 176M packets, 247G bytes)
  pkts bytes target     prot opt in     out     source 
destination
     0     0 ACCEPT     udp  --  ppp0   *       0.0.0.0/0 
0.0.0.0/0           udp dpt:4569
1187K  582M ACCEPT     udp  --  ppp0   *       0.0.0.0/0 
0.0.0.0/0           udp dpt:1194
     2   577 ACCEPT     udp  --  ppp0   *       0.0.0.0/0 
0.0.0.0/0           udp dpt:1195
    28  1224 ACCEPT     tcp  --  ppp0   *       0.0.0.0/0 
0.0.0.0/0           tcp dpt:3389
   230 12372            tcp  --  ppp0   *       0.0.0.0/0 
0.0.0.0/0           tcp dpt:22 state NEW recent: SET name: DEFAULT side: 
source
     3   180 DROP       tcp  --  ppp0   *       0.0.0.0/0 
0.0.0.0/0           tcp dpt:22 state NEW recent: UPDATE seconds: 300 
hit_count: 4 name: DEFAULT side: source
  1750  143K ACCEPT     tcp  --  ppp0   *       0.0.0.0/0 
0.0.0.0/0           tcp dpt:22
     3   144 ACCEPT     tcp  --  ppp0   *       0.0.0.0/0 
0.0.0.0/0           tcp dpt:113
   120  6090 ACCEPT     tcp  --  ppp0   *       0.0.0.0/0 
0.0.0.0/0           tcp dpt:81
36094   29M ACCEPT     tcp  --  ppp0   *       0.0.0.0/0 
0.0.0.0/0           tcp dpt:25
1456K 1706M ACCEPT     all  --  ppp0   *       0.0.0.0/0 
0.0.0.0/0           state RELATED,ESTABLISHED
31047 2334K REJECT     tcp  --  ppp0   *       0.0.0.0/0 
0.0.0.0/0           tcp option=!2 reject-with tcp-reset
  552K   60M ACCEPT     all  --  !ppp0  *       0.0.0.0/0 
0.0.0.0/0           state NEW
13552 1207K ACCEPT     icmp --  ppp0   *       0.0.0.0/0 
0.0.0.0/0
  5712  392K DROP       all  --  ppp0   *       0.0.0.0/0 
0.0.0.0/0

Chain FORWARD (policy ACCEPT 98M packets, 44G bytes)
  pkts bytes target     prot opt in     out     source 
destination
1207K   68M TCPMSS     tcp  --  *      ppp0    0.0.0.0/0 
0.0.0.0/0           tcp flags:0x06/0x02 TCPMSS clamp to PMTU

Chain OUTPUT (policy ACCEPT 155M packets, 180G bytes)
  pkts bytes target     prot opt in     out     source 
destination
31675 1895K TCPMSS     tcp  --  *      ppp0    0.0.0.0/0 
0.0.0.0/0           tcp flags:0x06/0x02 TCPMSS clamp to PMTU

lsmod

ipt_REJECT              2277  1
ipt_MASQUERADE          1759  7
ipt_REDIRECT            1133  1
xt_recent               8223  2
xt_state                1226  5
iptable_nat             3993  1
nf_nat                 16773  3 ipt_MASQUERADE,ipt_REDIRECT,iptable_nat
nf_conntrack_ipv4      11868  8 iptable_nat,nf_nat
nf_conntrack           60962  5 
ipt_MASQUERADE,xt_state,iptable_nat,nf_nat,nf_conntrack_ipv4
nf_defrag_ipv4          1417  1 nf_conntrack_ipv4
xt_TCPMSS               2567  2
xt_tcpmss               1469  0
xt_tcpudp               2467  56
iptable_mangle          1487  1
pppoe                   9574  2
pppox                   2188  1 pppoe
iptable_filter          1442  1
ip_tables              16762  3 iptable_nat,iptable_mangle,iptable_filter
x_tables               20462  17 
xt_iprange,xt_DSCP,xt_length,xt_CLASSIFY,xt_CHECKSUM,ipt_REJECT,ipt_MASQUERADE,ipt_REDIRECT,xt_recent,xt_state,iptable_nat,xt_TCPMSS,xt_tcpmss,xt_tcpudp,iptable_mangle,iptable_filter,ip_tables
ppp_generic            24243  6 pppoe,pppox
slhc                    5293  1 ppp_generic
cls_u32                 6468  6
sch_htb                14432  2
deflate                 1937  0
zlib_deflate           21228  1 deflate
des_generic            16135  0
cbc                     2721  0
ecb                     1975  0
crypto_blkcipher       13645  2 cbc,ecb
sha1_generic            2095  0
md5                     4001  0
hmac                    2977  0
crypto_hash            14519  3 sha1_generic,md5,hmac
cryptomgr               2636  0
aead                    6137  1 cryptomgr
crypto_algapi          15289  9 
deflate,des_generic,cbc,ecb,crypto_blkcipher,hmac,crypto_hash,cryptomgr,aead
af_key                 27372  0
fuse                   66747  1
w83627ehf              32052  0
hwmon_vid               2867  1 w83627ehf
vhost_net              16802  6
powernow_k8            12932  0
mperf                   1263  1 powernow_k8
kvm_amd                53431  24
kvm                   235155  1 kvm_amd
pl2303                 12732  1
xhci_hcd               62865  0
i2c_piix4               8391  0
k10temp                 3183  0
usbserial              34452  3 pl2303
usb_storage            37887  1
usb_libusual           10999  1 usb_storage
ohci_hcd               18105  0
ehci_hcd               33641  0
ahci                   20748  4
usbcore               130936  7 
pl2303,xhci_hcd,usbserial,usb_storage,usb_libusual,ohci_hcd,ehci_hcd
libahci                21202  1 ahci
sata_mv                26939  0
megaraid_sas           71659  14

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
  2011-06-06 20:22                                           ` Eric Dumazet
  (?)
  (?)
@ 2011-06-07 13:27                                           ` Brad Campbell
  2011-06-07 13:37                                             ` Eric Dumazet
  -1 siblings, 1 reply; 111+ messages in thread
From: Brad Campbell @ 2011-06-07 13:27 UTC (permalink / raw)
  To: Eric Dumazet; +Cc: Avi Kivity, CaT, Borislav Petkov, linux-kernel, kvm, netdev

On 07/06/11 04:22, Eric Dumazet wrote:

> Could you please try latest linux-2.6 tree ?
>
> We fixed many networking bugs that could explain your crash.
>
>
>
>

No good I'm afraid.

[  543.040056] 
=============================================================================
[  543.040136] BUG ip_dst_cache: Padding overwritten. 
0xffff8803e4217ffe-0xffff8803e4217fff
[  543.040194] 
-----------------------------------------------------------------------------
[  543.040198]
[  543.040298] INFO: Slab 0xffffea000d9e74d0 objects=25 used=25 fp=0x 
        (null) flags=0x8000000000004081
[  543.040364] Pid: 4576, comm: kworker/1:2 Not tainted 3.0.0-rc2 #1
[  543.040415] Call Trace:
[  543.040472]  [<ffffffff810b9c1d>] ? slab_err+0xad/0xd0
[  543.040528]  [<ffffffff8102e034>] ? check_preempt_wakeup+0xa4/0x160
[  543.040595]  [<ffffffff810ba206>] ? slab_pad_check+0x126/0x170
[  543.040650]  [<ffffffff8133045b>] ? dst_destroy+0x8b/0x110
[  543.040701]  [<ffffffff810ba29a>] ? check_slab+0x4a/0xc0
[  543.040753]  [<ffffffff810baf2d>] ? free_debug_processing+0x2d/0x250
[  543.040808]  [<ffffffff810bb27b>] ? __slab_free+0x12b/0x140
[  543.040862]  [<ffffffff810bbe99>] ? kmem_cache_free+0x99/0xa0
[  543.040915]  [<ffffffff8133045b>] ? dst_destroy+0x8b/0x110
[  543.040967]  [<ffffffff813307f6>] ? dst_gc_task+0x196/0x1f0
[  543.041021]  [<ffffffff8104e954>] ? queue_delayed_work_on+0x154/0x160
[  543.041081]  [<ffffffff813066fe>] ? do_dbs_timer+0x20e/0x3d0
[  543.041133]  [<ffffffff81330660>] ? dst_alloc+0x180/0x180
[  543.041187]  [<ffffffff8104f28b>] ? process_one_work+0xfb/0x3b0
[  543.041242]  [<ffffffff8104f964>] ? worker_thread+0x144/0x3d0
[  543.041296]  [<ffffffff8102cc10>] ? __wake_up_common+0x50/0x80
[  543.041678]  [<ffffffff8104f820>] ? rescuer_thread+0x2e0/0x2e0
[  543.041729]  [<ffffffff8104f820>] ? rescuer_thread+0x2e0/0x2e0
[  543.041782]  [<ffffffff81053436>] ? kthread+0x96/0xa0
[  543.041835]  [<ffffffff813e1d14>] ? kernel_thread_helper+0x4/0x10
[  543.041890]  [<ffffffff810533a0>] ? kthread_worker_fn+0x120/0x120
[  543.041944]  [<ffffffff813e1d10>] ? gs_change+0xb/0xb
[  543.041993]  Padding 0xffff8803e4217f40:  5a 5a 5a 5a 5a 5a 5a 5a 5a 
5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
[  543.042718]  Padding 0xffff8803e4217f50:  5a 5a 5a 5a 5a 5a 5a 5a 5a 
5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
[  543.043433]  Padding 0xffff8803e4217f60:  5a 5a 5a 5a 5a 5a 5a 5a 5a 
5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
[  543.044155]  Padding 0xffff8803e4217f70:  5a 5a 5a 5a 5a 5a 5a 5a 5a 
5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
[  543.044866]  Padding 0xffff8803e4217f80:  5a 5a 5a 5a 5a 5a 5a 5a 5a 
5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
[  543.045590]  Padding 0xffff8803e4217f90:  5a 5a 5a 5a 5a 5a 5a 5a 5a 
5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
[  543.046311]  Padding 0xffff8803e4217fa0:  5a 5a 5a 5a 5a 5a 5a 5a 5a 
5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
[  543.047034]  Padding 0xffff8803e4217fb0:  5a 5a 5a 5a 5a 5a 5a 5a 5a 
5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
[  543.047755]  Padding 0xffff8803e4217fc0:  5a 5a 5a 5a 5a 5a 5a 5a 5a 
5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
[  543.048474]  Padding 0xffff8803e4217fd0:  5a 5a 5a 5a 5a 5a 5a 5a 5a 
5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
[  543.049203]  Padding 0xffff8803e4217fe0:  5a 5a 5a 5a 5a 5a 5a 5a 5a 
5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
[  543.049909]  Padding 0xffff8803e4217ff0:  5a 5a 5a 5a 5a 5a 5a 5a 5a 
5a 5a 5a 5a 5a 00 00 ZZZZZZZZZZZZZZ..
[  543.050021] FIX ip_dst_cache: Restoring 
0xffff8803e4217f40-0xffff8803e4217fff=0x5a
[  543.050021]

Dropped -mm, Hugh and Andrea from CC as this does not appear to be mm or 
ksm related.

I'll pare down the firewall and see if I can make it break easier with a 
smaller test set.

^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
  2011-06-07  3:33                                             ` Brad Campbell
@ 2011-06-07 13:30                                               ` Patrick McHardy
  -1 siblings, 0 replies; 111+ messages in thread
From: Patrick McHardy @ 2011-06-07 13:30 UTC (permalink / raw)
  To: Brad Campbell
  Cc: Bart De Schuymer, kvm, linux-mm, linux-kernel, netdev, netfilter-devel

On 07.06.2011 05:33, Brad Campbell wrote:
> On 07/06/11 04:10, Bart De Schuymer wrote:
>> Hi Brad,
>>
>> This has probably nothing to do with ebtables, so please rmmod in case
>> it's loaded.
>> A few questions I didn't directly see an answer to in the threads I
>> scanned...
>> I'm assuming you actually use the bridging firewall functionality. So,
>> what iptables modules do you use? Can you reduce your iptables rules to
>> a core that triggers the bug?
>> Or does it get triggered even with an empty set of firewall rules?
>> Are you using a stock .35 kernel or is it patched?
>> Is this something I can trigger on a poor guy's laptop or does it
>> require specialized hardware (I'm catching up on qemu/kvm...)?
> 
> Not specialised hardware as such, I've just not been able to reproduce
> it outside of this specific operating scenario.

The last similar problem we've had was related to the 32/64 bit compat
code. Are you running 32 bit userspace on a 64 bit kernel?

> I can't trigger it with empty firewall rules as it relies on a DNAT to
> occur. If I try it directly to the internal IP address (as I have to
> without netfilter loaded) then of course nothing fails.
> 
> It's a pain in the bum as a fault, but it's one I can easily reproduce
> as long as I use the same set of circumstances.
> 
> I'll try using 3.0-rc2 (current git) tonight, and if I can reproduce it
> on that then I'll attempt to pare down the IPTABLES rules to a bare
> minimum.
> 
> It is nothing to do with ebtables as I don't compile it. I'm not really
> sure about "bridging firewall" functionality. I just use a couple of
> hand coded bash scripts to set the tables up.

>From one of your previous mails:

> # CONFIG_BRIDGE_NF_EBTABLES is not set

How about CONFIG_BRIDGE_NETFILTER?

^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
@ 2011-06-07 13:30                                               ` Patrick McHardy
  0 siblings, 0 replies; 111+ messages in thread
From: Patrick McHardy @ 2011-06-07 13:30 UTC (permalink / raw)
  To: Brad Campbell
  Cc: Bart De Schuymer, kvm, linux-mm, linux-kernel, netdev, netfilter-devel

On 07.06.2011 05:33, Brad Campbell wrote:
> On 07/06/11 04:10, Bart De Schuymer wrote:
>> Hi Brad,
>>
>> This has probably nothing to do with ebtables, so please rmmod in case
>> it's loaded.
>> A few questions I didn't directly see an answer to in the threads I
>> scanned...
>> I'm assuming you actually use the bridging firewall functionality. So,
>> what iptables modules do you use? Can you reduce your iptables rules to
>> a core that triggers the bug?
>> Or does it get triggered even with an empty set of firewall rules?
>> Are you using a stock .35 kernel or is it patched?
>> Is this something I can trigger on a poor guy's laptop or does it
>> require specialized hardware (I'm catching up on qemu/kvm...)?
> 
> Not specialised hardware as such, I've just not been able to reproduce
> it outside of this specific operating scenario.

The last similar problem we've had was related to the 32/64 bit compat
code. Are you running 32 bit userspace on a 64 bit kernel?

> I can't trigger it with empty firewall rules as it relies on a DNAT to
> occur. If I try it directly to the internal IP address (as I have to
> without netfilter loaded) then of course nothing fails.
> 
> It's a pain in the bum as a fault, but it's one I can easily reproduce
> as long as I use the same set of circumstances.
> 
> I'll try using 3.0-rc2 (current git) tonight, and if I can reproduce it
> on that then I'll attempt to pare down the IPTABLES rules to a bare
> minimum.
> 
> It is nothing to do with ebtables as I don't compile it. I'm not really
> sure about "bridging firewall" functionality. I just use a couple of
> hand coded bash scripts to set the tables up.

>From one of your previous mails:

> # CONFIG_BRIDGE_NF_EBTABLES is not set

How about CONFIG_BRIDGE_NETFILTER?

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
  2011-06-07 13:27                                           ` Brad Campbell
@ 2011-06-07 13:37                                             ` Eric Dumazet
  2011-06-07 15:15                                               ` Brad Campbell
  2011-08-20 13:16                                               ` Brad Campbell
  0 siblings, 2 replies; 111+ messages in thread
From: Eric Dumazet @ 2011-06-07 13:37 UTC (permalink / raw)
  To: Brad Campbell; +Cc: Avi Kivity, CaT, Borislav Petkov, linux-kernel, kvm, netdev

Le mardi 07 juin 2011 à 21:27 +0800, Brad Campbell a écrit :
> On 07/06/11 04:22, Eric Dumazet wrote:
> 
> > Could you please try latest linux-2.6 tree ?
> >
> > We fixed many networking bugs that could explain your crash.
> >
> >
> >
> >
> 
> No good I'm afraid.
> 
> [  543.040056] 
> =============================================================================
> [  543.040136] BUG ip_dst_cache: Padding overwritten. 
> 0xffff8803e4217ffe-0xffff8803e4217fff
> [  543.040194] 

Thats pretty strange : These are the last two bytes of a page, set to
0x0000 (a 16 bit value)

There is no way a dst field could actually sit on this location (its a
padding), since a dst is a bit less than 256 bytes (0xe8), and each
entry is aligned on a 64byte address.

grep dst /proc/slabinfo 

ip_dst_cache       32823  62944    256   32    2 : tunables    0    0
0 : slabdata   1967   1967      0

sizeof(struct rtable)=0xe8


> -----------------------------------------------------------------------------
> [  543.040198]
> [  543.040298] INFO: Slab 0xffffea000d9e74d0 objects=25 used=25 fp=0x 
>         (null) flags=0x8000000000004081
> [  543.040364] Pid: 4576, comm: kworker/1:2 Not tainted 3.0.0-rc2 #1
> [  543.040415] Call Trace:
> [  543.040472]  [<ffffffff810b9c1d>] ? slab_err+0xad/0xd0
> [  543.040528]  [<ffffffff8102e034>] ? check_preempt_wakeup+0xa4/0x160
> [  543.040595]  [<ffffffff810ba206>] ? slab_pad_check+0x126/0x170
> [  543.040650]  [<ffffffff8133045b>] ? dst_destroy+0x8b/0x110
> [  543.040701]  [<ffffffff810ba29a>] ? check_slab+0x4a/0xc0
> [  543.040753]  [<ffffffff810baf2d>] ? free_debug_processing+0x2d/0x250
> [  543.040808]  [<ffffffff810bb27b>] ? __slab_free+0x12b/0x140
> [  543.040862]  [<ffffffff810bbe99>] ? kmem_cache_free+0x99/0xa0
> [  543.040915]  [<ffffffff8133045b>] ? dst_destroy+0x8b/0x110
> [  543.040967]  [<ffffffff813307f6>] ? dst_gc_task+0x196/0x1f0
> [  543.041021]  [<ffffffff8104e954>] ? queue_delayed_work_on+0x154/0x160
> [  543.041081]  [<ffffffff813066fe>] ? do_dbs_timer+0x20e/0x3d0
> [  543.041133]  [<ffffffff81330660>] ? dst_alloc+0x180/0x180
> [  543.041187]  [<ffffffff8104f28b>] ? process_one_work+0xfb/0x3b0
> [  543.041242]  [<ffffffff8104f964>] ? worker_thread+0x144/0x3d0
> [  543.041296]  [<ffffffff8102cc10>] ? __wake_up_common+0x50/0x80
> [  543.041678]  [<ffffffff8104f820>] ? rescuer_thread+0x2e0/0x2e0
> [  543.041729]  [<ffffffff8104f820>] ? rescuer_thread+0x2e0/0x2e0
> [  543.041782]  [<ffffffff81053436>] ? kthread+0x96/0xa0
> [  543.041835]  [<ffffffff813e1d14>] ? kernel_thread_helper+0x4/0x10
> [  543.041890]  [<ffffffff810533a0>] ? kthread_worker_fn+0x120/0x120
> [  543.041944]  [<ffffffff813e1d10>] ? gs_change+0xb/0xb
> [  543.041993]  Padding 0xffff8803e4217f40:  5a 5a 5a 5a 5a 5a 5a 5a 5a 
> 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
> [  543.042718]  Padding 0xffff8803e4217f50:  5a 5a 5a 5a 5a 5a 5a 5a 5a 
> 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
> [  543.043433]  Padding 0xffff8803e4217f60:  5a 5a 5a 5a 5a 5a 5a 5a 5a 
> 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
> [  543.044155]  Padding 0xffff8803e4217f70:  5a 5a 5a 5a 5a 5a 5a 5a 5a 
> 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
> [  543.044866]  Padding 0xffff8803e4217f80:  5a 5a 5a 5a 5a 5a 5a 5a 5a 
> 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
> [  543.045590]  Padding 0xffff8803e4217f90:  5a 5a 5a 5a 5a 5a 5a 5a 5a 
> 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
> [  543.046311]  Padding 0xffff8803e4217fa0:  5a 5a 5a 5a 5a 5a 5a 5a 5a 
> 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
> [  543.047034]  Padding 0xffff8803e4217fb0:  5a 5a 5a 5a 5a 5a 5a 5a 5a 
> 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
> [  543.047755]  Padding 0xffff8803e4217fc0:  5a 5a 5a 5a 5a 5a 5a 5a 5a 
> 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
> [  543.048474]  Padding 0xffff8803e4217fd0:  5a 5a 5a 5a 5a 5a 5a 5a 5a 
> 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
> [  543.049203]  Padding 0xffff8803e4217fe0:  5a 5a 5a 5a 5a 5a 5a 5a 5a 
> 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
> [  543.049909]  Padding 0xffff8803e4217ff0:  5a 5a 5a 5a 5a 5a 5a 5a 5a 
> 5a 5a 5a 5a 5a 00 00 ZZZZZZZZZZZZZZ..
> [  543.050021] FIX ip_dst_cache: Restoring 
> 0xffff8803e4217f40-0xffff8803e4217fff=0x5a
> [  543.050021]
> 
> Dropped -mm, Hugh and Andrea from CC as this does not appear to be mm or 
> ksm related.
> 
> I'll pare down the firewall and see if I can make it break easier with a 
> smaller test set.

Hmm, not sure now :(

Could you reproduce another bug please ?




^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
  2011-06-07 13:30                                               ` Patrick McHardy
@ 2011-06-07 14:40                                                 ` Brad Campbell
  -1 siblings, 0 replies; 111+ messages in thread
From: Brad Campbell @ 2011-06-07 14:40 UTC (permalink / raw)
  To: Patrick McHardy
  Cc: Bart De Schuymer, kvm, linux-mm, linux-kernel, netdev, netfilter-devel

On 07/06/11 21:30, Patrick McHardy wrote:
> On 07.06.2011 05:33, Brad Campbell wrote:
>> On 07/06/11 04:10, Bart De Schuymer wrote:
>>> Hi Brad,
>>>
>>> This has probably nothing to do with ebtables, so please rmmod in case
>>> it's loaded.
>>> A few questions I didn't directly see an answer to in the threads I
>>> scanned...
>>> I'm assuming you actually use the bridging firewall functionality. So,
>>> what iptables modules do you use? Can you reduce your iptables rules to
>>> a core that triggers the bug?
>>> Or does it get triggered even with an empty set of firewall rules?
>>> Are you using a stock .35 kernel or is it patched?
>>> Is this something I can trigger on a poor guy's laptop or does it
>>> require specialized hardware (I'm catching up on qemu/kvm...)?
>>
>> Not specialised hardware as such, I've just not been able to reproduce
>> it outside of this specific operating scenario.
>
> The last similar problem we've had was related to the 32/64 bit compat
> code. Are you running 32 bit userspace on a 64 bit kernel?

No, 32 bit Guest OS, but a completely 64 bit userspace on a 64 bit kernel.

Userspace is current Debian Stable. Kernel is Vanilla and qemu-kvm is 
current git


>> I can't trigger it with empty firewall rules as it relies on a DNAT to
>> occur. If I try it directly to the internal IP address (as I have to
>> without netfilter loaded) then of course nothing fails.
>>
>> It's a pain in the bum as a fault, but it's one I can easily reproduce
>> as long as I use the same set of circumstances.
>>
>> I'll try using 3.0-rc2 (current git) tonight, and if I can reproduce it
>> on that then I'll attempt to pare down the IPTABLES rules to a bare
>> minimum.
>>
>> It is nothing to do with ebtables as I don't compile it. I'm not really
>> sure about "bridging firewall" functionality. I just use a couple of
>> hand coded bash scripts to set the tables up.
>
>  From one of your previous mails:
>
>> # CONFIG_BRIDGE_NF_EBTABLES is not set
>
> How about CONFIG_BRIDGE_NETFILTER?
>

It was compiled in.

With the following table set I was able to reproduce the problem on 
3.0-rc2. Replaced my IP with xxx.xxx.xxx.xxx, but otherwise unmodified

root@srv:~# iptables-save
# Generated by iptables-save v1.4.10 on Tue Jun  7 22:11:30 2011
*filter
:INPUT ACCEPT [978:107619]
:FORWARD ACCEPT [142:7068]
:OUTPUT ACCEPT [1659:291870]
-A INPUT -i ppp0 -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT ! -i ppp0 -m state --state NEW -j ACCEPT
-A INPUT -i ppp0 -j DROP
COMMIT
# Completed on Tue Jun  7 22:11:30 2011
# Generated by iptables-save v1.4.10 on Tue Jun  7 22:11:30 2011
*nat
:PREROUTING ACCEPT [813:49170]
:INPUT ACCEPT [91:7090]
:OUTPUT ACCEPT [267:20731]
:POSTROUTING ACCEPT [296:22281]
-A PREROUTING -d xxx.xxx.xxx.xxx/32 ! -i ppp0 -p tcp -m tcp --dport 443 
-j DNAT --to-destination 192.168.253.198
COMMIT
# Completed on Tue Jun  7 22:11:30 2011
# Generated by iptables-save v1.4.10 on Tue Jun  7 22:11:30 2011
*mangle
:PREROUTING ACCEPT [2729:274392]
:INPUT ACCEPT [2508:262976]
:FORWARD ACCEPT [142:7068]
:OUTPUT ACCEPT [1674:293701]
:POSTROUTING ACCEPT [2131:346411]
-A FORWARD -o ppp0 -p tcp -m tcp --tcp-flags SYN,RST SYN -m tcpmss --mss 
1400:1536 -j TCPMSS --clamp-mss-to-pmtu
COMMIT
# Completed on Tue Jun  7 22:11:30 2011

I've just compiled out CONFIG_BRIDGE_NETFILTER and can no longer access 
the address the way I was doing it, so that's a no-go for me.



^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
@ 2011-06-07 14:40                                                 ` Brad Campbell
  0 siblings, 0 replies; 111+ messages in thread
From: Brad Campbell @ 2011-06-07 14:40 UTC (permalink / raw)
  To: Patrick McHardy
  Cc: Bart De Schuymer, kvm, linux-mm, linux-kernel, netdev, netfilter-devel

On 07/06/11 21:30, Patrick McHardy wrote:
> On 07.06.2011 05:33, Brad Campbell wrote:
>> On 07/06/11 04:10, Bart De Schuymer wrote:
>>> Hi Brad,
>>>
>>> This has probably nothing to do with ebtables, so please rmmod in case
>>> it's loaded.
>>> A few questions I didn't directly see an answer to in the threads I
>>> scanned...
>>> I'm assuming you actually use the bridging firewall functionality. So,
>>> what iptables modules do you use? Can you reduce your iptables rules to
>>> a core that triggers the bug?
>>> Or does it get triggered even with an empty set of firewall rules?
>>> Are you using a stock .35 kernel or is it patched?
>>> Is this something I can trigger on a poor guy's laptop or does it
>>> require specialized hardware (I'm catching up on qemu/kvm...)?
>>
>> Not specialised hardware as such, I've just not been able to reproduce
>> it outside of this specific operating scenario.
>
> The last similar problem we've had was related to the 32/64 bit compat
> code. Are you running 32 bit userspace on a 64 bit kernel?

No, 32 bit Guest OS, but a completely 64 bit userspace on a 64 bit kernel.

Userspace is current Debian Stable. Kernel is Vanilla and qemu-kvm is 
current git


>> I can't trigger it with empty firewall rules as it relies on a DNAT to
>> occur. If I try it directly to the internal IP address (as I have to
>> without netfilter loaded) then of course nothing fails.
>>
>> It's a pain in the bum as a fault, but it's one I can easily reproduce
>> as long as I use the same set of circumstances.
>>
>> I'll try using 3.0-rc2 (current git) tonight, and if I can reproduce it
>> on that then I'll attempt to pare down the IPTABLES rules to a bare
>> minimum.
>>
>> It is nothing to do with ebtables as I don't compile it. I'm not really
>> sure about "bridging firewall" functionality. I just use a couple of
>> hand coded bash scripts to set the tables up.
>
>  From one of your previous mails:
>
>> # CONFIG_BRIDGE_NF_EBTABLES is not set
>
> How about CONFIG_BRIDGE_NETFILTER?
>

It was compiled in.

With the following table set I was able to reproduce the problem on 
3.0-rc2. Replaced my IP with xxx.xxx.xxx.xxx, but otherwise unmodified

root@srv:~# iptables-save
# Generated by iptables-save v1.4.10 on Tue Jun  7 22:11:30 2011
*filter
:INPUT ACCEPT [978:107619]
:FORWARD ACCEPT [142:7068]
:OUTPUT ACCEPT [1659:291870]
-A INPUT -i ppp0 -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT ! -i ppp0 -m state --state NEW -j ACCEPT
-A INPUT -i ppp0 -j DROP
COMMIT
# Completed on Tue Jun  7 22:11:30 2011
# Generated by iptables-save v1.4.10 on Tue Jun  7 22:11:30 2011
*nat
:PREROUTING ACCEPT [813:49170]
:INPUT ACCEPT [91:7090]
:OUTPUT ACCEPT [267:20731]
:POSTROUTING ACCEPT [296:22281]
-A PREROUTING -d xxx.xxx.xxx.xxx/32 ! -i ppp0 -p tcp -m tcp --dport 443 
-j DNAT --to-destination 192.168.253.198
COMMIT
# Completed on Tue Jun  7 22:11:30 2011
# Generated by iptables-save v1.4.10 on Tue Jun  7 22:11:30 2011
*mangle
:PREROUTING ACCEPT [2729:274392]
:INPUT ACCEPT [2508:262976]
:FORWARD ACCEPT [142:7068]
:OUTPUT ACCEPT [1674:293701]
:POSTROUTING ACCEPT [2131:346411]
-A FORWARD -o ppp0 -p tcp -m tcp --tcp-flags SYN,RST SYN -m tcpmss --mss 
1400:1536 -j TCPMSS --clamp-mss-to-pmtu
COMMIT
# Completed on Tue Jun  7 22:11:30 2011

I've just compiled out CONFIG_BRIDGE_NETFILTER and can no longer access 
the address the way I was doing it, so that's a no-go for me.


--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
  2011-06-07 13:37                                             ` Eric Dumazet
@ 2011-06-07 15:15                                               ` Brad Campbell
  2011-08-20 13:16                                               ` Brad Campbell
  1 sibling, 0 replies; 111+ messages in thread
From: Brad Campbell @ 2011-06-07 15:15 UTC (permalink / raw)
  To: Eric Dumazet; +Cc: Avi Kivity, CaT, Borislav Petkov, linux-kernel, kvm, netdev

On 07/06/11 21:37, Eric Dumazet wrote:
> Le mardi 07 juin 2011 à 21:27 +0800, Brad Campbell a écrit :
>> On 07/06/11 04:22, Eric Dumazet wrote:
>>
>>> Could you please try latest linux-2.6 tree ?
>>>
>>> We fixed many networking bugs that could explain your crash.
>>>
>>>
>>>
>>>
>>
>> No good I'm afraid.
>>
>> [  543.040056]
>> =============================================================================
>> [  543.040136] BUG ip_dst_cache: Padding overwritten.
>> 0xffff8803e4217ffe-0xffff8803e4217fff
>> [  543.040194]
>
> Thats pretty strange : These are the last two bytes of a page, set to
> 0x0000 (a 16 bit value)
>
> There is no way a dst field could actually sit on this location (its a
> padding), since a dst is a bit less than 256 bytes (0xe8), and each
> entry is aligned on a 64byte address.
>
> grep dst /proc/slabinfo
>
> ip_dst_cache       32823  62944    256   32    2 : tunables    0    0
> 0 : slabdata   1967   1967      0
>
> sizeof(struct rtable)=0xe8
>
>
>> -----------------------------------------------------------------------------
>> [  543.040198]
>> [  543.040298] INFO: Slab 0xffffea000d9e74d0 objects=25 used=25 fp=0x
>>          (null) flags=0x8000000000004081
>> [  543.040364] Pid: 4576, comm: kworker/1:2 Not tainted 3.0.0-rc2 #1
>> [  543.040415] Call Trace:
>> [  543.040472]  [<ffffffff810b9c1d>] ? slab_err+0xad/0xd0
>> [  543.040528]  [<ffffffff8102e034>] ? check_preempt_wakeup+0xa4/0x160
>> [  543.040595]  [<ffffffff810ba206>] ? slab_pad_check+0x126/0x170
>> [  543.040650]  [<ffffffff8133045b>] ? dst_destroy+0x8b/0x110
>> [  543.040701]  [<ffffffff810ba29a>] ? check_slab+0x4a/0xc0
>> [  543.040753]  [<ffffffff810baf2d>] ? free_debug_processing+0x2d/0x250
>> [  543.040808]  [<ffffffff810bb27b>] ? __slab_free+0x12b/0x140
>> [  543.040862]  [<ffffffff810bbe99>] ? kmem_cache_free+0x99/0xa0
>> [  543.040915]  [<ffffffff8133045b>] ? dst_destroy+0x8b/0x110
>> [  543.040967]  [<ffffffff813307f6>] ? dst_gc_task+0x196/0x1f0
>> [  543.041021]  [<ffffffff8104e954>] ? queue_delayed_work_on+0x154/0x160
>> [  543.041081]  [<ffffffff813066fe>] ? do_dbs_timer+0x20e/0x3d0
>> [  543.041133]  [<ffffffff81330660>] ? dst_alloc+0x180/0x180
>> [  543.041187]  [<ffffffff8104f28b>] ? process_one_work+0xfb/0x3b0
>> [  543.041242]  [<ffffffff8104f964>] ? worker_thread+0x144/0x3d0
>> [  543.041296]  [<ffffffff8102cc10>] ? __wake_up_common+0x50/0x80
>> [  543.041678]  [<ffffffff8104f820>] ? rescuer_thread+0x2e0/0x2e0
>> [  543.041729]  [<ffffffff8104f820>] ? rescuer_thread+0x2e0/0x2e0
>> [  543.041782]  [<ffffffff81053436>] ? kthread+0x96/0xa0
>> [  543.041835]  [<ffffffff813e1d14>] ? kernel_thread_helper+0x4/0x10
>> [  543.041890]  [<ffffffff810533a0>] ? kthread_worker_fn+0x120/0x120
>> [  543.041944]  [<ffffffff813e1d10>] ? gs_change+0xb/0xb
>> [  543.041993]  Padding 0xffff8803e4217f40:  5a 5a 5a 5a 5a 5a 5a 5a 5a
>> 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
>> [  543.042718]  Padding 0xffff8803e4217f50:  5a 5a 5a 5a 5a 5a 5a 5a 5a
>> 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
>> [  543.043433]  Padding 0xffff8803e4217f60:  5a 5a 5a 5a 5a 5a 5a 5a 5a
>> 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
>> [  543.044155]  Padding 0xffff8803e4217f70:  5a 5a 5a 5a 5a 5a 5a 5a 5a
>> 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
>> [  543.044866]  Padding 0xffff8803e4217f80:  5a 5a 5a 5a 5a 5a 5a 5a 5a
>> 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
>> [  543.045590]  Padding 0xffff8803e4217f90:  5a 5a 5a 5a 5a 5a 5a 5a 5a
>> 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
>> [  543.046311]  Padding 0xffff8803e4217fa0:  5a 5a 5a 5a 5a 5a 5a 5a 5a
>> 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
>> [  543.047034]  Padding 0xffff8803e4217fb0:  5a 5a 5a 5a 5a 5a 5a 5a 5a
>> 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
>> [  543.047755]  Padding 0xffff8803e4217fc0:  5a 5a 5a 5a 5a 5a 5a 5a 5a
>> 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
>> [  543.048474]  Padding 0xffff8803e4217fd0:  5a 5a 5a 5a 5a 5a 5a 5a 5a
>> 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
>> [  543.049203]  Padding 0xffff8803e4217fe0:  5a 5a 5a 5a 5a 5a 5a 5a 5a
>> 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
>> [  543.049909]  Padding 0xffff8803e4217ff0:  5a 5a 5a 5a 5a 5a 5a 5a 5a
>> 5a 5a 5a 5a 5a 00 00 ZZZZZZZZZZZZZZ..
>> [  543.050021] FIX ip_dst_cache: Restoring
>> 0xffff8803e4217f40-0xffff8803e4217fff=0x5a
>> [  543.050021]
>>
>> Dropped -mm, Hugh and Andrea from CC as this does not appear to be mm or
>> ksm related.
>>
>> I'll pare down the firewall and see if I can make it break easier with a
>> smaller test set.
>
> Hmm, not sure now :(
>
> Could you reproduce another bug please ?

Here is the remainder of the log from that bug that I managed to scrape 
off the disk. Odd as usually the netconsole dies and so does the disk 
subsystem. In this case it managed to get the log to disk.

[  543.040056] 
=============================================================================
[  543.040136] BUG ip_dst_cache: Padding overwritten. 
0xffff8803e4217ffe-0xffff8803e4217fff
[  543.040194] 
-----------------------------------------------------------------------------
[  543.040198]
[  543.040298] INFO: Slab 0xffffea000d9e74d0 objects=25 used=25 fp=0x 
        (null) flags=0x8000000000004081
[  543.040364] Pid: 4576, comm: kworker/1:2 Not tainted 3.0.0-rc2 #1
[  543.040415] Call Trace:
[  543.040472]  [<ffffffff810b9c1d>] ? slab_err+0xad/0xd0
[  543.040528]  [<ffffffff8102e034>] ? check_preempt_wakeup+0xa4/0x160
[  543.040595]  [<ffffffff810ba206>] ? slab_pad_check+0x126/0x170
[  543.040650]  [<ffffffff8133045b>] ? dst_destroy+0x8b/0x110
[  543.040701]  [<ffffffff810ba29a>] ? check_slab+0x4a/0xc0
[  543.040753]  [<ffffffff810baf2d>] ? free_debug_processing+0x2d/0x250
[  543.040808]  [<ffffffff810bb27b>] ? __slab_free+0x12b/0x140
[  543.040862]  [<ffffffff810bbe99>] ? kmem_cache_free+0x99/0xa0
[  543.040915]  [<ffffffff8133045b>] ? dst_destroy+0x8b/0x110
[  543.040967]  [<ffffffff813307f6>] ? dst_gc_task+0x196/0x1f0
[  543.041021]  [<ffffffff8104e954>] ? queue_delayed_work_on+0x154/0x160
[  543.041081]  [<ffffffff813066fe>] ? do_dbs_timer+0x20e/0x3d0
[  543.041133]  [<ffffffff81330660>] ? dst_alloc+0x180/0x180
[  543.041187]  [<ffffffff8104f28b>] ? process_one_work+0xfb/0x3b0
[  543.041242]  [<ffffffff8104f964>] ? worker_thread+0x144/0x3d0
[  543.041296]  [<ffffffff8102cc10>] ? __wake_up_common+0x50/0x80
[  543.041678]  [<ffffffff8104f820>] ? rescuer_thread+0x2e0/0x2e0
[  543.041729]  [<ffffffff8104f820>] ? rescuer_thread+0x2e0/0x2e0
[  543.041782]  [<ffffffff81053436>] ? kthread+0x96/0xa0
[  543.041835]  [<ffffffff813e1d14>] ? kernel_thread_helper+0x4/0x10
[  543.041890]  [<ffffffff810533a0>] ? kthread_worker_fn+0x120/0x120
[  543.041944]  [<ffffffff813e1d10>] ? gs_change+0xb/0xb
[  543.041993]  Padding 0xffff8803e4217f40:  5a 5a 5a 5a 5a 5a 5a 5a 5a 
5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
[  543.042718]  Padding 0xffff8803e4217f50:  5a 5a 5a 5a 5a 5a 5a 5a 5a 
5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
[  543.043433]  Padding 0xffff8803e4217f60:  5a 5a 5a 5a 5a 5a 5a 5a 5a 
5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
[  543.044155]  Padding 0xffff8803e4217f70:  5a 5a 5a 5a 5a 5a 5a 5a 5a 
5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
[  543.044866]  Padding 0xffff8803e4217f80:  5a 5a 5a 5a 5a 5a 5a 5a 5a 
5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
[  543.045590]  Padding 0xffff8803e4217f90:  5a 5a 5a 5a 5a 5a 5a 5a 5a 
5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
[  543.046311]  Padding 0xffff8803e4217fa0:  5a 5a 5a 5a 5a 5a 5a 5a 5a 
5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
[  543.047034]  Padding 0xffff8803e4217fb0:  5a 5a 5a 5a 5a 5a 5a 5a 5a 
5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
[  543.047755]  Padding 0xffff8803e4217fc0:  5a 5a 5a 5a 5a 5a 5a 5a 5a 
5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
[  543.048474]  Padding 0xffff8803e4217fd0:  5a 5a 5a 5a 5a 5a 5a 5a 5a 
5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
[  543.049203]  Padding 0xffff8803e4217fe0:  5a 5a 5a 5a 5a 5a 5a 5a 5a 
5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
[  543.049909]  Padding 0xffff8803e4217ff0:  5a 5a 5a 5a 5a 5a 5a 5a 5a 
5a 5a 5a 5a 5a 00 00 ZZZZZZZZZZZZZZ..
[  543.050021] FIX ip_dst_cache: Restoring 
0xffff8803e4217f40-0xffff8803e4217fff=0x5a
[  543.050021]
[  885.504318] br1: port 4(vnet2) entering forwarding state
[  885.504797] device vnet2 left promiscuous mode
[  885.504868] br1: port 4(vnet2) entering disabled state
[  885.847186] br1: port 3(vnet1) entering forwarding state
[  885.847667] device vnet1 left promiscuous mode
[  885.847740] br1: port 3(vnet1) entering disabled state
[  885.940042] 
=============================================================================
[  885.940122] BUG kmalloc-512: Object padding overwritten
[  885.940171] 
-----------------------------------------------------------------------------
[  885.940175]
[  885.940273] INFO: 0xffff8803dc09322e-0xffff8803dc09322f. First byte 
0x0 instead of 0x5a
[  885.940344] INFO: Allocated in qdisc_alloc+0x59/0x110 age=66496 cpu=2 
pid=4948
[  885.940404] INFO: Freed in qdisc_alloc+0x49/0x110 age=66496 cpu=2 
pid=4948
[  885.940457] INFO: Slab 0xffffea000d821f80 objects=28 used=21 
fp=0xffff8803dc090b68 flags=0x8000000000004081
[  885.940519] INFO: Object 0xffff8803dc092fe8 @offset=12264 
fp=0xffff8803dc092238
[  885.940523]
[  885.940618] Bytes b4 0xffff8803dc092fd8:  d3 e0 ff ff 00 00 00 00 5a 
5a 5a 5a 5a 5a 5a 5a Óàÿÿ....ZZZZZZZZ
[  885.941373]   Object 0xffff8803dc092fe8:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[  885.942120]   Object 0xffff8803dc092ff8:  00 00 00 00 00 00 00 00 70 
0a 34 81 ff ff ff ff ........p.4.ÿÿÿÿ
[  885.942847]   Object 0xffff8803dc093008:  b0 08 34 81 ff ff ff ff 04 
00 00 00 18 00 00 00 °.4.ÿÿÿÿ........
[  885.943585]   Object 0xffff8803dc093018:  60 ac 5d 81 ff ff ff ff 00 
00 00 00 00 00 00 00 `¬].ÿÿÿÿ........
[  885.944324]   Object 0xffff8803dc093028:  28 30 09 dc 03 88 ff ff 28 
30 09 dc 03 88 ff ff (0.Ü..ÿÿ(0.Ü..ÿÿ
[  885.945071]   Object 0xffff8803dc093038:  00 00 00 00 ff ff ff ff 00 
00 00 00 00 00 00 00 ....ÿÿÿÿ........
[  885.945794]   Object 0xffff8803dc093048:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[  885.946530]   Object 0xffff8803dc093058:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[  885.947281]   Object 0xffff8803dc093068:  98 f3 f9 0e 04 88 ff ff 00 
00 00 00 00 00 00 00 .óù...ÿÿ........
[  885.948011]   Object 0xffff8803dc093078:  00 00 00 00 00 00 00 00 02 
00 00 00 00 00 00 00 ................
[  885.948735]   Object 0xffff8803dc093088:  88 30 09 dc 03 88 ff ff 88 
30 09 dc 03 88 ff ff .0.Ü..ÿÿ.0.Ü..ÿÿ
[  885.949483]   Object 0xffff8803dc093098:  00 00 00 00 bf bf 00 00 ca 
29 01 00 00 00 00 00 ....¿¿..Ê)......
[  885.950007]   Object 0xffff8803dc0930a8:  de 01 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 Þ...............
[  885.950007]   Object 0xffff8803dc0930b8:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[  885.950007]   Object 0xffff8803dc0930c8:  10 cd 13 e7 03 88 ff ff a0 
13 34 81 ff ff ff ff .Í.ç..ÿÿ..4.ÿÿÿÿ
[  885.950007]   Object 0xffff8803dc0930d8:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[  885.950007]   Object 0xffff8803dc0930e8:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[  885.950007]   Object 0xffff8803dc0930f8:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[  885.950007]   Object 0xffff8803dc093108:  08 31 09 dc 03 88 ff ff 08 
31 09 dc 03 88 ff ff .1.Ü..ÿÿ.1.Ü..ÿÿ
[  885.950007]   Object 0xffff8803dc093118:  00 00 00 00 00 00 00 00 20 
31 09 dc 03 88 ff ff .........1.Ü..ÿÿ
[  885.950007]   Object 0xffff8803dc093128:  20 31 09 dc 03 88 ff ff 00 
00 00 00 00 00 00 00 .1.Ü..ÿÿ........
[  885.950007]   Object 0xffff8803dc093138:  38 31 09 dc 03 88 ff ff 38 
31 09 dc 03 88 ff ff 81.Ü..ÿÿ81.Ü..ÿÿ
[  885.950007]   Object 0xffff8803dc093148:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[  885.950007]   Object 0xffff8803dc093158:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[  885.950007]   Object 0xffff8803dc093168:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[  885.950007]   Object 0xffff8803dc093178:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[  885.950007]   Object 0xffff8803dc093188:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[  885.950007]   Object 0xffff8803dc093198:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[  885.950007]   Object 0xffff8803dc0931a8:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[  885.950007]   Object 0xffff8803dc0931b8:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[  885.950007]   Object 0xffff8803dc0931c8:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[  885.950007]   Object 0xffff8803dc0931d8:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[  885.950007]  Redzone 0xffff8803dc0931e8:  cc cc cc cc cc cc cc cc 
                      ÌÌÌÌÌÌÌÌ
[  885.950007]  Padding 0xffff8803dc093228:  5a 5a 5a 5a 5a 5a 00 00 
                      ZZZZZZ..
[  885.950007] Pid: 11, comm: rcuc1 Not tainted 3.0.0-rc2 #1
[  885.950007] Call Trace:
[  885.950007]  [<ffffffff810ba41d>] ? check_bytes_and_report+0x10d/0x150
[  885.950007]  [<ffffffff810ba509>] ? check_object+0xa9/0x260
[  885.950007]  [<ffffffff81075a49>] ? __rcu_process_callbacks+0x119/0x340
[  885.950007]  [<ffffffff810bb033>] ? free_debug_processing+0x133/0x250
[  885.950007]  [<ffffffff810bb27b>] ? __slab_free+0x12b/0x140
[  885.950007]  [<ffffffff81075a49>] ? __rcu_process_callbacks+0x119/0x340
[  885.950007]  [<ffffffff8107666b>] ? rcu_cpu_kthread+0x27b/0x3a0
[  885.950007]  [<ffffffff8102d5b9>] ? ttwu_do_wakeup+0x19/0xa0
[  885.950007]  [<ffffffff81074a80>] ? invoke_rcu_cpu_kthread+0x30/0x30
[  885.950007]  [<ffffffff810763f0>] ? rcu_check_callbacks+0x100/0x100
[  885.950007]  [<ffffffff810763f0>] ? rcu_check_callbacks+0x100/0x100
[  885.950007]  [<ffffffff81053436>] ? kthread+0x96/0xa0
[  885.950007]  [<ffffffff813e1d14>] ? kernel_thread_helper+0x4/0x10
[  885.950007]  [<ffffffff810533a0>] ? kthread_worker_fn+0x120/0x120
[  885.950007]  [<ffffffff813e1d10>] ? gs_change+0xb/0xb
[  885.950007] FIX kmalloc-512: Restoring 
0xffff8803dc09322e-0xffff8803dc09322f=0x5a
[  885.950007]
[  886.050119] 
=============================================================================
[  886.050198] BUG kmalloc-512: Object padding overwritten
[  886.050247] 
-----------------------------------------------------------------------------
[  886.050251]
[  886.050353] INFO: 0xffff8803dc091916-0xffff8803dc091917. First byte 
0x0 instead of 0x5a
[  886.050451] INFO: Allocated in kvm_ioapic_init+0x3c/0xd0 [kvm] 
age=66256 cpu=0 pid=5122
[  886.050517] INFO: Freed in __kfree_skb+0x11/0x90 age=66311 cpu=4 pid=5054
[  886.050568] INFO: Slab 0xffffea000d821f80 objects=28 used=17 
fp=0xffff8803dc091488 flags=0x8000000000004081
[  886.050630] INFO: Object 0xffff8803dc0916d0 @offset=5840 fp=0x 
    (null)
[  886.050634]
[  886.050726] Bytes b4 0xffff8803dc0916c0:  ea e4 00 00 01 00 00 00 5a 
5a 5a 5a 5a 5a 5a 5a êä......ZZZZZZZZ
[  886.051451]   Object 0xffff8803dc0916d0:  00 00 c0 fe 00 00 00 00 3f 
00 00 00 00 00 00 00 ..Àþ....?.......
[  886.052183]   Object 0xffff8803dc0916e0:  00 08 00 00 00 00 00 00 00 
00 01 00 00 00 00 00 ................
[  886.052898]   Object 0xffff8803dc0916f0:  00 00 01 00 00 00 00 00 00 
00 01 00 00 00 00 00 ................
[  886.053613]   Object 0xffff8803dc091700:  00 00 01 00 00 00 00 00 00 
00 01 00 00 00 00 00 ................
[  886.054325]   Object 0xffff8803dc091710:  00 00 01 00 00 00 00 00 00 
00 01 00 00 00 00 00 ................
[  886.055032]   Object 0xffff8803dc091720:  00 00 01 00 00 00 00 00 00 
00 01 00 00 00 00 00 ................
[  886.055743]   Object 0xffff8803dc091730:  00 00 01 00 00 00 00 00 00 
00 01 00 00 00 00 00 ................
[  886.056462]   Object 0xffff8803dc091740:  00 00 01 00 00 00 00 00 00 
00 01 00 00 00 00 00 ................
[  886.057177]   Object 0xffff8803dc091750:  00 00 01 00 00 00 00 00 00 
00 01 00 00 00 00 00 ................
[  886.058210]   Object 0xffff8803dc091760:  00 00 01 00 00 00 00 00 00 
00 01 00 00 00 00 00 ................
[  886.058957]   Object 0xffff8803dc091770:  00 00 01 00 00 00 00 00 00 
00 01 00 00 00 00 00 ................
[  886.059667]   Object 0xffff8803dc091780:  00 00 01 00 00 00 00 00 00 
00 01 00 00 00 00 00 ................
[  886.060104]   Object 0xffff8803dc091790:  00 00 01 00 00 00 00 00 00 
00 01 00 00 00 00 00 ................
[  886.060104]   Object 0xffff8803dc0917a0:  00 00 01 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[  886.060104]   Object 0xffff8803dc0917b0:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[  886.060104]   Object 0xffff8803dc0917c0:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[  886.060104]   Object 0xffff8803dc0917d0:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[  886.060104]   Object 0xffff8803dc0917e0:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[  886.060104]   Object 0xffff8803dc0917f0:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[  886.060104]   Object 0xffff8803dc091800:  01 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[  886.060104]   Object 0xffff8803dc091810:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[  886.060104]   Object 0xffff8803dc091820:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[  886.060104]   Object 0xffff8803dc091830:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[  886.060104]   Object 0xffff8803dc091840:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[  886.060104]   Object 0xffff8803dc091850:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[  886.060104]   Object 0xffff8803dc091860:  00 00 00 00 00 00 00 00 b0 
80 10 a0 ff ff ff ff ........°...ÿÿÿÿ
[  886.060104]   Object 0xffff8803dc091870:  00 00 df f5 03 88 ff ff 00 
00 00 00 00 00 00 00 ..ßõ..ÿÿ........
[  886.060104]   Object 0xffff8803dc091880:  43 43 00 00 00 00 00 00 01 
00 00 00 00 00 00 00 CC..............
[  886.060104]   Object 0xffff8803dc091890:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[  886.060104]   Object 0xffff8803dc0918a0:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[  886.060104]   Object 0xffff8803dc0918b0:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[  886.060104]   Object 0xffff8803dc0918c0:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[  886.060104]  Redzone 0xffff8803dc0918d0:  cc cc cc cc cc cc cc cc 
                      ÌÌÌÌÌÌÌÌ
[  886.060104]  Padding 0xffff8803dc091910:  5a 5a 5a 5a 5a 5a 00 00 
                      ZZZZZZ..
[  886.060104] Pid: 7359, comm: qemu Not tainted 3.0.0-rc2 #1
[  886.060104] Call Trace:
[  886.060104]  [<ffffffff810ba41d>] ? check_bytes_and_report+0x10d/0x150
[  886.060104]  [<ffffffff810ba509>] ? check_object+0xa9/0x260
[  886.060104]  [<ffffffffa00eb8bf>] ? kvm_arch_destroy_vm+0x2f/0x150 [kvm]
[  886.060104]  [<ffffffff810bb033>] ? free_debug_processing+0x133/0x250
[  886.060104]  [<ffffffff810bb27b>] ? __slab_free+0x12b/0x140
[  886.060104]  [<ffffffffa00eb8bf>] ? kvm_arch_destroy_vm+0x2f/0x150 [kvm]
[  886.060104]  [<ffffffffa00e2dd9>] ? kvm_put_kvm+0xe9/0x130 [kvm]
[  886.060104]  [<ffffffffa00e3763>] ? kvm_vcpu_release+0x13/0x20 [kvm]
[  886.060104]  [<ffffffff810c65fd>] ? fput+0xdd/0x270
[  886.060104]  [<ffffffff810c2e1c>] ? filp_close+0x5c/0x90
[  886.060104]  [<ffffffff8103ae1d>] ? put_files_struct+0xcd/0x100
[  886.060104]  [<ffffffff8103ccc7>] ? do_exit+0x6d7/0x7e0
[  886.060104]  [<ffffffff8103ce20>] ? do_group_exit+0x50/0xc0
[  886.060104]  [<ffffffff81048914>] ? get_signal_to_deliver+0x1a4/0x3a0
[  886.060104]  [<ffffffff8100188a>] ? do_signal+0x8a/0x7d0
[  886.060104]  [<ffffffff81049148>] ? kill_pid_info+0x38/0x60
[  886.060104]  [<ffffffff810493e4>] ? sys_kill+0x94/0x1b0
[  886.060104]  [<ffffffff8100201d>] ? do_notify_resume+0x4d/0x70
[  886.060104]  [<ffffffff810c59c4>] ? sys_pread64+0x74/0xa0
[  886.060104]  [<ffffffff813e11db>] ? int_signal+0x12/0x17
[  886.060104] FIX kmalloc-512: Restoring 
0xffff8803dc091916-0xffff8803dc091917=0x5a
[  886.060104]
[  926.148014] br1: port 6(vnet4) entering forwarding state
[  926.148385] device vnet4 left promiscuous mode
[  926.148457] br1: port 6(vnet4) entering disabled state
[  945.659523] br1: port 2(vnet0) entering forwarding state
[  945.660041] device vnet0 left promiscuous mode
[  945.660116] br1: port 2(vnet0) entering disabled state
[ 1121.085267] br1: port 5(vnet3) entering forwarding state
[ 1121.085783] device vnet3 left promiscuous mode
[ 1121.085860] br1: port 5(vnet3) entering disabled state
[ 1176.307099] nfsd: last server has exited, flushing export cache
[ 1176.430113] 
=============================================================================
[ 1176.430194] BUG kmalloc-512: Object padding overwritten
[ 1176.430199] 
-----------------------------------------------------------------------------
[ 1176.430204]
[ 1176.430210] INFO: 0xffff8803f5df5b5e-0xffff8803f5df5b5f. First byte 
0x0 instead of 0x5a
[ 1176.430225] INFO: Allocated in sock_alloc_send_pskb+0x1d0/0x320 
age=93370 cpu=0 pid=5126
[ 1176.430234] INFO: Freed in __kfree_skb+0x11/0x90 age=93370 cpu=0 pid=4495
[ 1176.430240] INFO: Slab 0xffffea000ddc8d60 objects=28 used=0 
fp=0xffff8803f5df76c0 flags=0x8000000000004080
[ 1176.430246] INFO: Object 0xffff8803f5df5918 @offset=6424 
fp=0xffff8803f5df7908
[ 1176.430249]
[ 1176.430253] Bytes b4 0xffff8803f5df5908:  ab f0 ff ff 00 00 00 00 5a 
5a 5a 5a 5a 5a 5a 5a «ðÿÿ....ZZZZZZZZ
[ 1176.430270]   Object 0xffff8803f5df5918:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.430285]   Object 0xffff8803f5df5928:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.430300]   Object 0xffff8803f5df5938:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.430314]   Object 0xffff8803f5df5948:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.430328]   Object 0xffff8803f5df5958:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.430343]   Object 0xffff8803f5df5968:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.430357]   Object 0xffff8803f5df5978:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.430371]   Object 0xffff8803f5df5988:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.430386]   Object 0xffff8803f5df5998:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.430400]   Object 0xffff8803f5df59a8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.430414]   Object 0xffff8803f5df59b8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.430429]   Object 0xffff8803f5df59c8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.430443]   Object 0xffff8803f5df59d8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.430457]   Object 0xffff8803f5df59e8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.430471]   Object 0xffff8803f5df59f8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.430485]   Object 0xffff8803f5df5a08:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.430500]   Object 0xffff8803f5df5a18:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.430514]   Object 0xffff8803f5df5a28:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.430528]   Object 0xffff8803f5df5a38:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.430542]   Object 0xffff8803f5df5a48:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.430557]   Object 0xffff8803f5df5a58:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.430571]   Object 0xffff8803f5df5a68:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.430585]   Object 0xffff8803f5df5a78:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.430599]   Object 0xffff8803f5df5a88:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.430614]   Object 0xffff8803f5df5a98:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.430628]   Object 0xffff8803f5df5aa8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.430642]   Object 0xffff8803f5df5ab8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.430656]   Object 0xffff8803f5df5ac8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.430671]   Object 0xffff8803f5df5ad8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.430685]   Object 0xffff8803f5df5ae8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.430699]   Object 0xffff8803f5df5af8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.430714]   Object 0xffff8803f5df5b08:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b a5 kkkkkkkkkkkkkkk¥
[ 1176.430729]  Redzone 0xffff8803f5df5b18:  bb bb bb bb bb bb bb bb 
                      »»»»»»»»
[ 1176.430742]  Padding 0xffff8803f5df5b58:  5a 5a 5a 5a 5a 5a 00 00 
                      ZZZZZZ..
[ 1176.430758] Pid: 24, comm: rcuc4 Not tainted 3.0.0-rc2 #1
[ 1176.430762] Call Trace:
[ 1176.430774]  [<ffffffff810ba41d>] ? check_bytes_and_report+0x10d/0x150
[ 1176.430782]  [<ffffffff810ba509>] ? check_object+0xa9/0x260
[ 1176.430789]  [<ffffffff810bac15>] ? init_object+0x85/0xa0
[ 1176.430795]  [<ffffffff810ba816>] ? __free_slab+0x156/0x180
[ 1176.430803]  [<ffffffff81075a49>] ? __rcu_process_callbacks+0x119/0x340
[ 1176.430810]  [<ffffffff8107666b>] ? rcu_cpu_kthread+0x27b/0x3a0
[ 1176.430818]  [<ffffffff8102d5b9>] ? ttwu_do_wakeup+0x19/0xa0
[ 1176.430825]  [<ffffffff81074a80>] ? invoke_rcu_cpu_kthread+0x30/0x30
[ 1176.430832]  [<ffffffff810763f0>] ? rcu_check_callbacks+0x100/0x100
[ 1176.430838]  [<ffffffff810763f0>] ? rcu_check_callbacks+0x100/0x100
[ 1176.430846]  [<ffffffff81053436>] ? kthread+0x96/0xa0
[ 1176.430855]  [<ffffffff813e1d14>] ? kernel_thread_helper+0x4/0x10
[ 1176.430863]  [<ffffffff810533a0>] ? kthread_worker_fn+0x120/0x120
[ 1176.430870]  [<ffffffff813e1d10>] ? gs_change+0xb/0xb
[ 1176.430876] FIX kmalloc-512: Restoring 
0xffff8803f5df5b5e-0xffff8803f5df5b5f=0x5a
[ 1176.430879]
[ 1176.430897] 
=============================================================================
[ 1176.430902] BUG kmalloc-512: Object padding overwritten
[ 1176.430905] 
-----------------------------------------------------------------------------
[ 1176.430909]
[ 1176.430912] INFO: 0xffff8803f5df6fe6-0xffff8803f5df6fe7. First byte 
0x0 instead of 0x5a
[ 1176.430921] INFO: Allocated in sock_alloc_send_pskb+0x1d0/0x320 
age=93370 cpu=0 pid=5126
[ 1176.430928] INFO: Freed in __kfree_skb+0x11/0x90 age=93370 cpu=0 pid=4495
[ 1176.430935] INFO: Slab 0xffffea000ddc8d60 objects=28 used=0 
fp=0xffff8803f5df76c0 flags=0x8000000000004080
[ 1176.430940] INFO: Object 0xffff8803f5df6da0 @offset=11680 
fp=0xffff8803f5df5488
[ 1176.430943]
[ 1176.430946] Bytes b4 0xffff8803f5df6d90:  cd 7a 00 00 01 00 00 00 5a 
5a 5a 5a 5a 5a 5a 5a Íz......ZZZZZZZZ
[ 1176.430962]   Object 0xffff8803f5df6da0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.430977]   Object 0xffff8803f5df6db0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.430992]   Object 0xffff8803f5df6dc0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.431006]   Object 0xffff8803f5df6dd0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.431021]   Object 0xffff8803f5df6de0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.431036]   Object 0xffff8803f5df6df0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.431050]   Object 0xffff8803f5df6e00:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.431065]   Object 0xffff8803f5df6e10:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.431080]   Object 0xffff8803f5df6e20:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.431094]   Object 0xffff8803f5df6e30:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.431109]   Object 0xffff8803f5df6e40:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.431124]   Object 0xffff8803f5df6e50:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.431138]   Object 0xffff8803f5df6e60:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.431153]   Object 0xffff8803f5df6e70:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.431168]   Object 0xffff8803f5df6e80:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.431182]   Object 0xffff8803f5df6e90:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.431197]   Object 0xffff8803f5df6ea0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.431213]   Object 0xffff8803f5df6eb0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.431228]   Object 0xffff8803f5df6ec0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.431243]   Object 0xffff8803f5df6ed0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.431258]   Object 0xffff8803f5df6ee0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.431273]   Object 0xffff8803f5df6ef0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.431287]   Object 0xffff8803f5df6f00:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.431303]   Object 0xffff8803f5df6f10:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.431317]   Object 0xffff8803f5df6f20:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.431332]   Object 0xffff8803f5df6f30:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.431346]   Object 0xffff8803f5df6f40:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.431361]   Object 0xffff8803f5df6f50:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.431376]   Object 0xffff8803f5df6f60:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.431391]   Object 0xffff8803f5df6f70:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.431405]   Object 0xffff8803f5df6f80:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.431420]   Object 0xffff8803f5df6f90:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b a5 kkkkkkkkkkkkkkk¥
[ 1176.431435]  Redzone 0xffff8803f5df6fa0:  bb bb bb bb bb bb bb bb 
                      »»»»»»»»
[ 1176.431448]  Padding 0xffff8803f5df6fe0:  5a 5a 5a 5a 5a 5a 00 00 
                      ZZZZZZ..
[ 1176.431463] Pid: 24, comm: rcuc4 Not tainted 3.0.0-rc2 #1
[ 1176.431467] Call Trace:
[ 1176.431474]  [<ffffffff810ba41d>] ? check_bytes_and_report+0x10d/0x150
[ 1176.431481]  [<ffffffff810ba509>] ? check_object+0xa9/0x260
[ 1176.431487]  [<ffffffff810bac15>] ? init_object+0x85/0xa0
[ 1176.431494]  [<ffffffff810ba816>] ? __free_slab+0x156/0x180
[ 1176.431501]  [<ffffffff81075a49>] ? __rcu_process_callbacks+0x119/0x340
[ 1176.431508]  [<ffffffff8107666b>] ? rcu_cpu_kthread+0x27b/0x3a0
[ 1176.431515]  [<ffffffff8102d5b9>] ? ttwu_do_wakeup+0x19/0xa0
[ 1176.431522]  [<ffffffff81074a80>] ? invoke_rcu_cpu_kthread+0x30/0x30
[ 1176.431529]  [<ffffffff810763f0>] ? rcu_check_callbacks+0x100/0x100
[ 1176.431535]  [<ffffffff810763f0>] ? rcu_check_callbacks+0x100/0x100
[ 1176.431543]  [<ffffffff81053436>] ? kthread+0x96/0xa0
[ 1176.431550]  [<ffffffff813e1d14>] ? kernel_thread_helper+0x4/0x10
[ 1176.431558]  [<ffffffff810533a0>] ? kthread_worker_fn+0x120/0x120
[ 1176.431565]  [<ffffffff813e1d10>] ? gs_change+0xb/0xb
[ 1176.431571] FIX kmalloc-512: Restoring 
0xffff8803f5df6fe6-0xffff8803f5df6fe7=0x5a
[ 1176.431574]
[ 1179.773660] 
=============================================================================
[ 1179.773742] BUG kmalloc-1024: Object padding overwritten
[ 1179.773792] 
-----------------------------------------------------------------------------
[ 1179.773796]
[ 1179.773901] INFO: 0xffff8803e430e6be-0xffff8803e430e6bf. First byte 
0x0 instead of 0x5a
[ 1179.773972] INFO: Allocated in tty_buffer_request_room+0xcc/0x1a0 
age=93277 cpu=1 pid=5109
[ 1179.774042] INFO: Freed in __blockdev_direct_IO+0xa16/0xa90 age=93277 
cpu=5 pid=5146
[ 1179.774102] INFO: Slab 0xffffea000d9ea9c0 objects=29 used=14 
fp=0xffff8803e430d9e8 flags=0x8000000000004081
[ 1179.774163] INFO: Object 0xffff8803e430e278 @offset=25208 
fp=0xffff8803e4309df8
[ 1179.774166]
[ 1179.774260] Bytes b4 0xffff8803e430e268:  a5 57 01 00 01 00 00 00 5a 
5a 5a 5a 5a 5a 5a 5a ¥W......ZZZZZZZZ
[ 1179.775297]   Object 0xffff8803e430e278:  08 eb 30 e4 03 88 ff ff a0 
e2 30 e4 03 88 ff ff .ë0ä..ÿÿ.â0ä..ÿÿ
[ 1179.776003]   Object 0xffff8803e430e288:  a0 e3 30 e4 03 88 ff ff be 
00 00 00 00 01 00 00 .ã0ä..ÿÿŸ.......
[ 1179.776974]   Object 0xffff8803e430e298:  be 00 00 00 be 00 00 00 5b 
20 20 20 20 32 2e 34 Ÿ...Ÿ...[....2.4
[ 1179.777656]   Object 0xffff8803e430e2a8:  31 30 32 34 36 5d 20 70 63 
69 20 30 30 30 30 3a 10246].pci.0000:
[ 1179.778609]   Object 0xffff8803e430e2b8:  30 30 3a 31 34 2e 34 3a 20 
20 20 62 72 69 64 67 00:14.4:...bridg
[ 1179.779289]   Object 0xffff8803e430e2c8:  65 20 77 69 6e 64 6f 77 20 
5b 69 6f 20 20 30 78 e.window.[io..0x
[ 1179.779976]   Object 0xffff8803e430e2d8:  30 64 30 30 2d 30 78 66 66 
66 66 5d 20 28 73 75 0d00-0xffff].(su
[ 1179.780327]   Object 0xffff8803e430e2e8:  62 74 72 61 63 74 69 76 65 
20 64 65 63 6f 64 65 btractive.decode
[ 1179.780327]   Object 0xffff8803e430e2f8:  29 0d 0a 5b 20 20 20 20 32 
2e 34 31 30 32 34 38 )..[....2.410248
[ 1179.780327]   Object 0xffff8803e430e308:  5d 20 70 63 69 20 30 30 30 
30 3a 30 30 3a 31 34 ].pci.0000:00:14
[ 1179.780327]   Object 0xffff8803e430e318:  2e 34 3a 20 20 20 62 72 69 
64 67 65 20 77 69 6e .4:...bridge.win
[ 1179.780327]   Object 0xffff8803e430e328:  64 6f 77 20 5b 6d 65 6d 20 
30 78 30 30 30 61 30 dow.[mem.0x000a0
[ 1179.780327]   Object 0xffff8803e430e338:  30 30 30 2d 30 78 30 30 30 
62 66 66 66 66 5d 20 000-0x000bffff].
[ 1179.780327]   Object 0xffff8803e430e348:  28 73 75 62 74 72 61 63 74 
69 76 65 20 64 65 63 (subtractive.dec
[ 1179.780327]   Object 0xffff8803e430e358:  6f 64 65 29 0d 0a 32 30 5d 
20 78 48 43 49 20 78 ode)..20].xHCI.x
[ 1179.780327]   Object 0xffff8803e430e368:  68 63 69 5f 63 68 65 63 6b 
5f 62 61 6e 64 77 69 hci_check_bandwi
[ 1179.780327]   Object 0xffff8803e430e378:  64 74 68 20 63 61 6c 6c 65 
64 20 66 6f 72 20 72 dth.called.for.r
[ 1179.780327]   Object 0xffff8803e430e388:  6f 6f 74 20 68 75 62 0d 0a 
0d 0a 0d 0a 0d 0a 0d oot.hub.........
[ 1179.780327]   Object 0xffff8803e430e398:  0a 30 30 0d 0a 0d 0a 0a 00 
00 00 00 00 00 00 00 .00.............
[ 1179.780327]   Object 0xffff8803e430e3a8:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[ 1179.780327]   Object 0xffff8803e430e3b8:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[ 1179.780327]   Object 0xffff8803e430e3c8:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[ 1179.780327]   Object 0xffff8803e430e3d8:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[ 1179.780327]   Object 0xffff8803e430e3e8:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[ 1179.780327]   Object 0xffff8803e430e3f8:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[ 1179.780327]   Object 0xffff8803e430e408:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[ 1179.780327]   Object 0xffff8803e430e418:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[ 1179.780327]   Object 0xffff8803e430e428:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[ 1179.780327]   Object 0xffff8803e430e438:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[ 1179.780327]   Object 0xffff8803e430e448:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[ 1179.780327]   Object 0xffff8803e430e458:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[ 1179.780327]   Object 0xffff8803e430e468:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[ 1179.780327]   Object 0xffff8803e430e478:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[ 1179.780327]   Object 0xffff8803e430e488:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[ 1179.780327]   Object 0xffff8803e430e498:  00 00 00 00 00 00 00 00 6b 
6b 6b 6b 6b 6b 6b 6b ........kkkkkkkk
[ 1179.780327]   Object 0xffff8803e430e4a8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1179.780327]   Object 0xffff8803e430e4b8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1179.780327]   Object 0xffff8803e430e4c8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1179.780327]   Object 0xffff8803e430e4d8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1179.780327]   Object 0xffff8803e430e4e8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1179.780327]   Object 0xffff8803e430e4f8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1179.780327]   Object 0xffff8803e430e508:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1179.780327]   Object 0xffff8803e430e518:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1179.780327]   Object 0xffff8803e430e528:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1179.780327]   Object 0xffff8803e430e538:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1179.780327]   Object 0xffff8803e430e548:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1179.780327]   Object 0xffff8803e430e558:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1179.780327]   Object 0xffff8803e430e568:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1179.780327]   Object 0xffff8803e430e578:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1179.780327]   Object 0xffff8803e430e588:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1179.780327]   Object 0xffff8803e430e598:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1179.780327]   Object 0xffff8803e430e5a8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1179.780327]   Object 0xffff8803e430e5b8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1179.780327]   Object 0xffff8803e430e5c8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1179.780327]   Object 0xffff8803e430e5d8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1179.780327]   Object 0xffff8803e430e5e8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1179.780327]   Object 0xffff8803e430e5f8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1179.780327]   Object 0xffff8803e430e608:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1179.780327]   Object 0xffff8803e430e618:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1179.780327]   Object 0xffff8803e430e628:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1179.780327]   Object 0xffff8803e430e638:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1179.780327]   Object 0xffff8803e430e648:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1179.780327]   Object 0xffff8803e430e658:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1179.780327]   Object 0xffff8803e430e668:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b a5 kkkkkkkkkkkkkkk¥
[ 1179.780327]  Redzone 0xffff8803e430e678:  cc cc cc cc cc cc cc cc 
                      ÌÌÌÌÌÌÌÌ
[ 1179.780327]  Padding 0xffff8803e430e6b8:  5a 5a 5a 5a 5a 5a 00 00 
                      ZZZZZZ..
[ 1179.780327] Pid: 2212, comm: kworker/5:2 Not tainted 3.0.0-rc2 #1
[ 1179.780327] Call Trace:
[ 1179.780327]  [<ffffffff810ba41d>] ? check_bytes_and_report+0x10d/0x150
[ 1179.780327]  [<ffffffff810ba509>] ? check_object+0xa9/0x260
[ 1179.780327]  [<ffffffff8125acf7>] ? tty_buffer_free_all+0x47/0x70
[ 1179.780327]  [<ffffffff810bb033>] ? free_debug_processing+0x133/0x250
[ 1179.780327]  [<ffffffff810bb27b>] ? __slab_free+0x12b/0x140
[ 1179.780327]  [<ffffffff81254c70>] ? free_tty_struct+0x30/0x30
[ 1179.780327]  [<ffffffff8125acf7>] ? tty_buffer_free_all+0x47/0x70
[ 1179.780327]  [<ffffffff81254c66>] ? free_tty_struct+0x26/0x30
[ 1179.780327]  [<ffffffff8104f28b>] ? process_one_work+0xfb/0x3b0
[ 1179.780327]  [<ffffffff8104f964>] ? worker_thread+0x144/0x3d0
[ 1179.780327]  [<ffffffff8102cc10>] ? __wake_up_common+0x50/0x80
[ 1179.780327]  [<ffffffff8104f820>] ? rescuer_thread+0x2e0/0x2e0
[ 1179.780327]  [<ffffffff8104f820>] ? rescuer_thread+0x2e0/0x2e0
[ 1179.780327]  [<ffffffff81053436>] ? kthread+0x96/0xa0
[ 1179.780327]  [<ffffffff813e1d14>] ? kernel_thread_helper+0x4/0x10
[ 1179.780327]  [<ffffffff810533a0>] ? kthread_worker_fn+0x120/0x120
[ 1179.780327]  [<ffffffff813e1d10>] ? gs_change+0xb/0xb
[ 1179.780327] FIX kmalloc-1024: Restoring 
0xffff8803e430e6be-0xffff8803e430e6bf=0x5a
[ 1179.780327]
[ 1180.826212] 
=============================================================================
[ 1180.826289] BUG kmalloc-1024: Object padding overwritten
[ 1180.826338] 
-----------------------------------------------------------------------------
[ 1180.826341]
[ 1180.826439] INFO: 0xffff8804097a9566-0xffff8804097a9567. First byte 
0x0 instead of 0x5a
[ 1180.826505] INFO: Allocated in tty_buffer_request_room+0xcc/0x1a0 
age=69291 cpu=0 pid=5978
[ 1180.826567] INFO: Freed in __blockdev_direct_IO+0xa16/0xa90 age=80371 
cpu=5 pid=5713
[ 1180.826626] INFO: Slab 0xffffea000e212cc0 objects=29 used=3 
fp=0xffff8804097a8890 flags=0x8000000000004081
[ 1180.826686] INFO: Object 0xffff8804097a9120 @offset=4384 
fp=0xffff8804097a8890
[ 1180.826689]
[ 1180.826781] Bytes b4 0xffff8804097a9110:  91 48 01 00 01 00 00 00 5a 
5a 5a 5a 5a 5a 5a 5a .H......ZZZZZZZZ
[ 1180.827478]   Object 0xffff8804097a9120:  30 de 70 fc 03 88 ff ff 48 
91 7a 09 04 88 ff ff 0Þpü..ÿÿH.z...ÿÿ
[ 1180.828179]   Object 0xffff8804097a9130:  48 92 7a 09 04 88 ff ff ce 
00 00 00 00 01 00 00 H.z...ÿÿÎ.......
[ 1180.829149]   Object 0xffff8804097a9140:  ce 00 00 00 ce 00 00 00 5b 
20 20 20 20 32 2e 33 Î...Î...[....2.3
[ 1180.830162]   Object 0xffff8804097a9150:  32 31 31 35 36 5d 20 70 63 
69 20 30 30 30 30 3a 21156].pci.0000:
[ 1180.831118]   Object 0xffff8804097a9160:  30 30 3a 31 33 2e 30 3a 20 
5b 31 30 30 32 3a 34 00:13.0:.[1002:4
[ 1180.831796]   Object 0xffff8804097a9170:  33 39 37 5d 20 74 79 70 65 
20 30 20 63 6c 61 73 397].type.0.clas
[ 1180.832492]   Object 0xffff8804097a9180:  73 20 30 78 30 30 30 63 30 
33 0d 0a 5b 20 20 20 s.0x000c03..[...
[ 1180.833482]   Object 0xffff8804097a9190:  20 32 2e 33 32 31 31 36 36 
5d 20 70 63 69 20 30 .2.321166].pci.0
[ 1180.834173]   Object 0xffff8804097a91a0:  30 30 30 3a 30 30 3a 31 33 
2e 30 3a 20 72 65 67 000:00:13.0:.reg
[ 1180.835134]   Object 0xffff8804097a91b0:  20 31 30 3a 20 5b 6d 65 6d 
20 30 78 66 65 31 66 .10:.[mem.0xfe1f
[ 1180.835240]   Object 0xffff8804097a91c0:  64 30 30 30 2d 30 78 66 65 
31 66 64 66 66 66 5d d000-0xfe1fdfff]
[ 1180.835240]   Object 0xffff8804097a91d0:  0d 0a 5b 20 20 20 20 32 2e 
33 32 31 32 31 38 5d ..[....2.321218]
[ 1180.835240]   Object 0xffff8804097a91e0:  20 70 63 69 20 30 30 30 30 
3a 30 30 3a 31 33 2e .pci.0000:00:13.
[ 1180.835240]   Object 0xffff8804097a91f0:  32 3a 20 5b 31 30 30 32 3a 
34 33 39 36 5d 20 74 2:.[1002:4396].t
[ 1180.835240]   Object 0xffff8804097a9200:  79 70 65 20 30 20 63 6c 61 
73 73 20 30 78 30 30 ype.0.class.0x00
[ 1180.835240]   Object 0xffff8804097a9210:  30 63 30 33 0d 0a 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b 0c03..kkkkkkkkkk
[ 1180.835240]   Object 0xffff8804097a9220:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.835240]   Object 0xffff8804097a9230:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.835240]   Object 0xffff8804097a9240:  6b 6b 6b 6b 6b 6b 6b 6b 00 
00 00 00 00 00 00 00 kkkkkkkk........
[ 1180.835240]   Object 0xffff8804097a9250:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[ 1180.835240]   Object 0xffff8804097a9260:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[ 1180.835240]   Object 0xffff8804097a9270:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[ 1180.835240]   Object 0xffff8804097a9280:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[ 1180.835240]   Object 0xffff8804097a9290:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[ 1180.835240]   Object 0xffff8804097a92a0:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[ 1180.835240]   Object 0xffff8804097a92b0:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[ 1180.835240]   Object 0xffff8804097a92c0:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[ 1180.835240]   Object 0xffff8804097a92d0:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[ 1180.835240]   Object 0xffff8804097a92e0:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[ 1180.835240]   Object 0xffff8804097a92f0:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[ 1180.835240]   Object 0xffff8804097a9300:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[ 1180.835240]   Object 0xffff8804097a9310:  00 00 00 00 00 00 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b ......kkkkkkkkkk
[ 1180.835240]   Object 0xffff8804097a9320:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.835240]   Object 0xffff8804097a9330:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.835240]   Object 0xffff8804097a9340:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.835240]   Object 0xffff8804097a9350:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.835240]   Object 0xffff8804097a9360:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.835240]   Object 0xffff8804097a9370:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.835240]   Object 0xffff8804097a9380:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.835240]   Object 0xffff8804097a9390:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.835240]   Object 0xffff8804097a93a0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.835240]   Object 0xffff8804097a93b0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.835240]   Object 0xffff8804097a93c0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.835240]   Object 0xffff8804097a93d0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.835240]   Object 0xffff8804097a93e0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.835240]   Object 0xffff8804097a93f0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.835240]   Object 0xffff8804097a9400:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.835240]   Object 0xffff8804097a9410:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.835240]   Object 0xffff8804097a9420:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.835240]   Object 0xffff8804097a9430:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.835240]   Object 0xffff8804097a9440:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.835240]   Object 0xffff8804097a9450:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.835240]   Object 0xffff8804097a9460:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.835240]   Object 0xffff8804097a9470:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.835240]   Object 0xffff8804097a9480:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.835240]   Object 0xffff8804097a9490:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.835240]   Object 0xffff8804097a94a0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.835240]   Object 0xffff8804097a94b0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.835240]   Object 0xffff8804097a94c0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.835240]   Object 0xffff8804097a94d0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.835240]   Object 0xffff8804097a94e0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.835240]   Object 0xffff8804097a94f0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.835240]   Object 0xffff8804097a9500:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.835240]   Object 0xffff8804097a9510:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b a5 kkkkkkkkkkkkkkk¥
[ 1180.835240]  Redzone 0xffff8804097a9520:  cc cc cc cc cc cc cc cc 
                      ÌÌÌÌÌÌÌÌ
[ 1180.835240]  Padding 0xffff8804097a9560:  5a 5a 5a 5a 5a 5a 00 00 
                      ZZZZZZ..
[ 1180.835240] Pid: 8295, comm: kworker/5:0 Not tainted 3.0.0-rc2 #1
[ 1180.835240] Call Trace:
[ 1180.835240]  [<ffffffff810ba41d>] ? check_bytes_and_report+0x10d/0x150
[ 1180.835240]  [<ffffffff810ba509>] ? check_object+0xa9/0x260
[ 1180.835240]  [<ffffffff8125acf7>] ? tty_buffer_free_all+0x47/0x70
[ 1180.835240]  [<ffffffff810bb033>] ? free_debug_processing+0x133/0x250
[ 1180.835240]  [<ffffffff810bb27b>] ? __slab_free+0x12b/0x140
[ 1180.835240]  [<ffffffff81254c70>] ? free_tty_struct+0x30/0x30
[ 1180.835240]  [<ffffffff8125acf7>] ? tty_buffer_free_all+0x47/0x70
[ 1180.835240]  [<ffffffff81254c66>] ? free_tty_struct+0x26/0x30
[ 1180.835240]  [<ffffffff8104f28b>] ? process_one_work+0xfb/0x3b0
[ 1180.835240]  [<ffffffff8104f964>] ? worker_thread+0x144/0x3d0
[ 1180.835240]  [<ffffffff8102cc10>] ? __wake_up_common+0x50/0x80
[ 1180.835240]  [<ffffffff8104f820>] ? rescuer_thread+0x2e0/0x2e0
[ 1180.835240]  [<ffffffff8104f820>] ? rescuer_thread+0x2e0/0x2e0
[ 1180.835240]  [<ffffffff81053436>] ? kthread+0x96/0xa0
[ 1180.835240]  [<ffffffff813e1d14>] ? kernel_thread_helper+0x4/0x10
[ 1180.835240]  [<ffffffff810533a0>] ? kthread_worker_fn+0x120/0x120
[ 1180.835240]  [<ffffffff813e1d10>] ? gs_change+0xb/0xb
[ 1180.835240] FIX kmalloc-1024: Restoring 
0xffff8804097a9566-0xffff8804097a9567=0x5a
[ 1180.835240]
[ 1180.881652] 
=============================================================================
[ 1180.881713] BUG kmalloc-1024: Object padding overwritten
[ 1180.881760] 
-----------------------------------------------------------------------------
[ 1180.881763]
[ 1180.881859] INFO: 0xffff8803fc70eb06-0xffff8803fc70eb07. First byte 
0x0 instead of 0x5a
[ 1180.881920] INFO: Allocated in tty_buffer_request_room+0xcc/0x1a0 
age=69297 cpu=0 pid=5978
[ 1180.881981] INFO: Freed in __blockdev_direct_IO+0xa16/0xa90 age=69384 
cpu=5 pid=5146
[ 1180.882040] INFO: Slab 0xffffea000df389c0 objects=29 used=3 
fp=0xffff8803fc70de30 flags=0x8000000000004081
[ 1180.882099] INFO: Object 0xffff8803fc70e6c0 @offset=26304 
fp=0xffff8803fc70de30
[ 1180.882102]
[ 1180.882192] Bytes b4 0xffff8803fc70e6b0:  aa 40 01 00 01 00 00 00 5a 
5a 5a 5a 5a 5a 5a 5a ª@......ZZZZZZZZ
[ 1180.882884]   Object 0xffff8803fc70e6c0:  68 95 6c fc 03 88 ff ff e8 
e6 70 fc 03 88 ff ff h.lü..ÿÿèæpü..ÿÿ
[ 1180.883583]   Object 0xffff8803fc70e6d0:  e8 e7 70 fc 03 88 ff ff 00 
01 00 00 00 01 00 00 èçpü..ÿÿ........
[ 1180.884542]   Object 0xffff8803fc70e6e0:  00 01 00 00 00 01 00 00 5b 
20 20 20 20 32 2e 33 ........[....2.3
[ 1180.885223]   Object 0xffff8803fc70e6f0:  32 30 38 31 38 5d 20 70 63 
69 20 30 30 30 30 3a 20818].pci.0000:
[ 1180.886169]   Object 0xffff8803fc70e700:  30 30 3a 30 34 2e 30 3a 20 
50 4d 45 23 20 73 75 00:04.0:.PME#.su
[ 1180.886850]   Object 0xffff8803fc70e710:  70 70 6f 72 74 65 64 20 66 
72 6f 6d 20 44 30 20 pported.from.D0.
[ 1180.887535]   Object 0xffff8803fc70e720:  44 33 68 6f 74 20 44 33 63 
6f 6c 64 0d 0a 5b 20 D3hot.D3cold..[.
[ 1180.888848]   Object 0xffff8803fc70e730:  20 20 20 32 2e 33 32 30 38 
32 30 5d 20 70 63 69 ...2.320820].pci
[ 1180.889537]   Object 0xffff8803fc70e740:  20 30 30 30 30 3a 30 30 3a 
30 34 2e 30 3a 20 50 .0000:00:04.0:.P
[ 1180.890481]   Object 0xffff8803fc70e750:  4d 45 23 20 64 69 73 61 62 
6c 65 64 0d 0a 5b 20 ME#.disabled..[.
[ 1180.891161]   Object 0xffff8803fc70e760:  20 20 20 32 2e 33 32 30 38 
33 36 5d 20 70 63 69 ...2.320836].pci
[ 1180.891365]   Object 0xffff8803fc70e770:  20 30 30 30 30 3a 30 30 3a 
30 39 2e 30 3a 20 5b .0000:00:09.0:.[
[ 1180.891365]   Object 0xffff8803fc70e780:  31 30 32 32 3a 39 36 30 38 
5d 20 74 79 70 65 20 1022:9608].type.
[ 1180.891365]   Object 0xffff8803fc70e790:  31 20 63 6c 61 73 73 20 30 
78 30 30 30 36 30 34 1.class.0x000604
[ 1180.891365]   Object 0xffff8803fc70e7a0:  0d 0a 5b 20 20 20 20 32 2e 
33 32 30 38 35 35 5d ..[....2.320855]
[ 1180.891365]   Object 0xffff8803fc70e7b0:  20 70 63 69 20 30 30 30 30 
3a 30 30 3a 30 39 2e .pci.0000:00:09.
[ 1180.891365]   Object 0xffff8803fc70e7c0:  30 3a 20 50 4d 45 23 20 73 
75 70 70 6f 72 74 65 0:.PME#.supporte
[ 1180.891365]   Object 0xffff8803fc70e7d0:  64 20 66 72 6f 6d 20 44 30 
20 44 33 68 6f 74 20 d.from.D0.D3hot.
[ 1180.891365]   Object 0xffff8803fc70e7e0:  44 33 63 6f 6c 64 0d 0a 00 
00 00 00 00 00 00 00 D3cold..........
[ 1180.891365]   Object 0xffff8803fc70e7f0:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[ 1180.891365]   Object 0xffff8803fc70e800:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[ 1180.891365]   Object 0xffff8803fc70e810:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[ 1180.891365]   Object 0xffff8803fc70e820:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[ 1180.891365]   Object 0xffff8803fc70e830:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[ 1180.891365]   Object 0xffff8803fc70e840:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[ 1180.891365]   Object 0xffff8803fc70e850:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[ 1180.891365]   Object 0xffff8803fc70e860:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[ 1180.891365]   Object 0xffff8803fc70e870:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[ 1180.891365]   Object 0xffff8803fc70e880:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[ 1180.891365]   Object 0xffff8803fc70e890:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[ 1180.891365]   Object 0xffff8803fc70e8a0:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[ 1180.891365]   Object 0xffff8803fc70e8b0:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[ 1180.891365]   Object 0xffff8803fc70e8c0:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[ 1180.891365]   Object 0xffff8803fc70e8d0:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[ 1180.891365]   Object 0xffff8803fc70e8e0:  00 00 00 00 00 00 00 00 6b 
6b 6b 6b 6b 6b 6b 6b ........kkkkkkkk
[ 1180.891365]   Object 0xffff8803fc70e8f0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.891365]   Object 0xffff8803fc70e900:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.891365]   Object 0xffff8803fc70e910:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.891365]   Object 0xffff8803fc70e920:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.891365]   Object 0xffff8803fc70e930:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.891365]   Object 0xffff8803fc70e940:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.891365]   Object 0xffff8803fc70e950:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.891365]   Object 0xffff8803fc70e960:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.891365]   Object 0xffff8803fc70e970:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.891365]   Object 0xffff8803fc70e980:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.891365]   Object 0xffff8803fc70e990:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.891365]   Object 0xffff8803fc70e9a0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.891365]   Object 0xffff8803fc70e9b0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.891365]   Object 0xffff8803fc70e9c0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.891365]   Object 0xffff8803fc70e9d0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.891365]   Object 0xffff8803fc70e9e0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.891365]   Object 0xffff8803fc70e9f0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.891365]   Object 0xffff8803fc70ea00:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.891365]   Object 0xffff8803fc70ea10:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.891365]   Object 0xffff8803fc70ea20:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.891365]   Object 0xffff8803fc70ea30:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.891365]   Object 0xffff8803fc70ea40:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.891365]   Object 0xffff8803fc70ea50:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.891365]   Object 0xffff8803fc70ea60:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.891365]   Object 0xffff8803fc70ea70:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.891365]   Object 0xffff8803fc70ea80:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.891365]   Object 0xffff8803fc70ea90:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.891365]   Object 0xffff8803fc70eaa0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.891365]   Object 0xffff8803fc70eab0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b a5 kkkkkkkkkkkkkkk¥
[ 1180.891365]  Redzone 0xffff8803fc70eac0:  cc cc cc cc cc cc cc cc 
                      ÌÌÌÌÌÌÌÌ
[ 1180.891365]  Padding 0xffff8803fc70eb00:  5a 5a 5a 5a 5a 5a 00 00 
                      ZZZZZZ..
[ 1180.891365] Pid: 8295, comm: kworker/5:0 Not tainted 3.0.0-rc2 #1
[ 1180.891365] Call Trace:
[ 1180.891365]  [<ffffffff810ba41d>] ? check_bytes_and_report+0x10d/0x150
[ 1180.891365]  [<ffffffff810ba509>] ? check_object+0xa9/0x260
[ 1180.891365]  [<ffffffff8125acf7>] ? tty_buffer_free_all+0x47/0x70
[ 1180.891365]  [<ffffffff810bb033>] ? free_debug_processing+0x133/0x250
[ 1180.891365]  [<ffffffff810bb27b>] ? __slab_free+0x12b/0x140
[ 1180.891365]  [<ffffffff81254c70>] ? free_tty_struct+0x30/0x30
[ 1180.891365]  [<ffffffff8125acf7>] ? tty_buffer_free_all+0x47/0x70
[ 1180.891365]  [<ffffffff81254c66>] ? free_tty_struct+0x26/0x30
[ 1180.891365]  [<ffffffff8104f28b>] ? process_one_work+0xfb/0x3b0
[ 1180.891365]  [<ffffffff8104f964>] ? worker_thread+0x144/0x3d0
[ 1180.891365]  [<ffffffff8102cc10>] ? __wake_up_common+0x50/0x80
[ 1180.891365]  [<ffffffff8104f820>] ? rescuer_thread+0x2e0/0x2e0
[ 1180.891365]  [<ffffffff8104f820>] ? rescuer_thread+0x2e0/0x2e0
[ 1180.891365]  [<ffffffff81053436>] ? kthread+0x96/0xa0
[ 1180.891365]  [<ffffffff813e1d14>] ? kernel_thread_helper+0x4/0x10
[ 1180.891365]  [<ffffffff810533a0>] ? kthread_worker_fn+0x120/0x120
[ 1180.891365]  [<ffffffff813e1d10>] ? gs_change+0xb/0xb
[ 1180.891365] FIX kmalloc-1024: Restoring 
0xffff8803fc70eb06-0xffff8803fc70eb07=0x5a
[ 1180.891365]

^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
  2011-06-07 14:40                                                 ` Brad Campbell
@ 2011-06-07 15:35                                                   ` Patrick McHardy
  -1 siblings, 0 replies; 111+ messages in thread
From: Patrick McHardy @ 2011-06-07 15:35 UTC (permalink / raw)
  To: Brad Campbell
  Cc: Bart De Schuymer, kvm, linux-mm, linux-kernel, netdev, netfilter-devel

On 07.06.2011 16:40, Brad Campbell wrote:
> On 07/06/11 21:30, Patrick McHardy wrote:
>> On 07.06.2011 05:33, Brad Campbell wrote:
>>> On 07/06/11 04:10, Bart De Schuymer wrote:
>>>> Hi Brad,
>>>>
>>>> This has probably nothing to do with ebtables, so please rmmod in case
>>>> it's loaded.
>>>> A few questions I didn't directly see an answer to in the threads I
>>>> scanned...
>>>> I'm assuming you actually use the bridging firewall functionality. So,
>>>> what iptables modules do you use? Can you reduce your iptables rules to
>>>> a core that triggers the bug?
>>>> Or does it get triggered even with an empty set of firewall rules?
>>>> Are you using a stock .35 kernel or is it patched?
>>>> Is this something I can trigger on a poor guy's laptop or does it
>>>> require specialized hardware (I'm catching up on qemu/kvm...)?
>>>
>>> Not specialised hardware as such, I've just not been able to reproduce
>>> it outside of this specific operating scenario.
>>
>> The last similar problem we've had was related to the 32/64 bit compat
>> code. Are you running 32 bit userspace on a 64 bit kernel?
> 
> No, 32 bit Guest OS, but a completely 64 bit userspace on a 64 bit kernel.
> 
> Userspace is current Debian Stable. Kernel is Vanilla and qemu-kvm is
> current git
> 
> 
>>> I can't trigger it with empty firewall rules as it relies on a DNAT to
>>> occur. If I try it directly to the internal IP address (as I have to
>>> without netfilter loaded) then of course nothing fails.
>>>
>>> It's a pain in the bum as a fault, but it's one I can easily reproduce
>>> as long as I use the same set of circumstances.
>>>
>>> I'll try using 3.0-rc2 (current git) tonight, and if I can reproduce it
>>> on that then I'll attempt to pare down the IPTABLES rules to a bare
>>> minimum.
>>>
>>> It is nothing to do with ebtables as I don't compile it. I'm not really
>>> sure about "bridging firewall" functionality. I just use a couple of
>>> hand coded bash scripts to set the tables up.
>>
>>  From one of your previous mails:
>>
>>> # CONFIG_BRIDGE_NF_EBTABLES is not set
>>
>> How about CONFIG_BRIDGE_NETFILTER?
>>
> 
> It was compiled in.
> 
> With the following table set I was able to reproduce the problem on
> 3.0-rc2. Replaced my IP with xxx.xxx.xxx.xxx, but otherwise unmodified

Which kernel was the last version without this problem?

> root@srv:~# iptables-save
> # Generated by iptables-save v1.4.10 on Tue Jun  7 22:11:30 2011
> *filter
> :INPUT ACCEPT [978:107619]
> :FORWARD ACCEPT [142:7068]
> :OUTPUT ACCEPT [1659:291870]
> -A INPUT -i ppp0 -m state --state RELATED,ESTABLISHED -j ACCEPT
> -A INPUT ! -i ppp0 -m state --state NEW -j ACCEPT
> -A INPUT -i ppp0 -j DROP
> COMMIT
> # Completed on Tue Jun  7 22:11:30 2011
> # Generated by iptables-save v1.4.10 on Tue Jun  7 22:11:30 2011
> *nat
> :PREROUTING ACCEPT [813:49170]
> :INPUT ACCEPT [91:7090]
> :OUTPUT ACCEPT [267:20731]
> :POSTROUTING ACCEPT [296:22281]
> -A PREROUTING -d xxx.xxx.xxx.xxx/32 ! -i ppp0 -p tcp -m tcp --dport 443
> -j DNAT --to-destination 192.168.253.198
> COMMIT
> # Completed on Tue Jun  7 22:11:30 2011
> # Generated by iptables-save v1.4.10 on Tue Jun  7 22:11:30 2011
> *mangle
> :PREROUTING ACCEPT [2729:274392]
> :INPUT ACCEPT [2508:262976]
> :FORWARD ACCEPT [142:7068]
> :OUTPUT ACCEPT [1674:293701]
> :POSTROUTING ACCEPT [2131:346411]
> -A FORWARD -o ppp0 -p tcp -m tcp --tcp-flags SYN,RST SYN -m tcpmss --mss
> 1400:1536 -j TCPMSS --clamp-mss-to-pmtu
> COMMIT
> # Completed on Tue Jun  7 22:11:30 2011

The main suspects would be NAT and TCPMSS. Did you also try whether
the crash occurs with only one of these these rules?

> I've just compiled out CONFIG_BRIDGE_NETFILTER and can no longer access
> the address the way I was doing it, so that's a no-go for me.

That's really weird since you're apparently not using any bridge
netfilter features. It shouldn't have any effect besides changing
at which point ip_tables is invoked. How are your network devices
configured (specifically any bridges)?

^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
@ 2011-06-07 15:35                                                   ` Patrick McHardy
  0 siblings, 0 replies; 111+ messages in thread
From: Patrick McHardy @ 2011-06-07 15:35 UTC (permalink / raw)
  To: Brad Campbell
  Cc: Bart De Schuymer, kvm, linux-mm, linux-kernel, netdev, netfilter-devel

On 07.06.2011 16:40, Brad Campbell wrote:
> On 07/06/11 21:30, Patrick McHardy wrote:
>> On 07.06.2011 05:33, Brad Campbell wrote:
>>> On 07/06/11 04:10, Bart De Schuymer wrote:
>>>> Hi Brad,
>>>>
>>>> This has probably nothing to do with ebtables, so please rmmod in case
>>>> it's loaded.
>>>> A few questions I didn't directly see an answer to in the threads I
>>>> scanned...
>>>> I'm assuming you actually use the bridging firewall functionality. So,
>>>> what iptables modules do you use? Can you reduce your iptables rules to
>>>> a core that triggers the bug?
>>>> Or does it get triggered even with an empty set of firewall rules?
>>>> Are you using a stock .35 kernel or is it patched?
>>>> Is this something I can trigger on a poor guy's laptop or does it
>>>> require specialized hardware (I'm catching up on qemu/kvm...)?
>>>
>>> Not specialised hardware as such, I've just not been able to reproduce
>>> it outside of this specific operating scenario.
>>
>> The last similar problem we've had was related to the 32/64 bit compat
>> code. Are you running 32 bit userspace on a 64 bit kernel?
> 
> No, 32 bit Guest OS, but a completely 64 bit userspace on a 64 bit kernel.
> 
> Userspace is current Debian Stable. Kernel is Vanilla and qemu-kvm is
> current git
> 
> 
>>> I can't trigger it with empty firewall rules as it relies on a DNAT to
>>> occur. If I try it directly to the internal IP address (as I have to
>>> without netfilter loaded) then of course nothing fails.
>>>
>>> It's a pain in the bum as a fault, but it's one I can easily reproduce
>>> as long as I use the same set of circumstances.
>>>
>>> I'll try using 3.0-rc2 (current git) tonight, and if I can reproduce it
>>> on that then I'll attempt to pare down the IPTABLES rules to a bare
>>> minimum.
>>>
>>> It is nothing to do with ebtables as I don't compile it. I'm not really
>>> sure about "bridging firewall" functionality. I just use a couple of
>>> hand coded bash scripts to set the tables up.
>>
>>  From one of your previous mails:
>>
>>> # CONFIG_BRIDGE_NF_EBTABLES is not set
>>
>> How about CONFIG_BRIDGE_NETFILTER?
>>
> 
> It was compiled in.
> 
> With the following table set I was able to reproduce the problem on
> 3.0-rc2. Replaced my IP with xxx.xxx.xxx.xxx, but otherwise unmodified

Which kernel was the last version without this problem?

> root@srv:~# iptables-save
> # Generated by iptables-save v1.4.10 on Tue Jun  7 22:11:30 2011
> *filter
> :INPUT ACCEPT [978:107619]
> :FORWARD ACCEPT [142:7068]
> :OUTPUT ACCEPT [1659:291870]
> -A INPUT -i ppp0 -m state --state RELATED,ESTABLISHED -j ACCEPT
> -A INPUT ! -i ppp0 -m state --state NEW -j ACCEPT
> -A INPUT -i ppp0 -j DROP
> COMMIT
> # Completed on Tue Jun  7 22:11:30 2011
> # Generated by iptables-save v1.4.10 on Tue Jun  7 22:11:30 2011
> *nat
> :PREROUTING ACCEPT [813:49170]
> :INPUT ACCEPT [91:7090]
> :OUTPUT ACCEPT [267:20731]
> :POSTROUTING ACCEPT [296:22281]
> -A PREROUTING -d xxx.xxx.xxx.xxx/32 ! -i ppp0 -p tcp -m tcp --dport 443
> -j DNAT --to-destination 192.168.253.198
> COMMIT
> # Completed on Tue Jun  7 22:11:30 2011
> # Generated by iptables-save v1.4.10 on Tue Jun  7 22:11:30 2011
> *mangle
> :PREROUTING ACCEPT [2729:274392]
> :INPUT ACCEPT [2508:262976]
> :FORWARD ACCEPT [142:7068]
> :OUTPUT ACCEPT [1674:293701]
> :POSTROUTING ACCEPT [2131:346411]
> -A FORWARD -o ppp0 -p tcp -m tcp --tcp-flags SYN,RST SYN -m tcpmss --mss
> 1400:1536 -j TCPMSS --clamp-mss-to-pmtu
> COMMIT
> # Completed on Tue Jun  7 22:11:30 2011

The main suspects would be NAT and TCPMSS. Did you also try whether
the crash occurs with only one of these these rules?

> I've just compiled out CONFIG_BRIDGE_NETFILTER and can no longer access
> the address the way I was doing it, so that's a no-go for me.

That's really weird since you're apparently not using any bridge
netfilter features. It shouldn't have any effect besides changing
at which point ip_tables is invoked. How are your network devices
configured (specifically any bridges)?

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
  2011-06-07 14:40                                                 ` Brad Campbell
@ 2011-06-07 18:04                                                   ` Bart De Schuymer
  -1 siblings, 0 replies; 111+ messages in thread
From: Bart De Schuymer @ 2011-06-07 18:04 UTC (permalink / raw)
  To: Brad Campbell
  Cc: Patrick McHardy, kvm, linux-mm, linux-kernel, netdev, netfilter-devel

Op 7/06/2011 16:40, Brad Campbell schreef:
> On 07/06/11 21:30, Patrick McHardy wrote:
>> On 07.06.2011 05:33, Brad Campbell wrote:
>>> On 07/06/11 04:10, Bart De Schuymer wrote:
>>>> Hi Brad,
>>>>
>>>> This has probably nothing to do with ebtables, so please rmmod in case
>>>> it's loaded.
>>>> A few questions I didn't directly see an answer to in the threads I
>>>> scanned...
>>>> I'm assuming you actually use the bridging firewall functionality. So,
>>>> what iptables modules do you use? Can you reduce your iptables 
>>>> rules to
>>>> a core that triggers the bug?
>>>> Or does it get triggered even with an empty set of firewall rules?
>>>> Are you using a stock .35 kernel or is it patched?
>>>> Is this something I can trigger on a poor guy's laptop or does it
>>>> require specialized hardware (I'm catching up on qemu/kvm...)?
>>>
>>> Not specialised hardware as such, I've just not been able to reproduce
>>> it outside of this specific operating scenario.
>>
>> The last similar problem we've had was related to the 32/64 bit compat
>> code. Are you running 32 bit userspace on a 64 bit kernel?
>
> No, 32 bit Guest OS, but a completely 64 bit userspace on a 64 bit 
> kernel.
>
> Userspace is current Debian Stable. Kernel is Vanilla and qemu-kvm is 
> current git
>
If the bug is easily triggered with your guest os, then you could try to 
capture the traffic with wireshark (or something else) in a 
configuration that doesn't crash your system. Save the traffic in a pcap 
file. Then you can see if resending that traffic in the vulnerable 
configuration triggers the bug (I don't know if something in Windows 
exists, but tcpreplay should work for Linux). Once you have such a 
capture , chances are the bug is even easily reproducible by us (unless 
it's hardware-specific). Success isn't guaranteed, but I think it's 
worth a shot...

cheers,
Bart


-- 
Bart De Schuymer
www.artinalgorithms.be


^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
@ 2011-06-07 18:04                                                   ` Bart De Schuymer
  0 siblings, 0 replies; 111+ messages in thread
From: Bart De Schuymer @ 2011-06-07 18:04 UTC (permalink / raw)
  To: Brad Campbell
  Cc: Patrick McHardy, kvm, linux-mm, linux-kernel, netdev, netfilter-devel

Op 7/06/2011 16:40, Brad Campbell schreef:
> On 07/06/11 21:30, Patrick McHardy wrote:
>> On 07.06.2011 05:33, Brad Campbell wrote:
>>> On 07/06/11 04:10, Bart De Schuymer wrote:
>>>> Hi Brad,
>>>>
>>>> This has probably nothing to do with ebtables, so please rmmod in case
>>>> it's loaded.
>>>> A few questions I didn't directly see an answer to in the threads I
>>>> scanned...
>>>> I'm assuming you actually use the bridging firewall functionality. So,
>>>> what iptables modules do you use? Can you reduce your iptables 
>>>> rules to
>>>> a core that triggers the bug?
>>>> Or does it get triggered even with an empty set of firewall rules?
>>>> Are you using a stock .35 kernel or is it patched?
>>>> Is this something I can trigger on a poor guy's laptop or does it
>>>> require specialized hardware (I'm catching up on qemu/kvm...)?
>>>
>>> Not specialised hardware as such, I've just not been able to reproduce
>>> it outside of this specific operating scenario.
>>
>> The last similar problem we've had was related to the 32/64 bit compat
>> code. Are you running 32 bit userspace on a 64 bit kernel?
>
> No, 32 bit Guest OS, but a completely 64 bit userspace on a 64 bit 
> kernel.
>
> Userspace is current Debian Stable. Kernel is Vanilla and qemu-kvm is 
> current git
>
If the bug is easily triggered with your guest os, then you could try to 
capture the traffic with wireshark (or something else) in a 
configuration that doesn't crash your system. Save the traffic in a pcap 
file. Then you can see if resending that traffic in the vulnerable 
configuration triggers the bug (I don't know if something in Windows 
exists, but tcpreplay should work for Linux). Once you have such a 
capture , chances are the bug is even easily reproducible by us (unless 
it's hardware-specific). Success isn't guaranteed, but I think it's 
worth a shot...

cheers,
Bart


-- 
Bart De Schuymer
www.artinalgorithms.be

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
  2011-06-07 15:35                                                   ` Patrick McHardy
  (?)
@ 2011-06-07 18:31                                                     ` Eric Dumazet
  -1 siblings, 0 replies; 111+ messages in thread
From: Eric Dumazet @ 2011-06-07 18:31 UTC (permalink / raw)
  To: Patrick McHardy
  Cc: Brad Campbell, Bart De Schuymer, kvm, linux-mm, linux-kernel,
	netdev, netfilter-devel

Le mardi 07 juin 2011 à 17:35 +0200, Patrick McHardy a écrit :

> The main suspects would be NAT and TCPMSS. Did you also try whether
> the crash occurs with only one of these these rules?
> 
> > I've just compiled out CONFIG_BRIDGE_NETFILTER and can no longer access
> > the address the way I was doing it, so that's a no-go for me.
> 
> That's really weird since you're apparently not using any bridge
> netfilter features. It shouldn't have any effect besides changing
> at which point ip_tables is invoked. How are your network devices
> configured (specifically any bridges)?

Something in the kernel does 

u16 *ptr = addr (given by kmalloc())

ptr[-1] = 0;

Could be an off-one error in a memmove()/memcopy() or loop...

I cant see a network issue here.

I checked arch/x86/lib/memmove_64.S and it seems fine.




^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
@ 2011-06-07 18:31                                                     ` Eric Dumazet
  0 siblings, 0 replies; 111+ messages in thread
From: Eric Dumazet @ 2011-06-07 18:31 UTC (permalink / raw)
  To: Patrick McHardy
  Cc: Brad Campbell, Bart De Schuymer, kvm, linux-mm, linux-kernel,
	netdev, netfilter-devel

Le mardi 07 juin 2011 à 17:35 +0200, Patrick McHardy a écrit :

> The main suspects would be NAT and TCPMSS. Did you also try whether
> the crash occurs with only one of these these rules?
> 
> > I've just compiled out CONFIG_BRIDGE_NETFILTER and can no longer access
> > the address the way I was doing it, so that's a no-go for me.
> 
> That's really weird since you're apparently not using any bridge
> netfilter features. It shouldn't have any effect besides changing
> at which point ip_tables is invoked. How are your network devices
> configured (specifically any bridges)?

Something in the kernel does 

u16 *ptr = addr (given by kmalloc())

ptr[-1] = 0;

Could be an off-one error in a memmove()/memcopy() or loop...

I cant see a network issue here.

I checked arch/x86/lib/memmove_64.S and it seems fine.



--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
@ 2011-06-07 18:31                                                     ` Eric Dumazet
  0 siblings, 0 replies; 111+ messages in thread
From: Eric Dumazet @ 2011-06-07 18:31 UTC (permalink / raw)
  To: Patrick McHardy
  Cc: Brad Campbell, Bart De Schuymer, kvm, linux-mm, linux-kernel,
	netdev, netfilter-devel

Le mardi 07 juin 2011 A  17:35 +0200, Patrick McHardy a A(C)crit :

> The main suspects would be NAT and TCPMSS. Did you also try whether
> the crash occurs with only one of these these rules?
> 
> > I've just compiled out CONFIG_BRIDGE_NETFILTER and can no longer access
> > the address the way I was doing it, so that's a no-go for me.
> 
> That's really weird since you're apparently not using any bridge
> netfilter features. It shouldn't have any effect besides changing
> at which point ip_tables is invoked. How are your network devices
> configured (specifically any bridges)?

Something in the kernel does 

u16 *ptr = addr (given by kmalloc())

ptr[-1] = 0;

Could be an off-one error in a memmove()/memcopy() or loop...

I cant see a network issue here.

I checked arch/x86/lib/memmove_64.S and it seems fine.



--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
  2011-06-07 18:31                                                     ` Eric Dumazet
  (?)
@ 2011-06-07 22:57                                                       ` Patrick McHardy
  -1 siblings, 0 replies; 111+ messages in thread
From: Patrick McHardy @ 2011-06-07 22:57 UTC (permalink / raw)
  To: Eric Dumazet
  Cc: Brad Campbell, Bart De Schuymer, kvm, linux-mm, linux-kernel,
	netdev, netfilter-devel

On 07.06.2011 20:31, Eric Dumazet wrote:
> Le mardi 07 juin 2011 à 17:35 +0200, Patrick McHardy a écrit :
> 
>> The main suspects would be NAT and TCPMSS. Did you also try whether
>> the crash occurs with only one of these these rules?
>>
>>> I've just compiled out CONFIG_BRIDGE_NETFILTER and can no longer access
>>> the address the way I was doing it, so that's a no-go for me.
>>
>> That's really weird since you're apparently not using any bridge
>> netfilter features. It shouldn't have any effect besides changing
>> at which point ip_tables is invoked. How are your network devices
>> configured (specifically any bridges)?
> 
> Something in the kernel does 
> 
> u16 *ptr = addr (given by kmalloc())
> 
> ptr[-1] = 0;
> 
> Could be an off-one error in a memmove()/memcopy() or loop...
> 
> I cant see a network issue here.

So far me neither, but netfilter appears to trigger the bug.

> I checked arch/x86/lib/memmove_64.S and it seems fine.

I was thinking it might be a missing skb_make_writable() combined
with vhost_net specifics in the netfilter code (TCPMSS and NAT are
both suspect), but was unable to find something. I also went
through the dst_metrics() conversion to see whether anything could
cause problems with the bridge fake_rttable, but also nothing
so far.

^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
@ 2011-06-07 22:57                                                       ` Patrick McHardy
  0 siblings, 0 replies; 111+ messages in thread
From: Patrick McHardy @ 2011-06-07 22:57 UTC (permalink / raw)
  To: Eric Dumazet
  Cc: Brad Campbell, Bart De Schuymer, kvm, linux-mm, linux-kernel,
	netdev, netfilter-devel

On 07.06.2011 20:31, Eric Dumazet wrote:
> Le mardi 07 juin 2011 à 17:35 +0200, Patrick McHardy a écrit :
> 
>> The main suspects would be NAT and TCPMSS. Did you also try whether
>> the crash occurs with only one of these these rules?
>>
>>> I've just compiled out CONFIG_BRIDGE_NETFILTER and can no longer access
>>> the address the way I was doing it, so that's a no-go for me.
>>
>> That's really weird since you're apparently not using any bridge
>> netfilter features. It shouldn't have any effect besides changing
>> at which point ip_tables is invoked. How are your network devices
>> configured (specifically any bridges)?
> 
> Something in the kernel does 
> 
> u16 *ptr = addr (given by kmalloc())
> 
> ptr[-1] = 0;
> 
> Could be an off-one error in a memmove()/memcopy() or loop...
> 
> I cant see a network issue here.

So far me neither, but netfilter appears to trigger the bug.

> I checked arch/x86/lib/memmove_64.S and it seems fine.

I was thinking it might be a missing skb_make_writable() combined
with vhost_net specifics in the netfilter code (TCPMSS and NAT are
both suspect), but was unable to find something. I also went
through the dst_metrics() conversion to see whether anything could
cause problems with the bridge fake_rttable, but also nothing
so far.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
@ 2011-06-07 22:57                                                       ` Patrick McHardy
  0 siblings, 0 replies; 111+ messages in thread
From: Patrick McHardy @ 2011-06-07 22:57 UTC (permalink / raw)
  To: Eric Dumazet
  Cc: Brad Campbell, Bart De Schuymer, kvm, linux-mm, linux-kernel,
	netdev, netfilter-devel

On 07.06.2011 20:31, Eric Dumazet wrote:
> Le mardi 07 juin 2011 a 17:35 +0200, Patrick McHardy a ecrit :
> 
>> The main suspects would be NAT and TCPMSS. Did you also try whether
>> the crash occurs with only one of these these rules?
>>
>>> I've just compiled out CONFIG_BRIDGE_NETFILTER and can no longer access
>>> the address the way I was doing it, so that's a no-go for me.
>>
>> That's really weird since you're apparently not using any bridge
>> netfilter features. It shouldn't have any effect besides changing
>> at which point ip_tables is invoked. How are your network devices
>> configured (specifically any bridges)?
> 
> Something in the kernel does 
> 
> u16 *ptr = addr (given by kmalloc())
> 
> ptr[-1] = 0;
> 
> Could be an off-one error in a memmove()/memcopy() or loop...
> 
> I cant see a network issue here.

So far me neither, but netfilter appears to trigger the bug.

> I checked arch/x86/lib/memmove_64.S and it seems fine.

I was thinking it might be a missing skb_make_writable() combined
with vhost_net specifics in the netfilter code (TCPMSS and NAT are
both suspect), but was unable to find something. I also went
through the dst_metrics() conversion to see whether anything could
cause problems with the bridge fake_rttable, but also nothing
so far.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
  2011-06-07 15:35                                                   ` Patrick McHardy
@ 2011-06-07 23:43                                                     ` Brad Campbell
  -1 siblings, 0 replies; 111+ messages in thread
From: Brad Campbell @ 2011-06-07 23:43 UTC (permalink / raw)
  To: Patrick McHardy
  Cc: Bart De Schuymer, kvm, linux-mm, linux-kernel, netdev, netfilter-devel

On 07/06/11 23:35, Patrick McHardy wrote:

> The main suspects would be NAT and TCPMSS. Did you also try whether
> the crash occurs with only one of these these rules?

To be honest I'm actually having trouble finding where TCPMSS is 
actually set in that ruleset. This is a production machine so I can only 
take it down after about 9PM at night. I'll have another crack at it 
tonight.

>> I've just compiled out CONFIG_BRIDGE_NETFILTER and can no longer access
>> the address the way I was doing it, so that's a no-go for me.
>
> That's really weird since you're apparently not using any bridge
> netfilter features. It shouldn't have any effect besides changing
> at which point ip_tables is invoked. How are your network devices
> configured (specifically any bridges)?
>

I have one bridge with all my virtual machines on it.

In this particular instance the packets leave VM A destined for the IP 
address of ppp0 (the external interface). This is intercepted by the 
DNAT PREROUTING rule above and shunted back to VM B.

The VM's are on br1 and the external address is ppp0. Without 
CONFIG_BRIDGE_NETFILTER compiled in I can see the traffic entering and 
leaving VM B with tcpdump, but the packets never seem to get back to VM A.

VM A is XP 32 bit, VM B is Linux. I have some other Linux VM's, so I'll 
do some more testing tonight between those to see where the packets are 
going without CONFIG_BRIDGE_NETFILTER set.

^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
@ 2011-06-07 23:43                                                     ` Brad Campbell
  0 siblings, 0 replies; 111+ messages in thread
From: Brad Campbell @ 2011-06-07 23:43 UTC (permalink / raw)
  To: Patrick McHardy
  Cc: Bart De Schuymer, kvm, linux-mm, linux-kernel, netdev, netfilter-devel

On 07/06/11 23:35, Patrick McHardy wrote:

> The main suspects would be NAT and TCPMSS. Did you also try whether
> the crash occurs with only one of these these rules?

To be honest I'm actually having trouble finding where TCPMSS is 
actually set in that ruleset. This is a production machine so I can only 
take it down after about 9PM at night. I'll have another crack at it 
tonight.

>> I've just compiled out CONFIG_BRIDGE_NETFILTER and can no longer access
>> the address the way I was doing it, so that's a no-go for me.
>
> That's really weird since you're apparently not using any bridge
> netfilter features. It shouldn't have any effect besides changing
> at which point ip_tables is invoked. How are your network devices
> configured (specifically any bridges)?
>

I have one bridge with all my virtual machines on it.

In this particular instance the packets leave VM A destined for the IP 
address of ppp0 (the external interface). This is intercepted by the 
DNAT PREROUTING rule above and shunted back to VM B.

The VM's are on br1 and the external address is ppp0. Without 
CONFIG_BRIDGE_NETFILTER compiled in I can see the traffic entering and 
leaving VM B with tcpdump, but the packets never seem to get back to VM A.

VM A is XP 32 bit, VM B is Linux. I have some other Linux VM's, so I'll 
do some more testing tonight between those to see where the packets are 
going without CONFIG_BRIDGE_NETFILTER set.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
  2011-06-07 18:04                                                   ` Bart De Schuymer
@ 2011-06-08  0:15                                                     ` Brad Campbell
  -1 siblings, 0 replies; 111+ messages in thread
From: Brad Campbell @ 2011-06-08  0:15 UTC (permalink / raw)
  To: Bart De Schuymer
  Cc: Patrick McHardy, kvm, linux-mm, linux-kernel, netdev, netfilter-devel

On 08/06/11 02:04, Bart De Schuymer wrote:

> If the bug is easily triggered with your guest os, then you could try to
> capture the traffic with wireshark (or something else) in a
> configuration that doesn't crash your system. Save the traffic in a pcap
> file. Then you can see if resending that traffic in the vulnerable
> configuration triggers the bug (I don't know if something in Windows
> exists, but tcpreplay should work for Linux). Once you have such a
> capture , chances are the bug is even easily reproducible by us (unless
> it's hardware-specific). Success isn't guaranteed, but I think it's
> worth a shot...

The issue with this is I don't have a configuration that does not crash 
the system. This only happens under the specific circumstance that 
traffic from VM A is being DNAT'd to VM B. If I disable 
CONFIG_BRIDGE_NETFILTER, or I leave out the DNAT then I can't replicate 
the problem as I don't seem to be able to get the packets to go where I 
want them to go.

Let me try and explain it a little more clearly with made up IP 
addresses to illustrate the problem.

I have VM A (1.1.1.2) and VM B (1.1.1.3) on br1 (1.1.1.1)
I have public IP on ppp0 (2.2.2.2).

VM B can talk to VM A using its host address (1.1.1.2) and there is no 
problem.

The DNAT says anything destined for PPP0 that is on port 443 and coming 
from anywhere other than PPP0 (ie inside the network) is to be DNAT'd to 
1.1.1.3.

So VM B (1.1.1.3) tries to connect to ppp0 (2.2.2.2) on port 443, and 
this is redirected to VM B on 1.1.1.2.

Only under this specific circumstance does the problem occur. I can get 
VM B (1.1.1.3) to talk directly to VM A (1.1.1.2) all day long and there 
is no problem, it's only when VM B tries to talk to ppp0 that there is 
an issue (and it happens within seconds of the initial connection).

All these tests have been performed with VM B being a Windows XP guest. 
Tonight I'll try it with a Linux guest and see if I can make it happen. 
If that works I might be able to come up with some reproducible test 
case for you. I have a desktop machine that has Intel VT extensions, so 
I'll work toward making a portable test case.

^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
@ 2011-06-08  0:15                                                     ` Brad Campbell
  0 siblings, 0 replies; 111+ messages in thread
From: Brad Campbell @ 2011-06-08  0:15 UTC (permalink / raw)
  To: Bart De Schuymer
  Cc: Patrick McHardy, kvm, linux-mm, linux-kernel, netdev, netfilter-devel

On 08/06/11 02:04, Bart De Schuymer wrote:

> If the bug is easily triggered with your guest os, then you could try to
> capture the traffic with wireshark (or something else) in a
> configuration that doesn't crash your system. Save the traffic in a pcap
> file. Then you can see if resending that traffic in the vulnerable
> configuration triggers the bug (I don't know if something in Windows
> exists, but tcpreplay should work for Linux). Once you have such a
> capture , chances are the bug is even easily reproducible by us (unless
> it's hardware-specific). Success isn't guaranteed, but I think it's
> worth a shot...

The issue with this is I don't have a configuration that does not crash 
the system. This only happens under the specific circumstance that 
traffic from VM A is being DNAT'd to VM B. If I disable 
CONFIG_BRIDGE_NETFILTER, or I leave out the DNAT then I can't replicate 
the problem as I don't seem to be able to get the packets to go where I 
want them to go.

Let me try and explain it a little more clearly with made up IP 
addresses to illustrate the problem.

I have VM A (1.1.1.2) and VM B (1.1.1.3) on br1 (1.1.1.1)
I have public IP on ppp0 (2.2.2.2).

VM B can talk to VM A using its host address (1.1.1.2) and there is no 
problem.

The DNAT says anything destined for PPP0 that is on port 443 and coming 
from anywhere other than PPP0 (ie inside the network) is to be DNAT'd to 
1.1.1.3.

So VM B (1.1.1.3) tries to connect to ppp0 (2.2.2.2) on port 443, and 
this is redirected to VM B on 1.1.1.2.

Only under this specific circumstance does the problem occur. I can get 
VM B (1.1.1.3) to talk directly to VM A (1.1.1.2) all day long and there 
is no problem, it's only when VM B tries to talk to ppp0 that there is 
an issue (and it happens within seconds of the initial connection).

All these tests have been performed with VM B being a Windows XP guest. 
Tonight I'll try it with a Linux guest and see if I can make it happen. 
If that works I might be able to come up with some reproducible test 
case for you. I have a desktop machine that has Intel VT extensions, so 
I'll work toward making a portable test case.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
  2011-06-07 22:57                                                       ` Patrick McHardy
  (?)
@ 2011-06-08  0:18                                                         ` Brad Campbell
  -1 siblings, 0 replies; 111+ messages in thread
From: Brad Campbell @ 2011-06-08  0:18 UTC (permalink / raw)
  To: Patrick McHardy
  Cc: Eric Dumazet, Bart De Schuymer, kvm, linux-mm, linux-kernel,
	netdev, netfilter-devel

On 08/06/11 06:57, Patrick McHardy wrote:
> On 07.06.2011 20:31, Eric Dumazet wrote:
>> Le mardi 07 juin 2011 à 17:35 +0200, Patrick McHardy a écrit :
>>
>>> The main suspects would be NAT and TCPMSS. Did you also try whether
>>> the crash occurs with only one of these these rules?
>>>
>>>> I've just compiled out CONFIG_BRIDGE_NETFILTER and can no longer access
>>>> the address the way I was doing it, so that's a no-go for me.
>>>
>>> That's really weird since you're apparently not using any bridge
>>> netfilter features. It shouldn't have any effect besides changing
>>> at which point ip_tables is invoked. How are your network devices
>>> configured (specifically any bridges)?
>>
>> Something in the kernel does
>>
>> u16 *ptr = addr (given by kmalloc())
>>
>> ptr[-1] = 0;
>>
>> Could be an off-one error in a memmove()/memcopy() or loop...
>>
>> I cant see a network issue here.
>
> So far me neither, but netfilter appears to trigger the bug.

Would it help if I tried some older kernels? This issue only surfaced 
for me recently as I only installed the VM's in question about 12 weeks 
ago and have only just started really using them in anger. I could try 
reproducing it on progressively older kernels to see if I can find one 
that works and then bisect from there.

^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
@ 2011-06-08  0:18                                                         ` Brad Campbell
  0 siblings, 0 replies; 111+ messages in thread
From: Brad Campbell @ 2011-06-08  0:18 UTC (permalink / raw)
  To: Patrick McHardy
  Cc: Eric Dumazet, Bart De Schuymer, kvm, linux-mm, linux-kernel,
	netdev, netfilter-devel

On 08/06/11 06:57, Patrick McHardy wrote:
> On 07.06.2011 20:31, Eric Dumazet wrote:
>> Le mardi 07 juin 2011 à 17:35 +0200, Patrick McHardy a écrit :
>>
>>> The main suspects would be NAT and TCPMSS. Did you also try whether
>>> the crash occurs with only one of these these rules?
>>>
>>>> I've just compiled out CONFIG_BRIDGE_NETFILTER and can no longer access
>>>> the address the way I was doing it, so that's a no-go for me.
>>>
>>> That's really weird since you're apparently not using any bridge
>>> netfilter features. It shouldn't have any effect besides changing
>>> at which point ip_tables is invoked. How are your network devices
>>> configured (specifically any bridges)?
>>
>> Something in the kernel does
>>
>> u16 *ptr = addr (given by kmalloc())
>>
>> ptr[-1] = 0;
>>
>> Could be an off-one error in a memmove()/memcopy() or loop...
>>
>> I cant see a network issue here.
>
> So far me neither, but netfilter appears to trigger the bug.

Would it help if I tried some older kernels? This issue only surfaced 
for me recently as I only installed the VM's in question about 12 weeks 
ago and have only just started really using them in anger. I could try 
reproducing it on progressively older kernels to see if I can find one 
that works and then bisect from there.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
@ 2011-06-08  0:18                                                         ` Brad Campbell
  0 siblings, 0 replies; 111+ messages in thread
From: Brad Campbell @ 2011-06-08  0:18 UTC (permalink / raw)
  To: Patrick McHardy
  Cc: Eric Dumazet, Bart De Schuymer, kvm, linux-mm, linux-kernel,
	netdev, netfilter-devel

On 08/06/11 06:57, Patrick McHardy wrote:
> On 07.06.2011 20:31, Eric Dumazet wrote:
>> Le mardi 07 juin 2011 a 17:35 +0200, Patrick McHardy a ecrit :
>>
>>> The main suspects would be NAT and TCPMSS. Did you also try whether
>>> the crash occurs with only one of these these rules?
>>>
>>>> I've just compiled out CONFIG_BRIDGE_NETFILTER and can no longer access
>>>> the address the way I was doing it, so that's a no-go for me.
>>>
>>> That's really weird since you're apparently not using any bridge
>>> netfilter features. It shouldn't have any effect besides changing
>>> at which point ip_tables is invoked. How are your network devices
>>> configured (specifically any bridges)?
>>
>> Something in the kernel does
>>
>> u16 *ptr = addr (given by kmalloc())
>>
>> ptr[-1] = 0;
>>
>> Could be an off-one error in a memmove()/memcopy() or loop...
>>
>> I cant see a network issue here.
>
> So far me neither, but netfilter appears to trigger the bug.

Would it help if I tried some older kernels? This issue only surfaced 
for me recently as I only installed the VM's in question about 12 weeks 
ago and have only just started really using them in anger. I could try 
reproducing it on progressively older kernels to see if I can find one 
that works and then bisect from there.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
  2011-06-08  0:18                                                         ` Brad Campbell
  (?)
@ 2011-06-08  3:59                                                           ` Eric Dumazet
  -1 siblings, 0 replies; 111+ messages in thread
From: Eric Dumazet @ 2011-06-08  3:59 UTC (permalink / raw)
  To: Brad Campbell
  Cc: Patrick McHardy, Bart De Schuymer, kvm, linux-mm, linux-kernel,
	netdev, netfilter-devel

Le mercredi 08 juin 2011 à 08:18 +0800, Brad Campbell a écrit :
> On 08/06/11 06:57, Patrick McHardy wrote:
> > On 07.06.2011 20:31, Eric Dumazet wrote:
> >> Le mardi 07 juin 2011 à 17:35 +0200, Patrick McHardy a écrit :
> >>
> >>> The main suspects would be NAT and TCPMSS. Did you also try whether
> >>> the crash occurs with only one of these these rules?
> >>>
> >>>> I've just compiled out CONFIG_BRIDGE_NETFILTER and can no longer access
> >>>> the address the way I was doing it, so that's a no-go for me.
> >>>
> >>> That's really weird since you're apparently not using any bridge
> >>> netfilter features. It shouldn't have any effect besides changing
> >>> at which point ip_tables is invoked. How are your network devices
> >>> configured (specifically any bridges)?
> >>
> >> Something in the kernel does
> >>
> >> u16 *ptr = addr (given by kmalloc())
> >>
> >> ptr[-1] = 0;
> >>
> >> Could be an off-one error in a memmove()/memcopy() or loop...
> >>
> >> I cant see a network issue here.
> >
> > So far me neither, but netfilter appears to trigger the bug.
> 
> Would it help if I tried some older kernels? This issue only surfaced 
> for me recently as I only installed the VM's in question about 12 weeks 
> ago and have only just started really using them in anger. I could try 
> reproducing it on progressively older kernels to see if I can find one 
> that works and then bisect from there.

Well, a bisection definitely should help, but needs a lot of time in
your case.

Could you try following patch, because this is the 'usual suspect' I had
yesterday :

diff --git a/net/core/skbuff.c b/net/core/skbuff.c
index 46cbd28..9f548f9 100644
--- a/net/core/skbuff.c
+++ b/net/core/skbuff.c
@@ -792,6 +792,7 @@ int pskb_expand_head(struct sk_buff *skb, int nhead, int ntail,
 		fastpath = atomic_read(&skb_shinfo(skb)->dataref) == delta;
 	}
 
+#if 0
 	if (fastpath &&
 	    size + sizeof(struct skb_shared_info) <= ksize(skb->head)) {
 		memmove(skb->head + size, skb_shinfo(skb),
@@ -802,7 +803,7 @@ int pskb_expand_head(struct sk_buff *skb, int nhead, int ntail,
 		off = nhead;
 		goto adjust_others;
 	}
-
+#endif
 	data = kmalloc(size + sizeof(struct skb_shared_info), gfp_mask);
 	if (!data)
 		goto nodata;



^ permalink raw reply related	[flat|nested] 111+ messages in thread

* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
@ 2011-06-08  3:59                                                           ` Eric Dumazet
  0 siblings, 0 replies; 111+ messages in thread
From: Eric Dumazet @ 2011-06-08  3:59 UTC (permalink / raw)
  To: Brad Campbell
  Cc: Patrick McHardy, Bart De Schuymer, kvm, linux-mm, linux-kernel,
	netdev, netfilter-devel

Le mercredi 08 juin 2011 à 08:18 +0800, Brad Campbell a écrit :
> On 08/06/11 06:57, Patrick McHardy wrote:
> > On 07.06.2011 20:31, Eric Dumazet wrote:
> >> Le mardi 07 juin 2011 à 17:35 +0200, Patrick McHardy a écrit :
> >>
> >>> The main suspects would be NAT and TCPMSS. Did you also try whether
> >>> the crash occurs with only one of these these rules?
> >>>
> >>>> I've just compiled out CONFIG_BRIDGE_NETFILTER and can no longer access
> >>>> the address the way I was doing it, so that's a no-go for me.
> >>>
> >>> That's really weird since you're apparently not using any bridge
> >>> netfilter features. It shouldn't have any effect besides changing
> >>> at which point ip_tables is invoked. How are your network devices
> >>> configured (specifically any bridges)?
> >>
> >> Something in the kernel does
> >>
> >> u16 *ptr = addr (given by kmalloc())
> >>
> >> ptr[-1] = 0;
> >>
> >> Could be an off-one error in a memmove()/memcopy() or loop...
> >>
> >> I cant see a network issue here.
> >
> > So far me neither, but netfilter appears to trigger the bug.
> 
> Would it help if I tried some older kernels? This issue only surfaced 
> for me recently as I only installed the VM's in question about 12 weeks 
> ago and have only just started really using them in anger. I could try 
> reproducing it on progressively older kernels to see if I can find one 
> that works and then bisect from there.

Well, a bisection definitely should help, but needs a lot of time in
your case.

Could you try following patch, because this is the 'usual suspect' I had
yesterday :

diff --git a/net/core/skbuff.c b/net/core/skbuff.c
index 46cbd28..9f548f9 100644
--- a/net/core/skbuff.c
+++ b/net/core/skbuff.c
@@ -792,6 +792,7 @@ int pskb_expand_head(struct sk_buff *skb, int nhead, int ntail,
 		fastpath = atomic_read(&skb_shinfo(skb)->dataref) == delta;
 	}
 
+#if 0
 	if (fastpath &&
 	    size + sizeof(struct skb_shared_info) <= ksize(skb->head)) {
 		memmove(skb->head + size, skb_shinfo(skb),
@@ -802,7 +803,7 @@ int pskb_expand_head(struct sk_buff *skb, int nhead, int ntail,
 		off = nhead;
 		goto adjust_others;
 	}
-
+#endif
 	data = kmalloc(size + sizeof(struct skb_shared_info), gfp_mask);
 	if (!data)
 		goto nodata;


--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

^ permalink raw reply related	[flat|nested] 111+ messages in thread

* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
@ 2011-06-08  3:59                                                           ` Eric Dumazet
  0 siblings, 0 replies; 111+ messages in thread
From: Eric Dumazet @ 2011-06-08  3:59 UTC (permalink / raw)
  To: Brad Campbell
  Cc: Patrick McHardy, Bart De Schuymer, kvm, linux-mm, linux-kernel,
	netdev, netfilter-devel

Le mercredi 08 juin 2011 A  08:18 +0800, Brad Campbell a A(C)crit :
> On 08/06/11 06:57, Patrick McHardy wrote:
> > On 07.06.2011 20:31, Eric Dumazet wrote:
> >> Le mardi 07 juin 2011 A  17:35 +0200, Patrick McHardy a A(C)crit :
> >>
> >>> The main suspects would be NAT and TCPMSS. Did you also try whether
> >>> the crash occurs with only one of these these rules?
> >>>
> >>>> I've just compiled out CONFIG_BRIDGE_NETFILTER and can no longer access
> >>>> the address the way I was doing it, so that's a no-go for me.
> >>>
> >>> That's really weird since you're apparently not using any bridge
> >>> netfilter features. It shouldn't have any effect besides changing
> >>> at which point ip_tables is invoked. How are your network devices
> >>> configured (specifically any bridges)?
> >>
> >> Something in the kernel does
> >>
> >> u16 *ptr = addr (given by kmalloc())
> >>
> >> ptr[-1] = 0;
> >>
> >> Could be an off-one error in a memmove()/memcopy() or loop...
> >>
> >> I cant see a network issue here.
> >
> > So far me neither, but netfilter appears to trigger the bug.
> 
> Would it help if I tried some older kernels? This issue only surfaced 
> for me recently as I only installed the VM's in question about 12 weeks 
> ago and have only just started really using them in anger. I could try 
> reproducing it on progressively older kernels to see if I can find one 
> that works and then bisect from there.

Well, a bisection definitely should help, but needs a lot of time in
your case.

Could you try following patch, because this is the 'usual suspect' I had
yesterday :

diff --git a/net/core/skbuff.c b/net/core/skbuff.c
index 46cbd28..9f548f9 100644
--- a/net/core/skbuff.c
+++ b/net/core/skbuff.c
@@ -792,6 +792,7 @@ int pskb_expand_head(struct sk_buff *skb, int nhead, int ntail,
 		fastpath = atomic_read(&skb_shinfo(skb)->dataref) == delta;
 	}
 
+#if 0
 	if (fastpath &&
 	    size + sizeof(struct skb_shared_info) <= ksize(skb->head)) {
 		memmove(skb->head + size, skb_shinfo(skb),
@@ -802,7 +803,7 @@ int pskb_expand_head(struct sk_buff *skb, int nhead, int ntail,
 		off = nhead;
 		goto adjust_others;
 	}
-
+#endif
 	data = kmalloc(size + sizeof(struct skb_shared_info), gfp_mask);
 	if (!data)
 		goto nodata;


--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

^ permalink raw reply related	[flat|nested] 111+ messages in thread

* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
  2011-06-08  3:59                                                           ` Eric Dumazet
  (?)
@ 2011-06-08 17:02                                                             ` Brad Campbell
  -1 siblings, 0 replies; 111+ messages in thread
From: Brad Campbell @ 2011-06-08 17:02 UTC (permalink / raw)
  To: Eric Dumazet
  Cc: Patrick McHardy, Bart De Schuymer, kvm, linux-mm, linux-kernel,
	netdev, netfilter-devel

On 08/06/11 11:59, Eric Dumazet wrote:

> Well, a bisection definitely should help, but needs a lot of time in
> your case.

Yes. compile, test, crash, walk out to the other building to press 
reset, lather, rinse, repeat.

I need a reset button on the end of a 50M wire, or a hardware watchdog!

Actually it's not so bad. If I turn off slub debugging the kernel panics 
and reboots itself.

This.. :
[    2.913034] netconsole: remote ethernet address 00:16:cb:a7:dd:d1
[    2.913066] netconsole: device eth0 not up yet, forcing it
[    3.660062] Refined TSC clocksource calibration: 3213.422 MHz.
[    3.660118] Switching to clocksource tsc
[   63.200273] r8169 0000:03:00.0: eth0: unable to load firmware patch 
rtl_nic/rtl8168e-1.fw (-2)
[   63.223513] r8169 0000:03:00.0: eth0: link down
[   63.223556] r8169 0000:03:00.0: eth0: link down

..is slowing down reboots considerably. 3.0-rc does _not_ like some 
timing hardware in my machine. Having said that, at least it does not 
randomly panic on SCSI like 2.6.39 does.

Ok, I've ruled out TCPMSS. Found out where it was being set and neutered 
it. I've replicated it with only the single DNAT rule.


> Could you try following patch, because this is the 'usual suspect' I had
> yesterday :
>
> diff --git a/net/core/skbuff.c b/net/core/skbuff.c
> index 46cbd28..9f548f9 100644
> --- a/net/core/skbuff.c
> +++ b/net/core/skbuff.c
> @@ -792,6 +792,7 @@ int pskb_expand_head(struct sk_buff *skb, int nhead, int ntail,
>   		fastpath = atomic_read(&skb_shinfo(skb)->dataref) == delta;
>   	}
>
> +#if 0
>   	if (fastpath&&
>   	size + sizeof(struct skb_shared_info)<= ksize(skb->head)) {
>   		memmove(skb->head + size, skb_shinfo(skb),
> @@ -802,7 +803,7 @@ int pskb_expand_head(struct sk_buff *skb, int nhead, int ntail,
>   		off = nhead;
>   		goto adjust_others;
>   	}
> -
> +#endif
>   	data = kmalloc(size + sizeof(struct skb_shared_info), gfp_mask);
>   	if (!data)
>   		goto nodata;
>
>
>

Nope.. that's not it. <sigh> That might have changed the characteristic 
of the fault slightly, but unfortunately I got caught with a couple of 
fsck's, so I only got to test it 3 times tonight.

It's unfortunate that this is a production system, so I can only take it 
down between about 9pm and 1am. That would normally be pretty 
productive, except that an fsck of a 14TB ext4 can take 30 minutes if it 
panics at the wrong time.

I'm out of time tonight, but I'll have a crack at some bisection 
tomorrow night. Now I just have to go back far enough that it works, and 
be near enough not to have to futz around with /proc /sys or drivers.

I really, really, really appreciate you guys helping me with this. It 
has been driving me absolutely bonkers. If I'm ever in the same town as 
any of you, dinner and drinks are on me.

^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
@ 2011-06-08 17:02                                                             ` Brad Campbell
  0 siblings, 0 replies; 111+ messages in thread
From: Brad Campbell @ 2011-06-08 17:02 UTC (permalink / raw)
  To: Eric Dumazet
  Cc: Patrick McHardy, Bart De Schuymer, kvm, linux-mm, linux-kernel,
	netdev, netfilter-devel

On 08/06/11 11:59, Eric Dumazet wrote:

> Well, a bisection definitely should help, but needs a lot of time in
> your case.

Yes. compile, test, crash, walk out to the other building to press 
reset, lather, rinse, repeat.

I need a reset button on the end of a 50M wire, or a hardware watchdog!

Actually it's not so bad. If I turn off slub debugging the kernel panics 
and reboots itself.

This.. :
[    2.913034] netconsole: remote ethernet address 00:16:cb:a7:dd:d1
[    2.913066] netconsole: device eth0 not up yet, forcing it
[    3.660062] Refined TSC clocksource calibration: 3213.422 MHz.
[    3.660118] Switching to clocksource tsc
[   63.200273] r8169 0000:03:00.0: eth0: unable to load firmware patch 
rtl_nic/rtl8168e-1.fw (-2)
[   63.223513] r8169 0000:03:00.0: eth0: link down
[   63.223556] r8169 0000:03:00.0: eth0: link down

..is slowing down reboots considerably. 3.0-rc does _not_ like some 
timing hardware in my machine. Having said that, at least it does not 
randomly panic on SCSI like 2.6.39 does.

Ok, I've ruled out TCPMSS. Found out where it was being set and neutered 
it. I've replicated it with only the single DNAT rule.


> Could you try following patch, because this is the 'usual suspect' I had
> yesterday :
>
> diff --git a/net/core/skbuff.c b/net/core/skbuff.c
> index 46cbd28..9f548f9 100644
> --- a/net/core/skbuff.c
> +++ b/net/core/skbuff.c
> @@ -792,6 +792,7 @@ int pskb_expand_head(struct sk_buff *skb, int nhead, int ntail,
>   		fastpath = atomic_read(&skb_shinfo(skb)->dataref) == delta;
>   	}
>
> +#if 0
>   	if (fastpath&&
>   	size + sizeof(struct skb_shared_info)<= ksize(skb->head)) {
>   		memmove(skb->head + size, skb_shinfo(skb),
> @@ -802,7 +803,7 @@ int pskb_expand_head(struct sk_buff *skb, int nhead, int ntail,
>   		off = nhead;
>   		goto adjust_others;
>   	}
> -
> +#endif
>   	data = kmalloc(size + sizeof(struct skb_shared_info), gfp_mask);
>   	if (!data)
>   		goto nodata;
>
>
>

Nope.. that's not it. <sigh> That might have changed the characteristic 
of the fault slightly, but unfortunately I got caught with a couple of 
fsck's, so I only got to test it 3 times tonight.

It's unfortunate that this is a production system, so I can only take it 
down between about 9pm and 1am. That would normally be pretty 
productive, except that an fsck of a 14TB ext4 can take 30 minutes if it 
panics at the wrong time.

I'm out of time tonight, but I'll have a crack at some bisection 
tomorrow night. Now I just have to go back far enough that it works, and 
be near enough not to have to futz around with /proc /sys or drivers.

I really, really, really appreciate you guys helping me with this. It 
has been driving me absolutely bonkers. If I'm ever in the same town as 
any of you, dinner and drinks are on me.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
@ 2011-06-08 17:02                                                             ` Brad Campbell
  0 siblings, 0 replies; 111+ messages in thread
From: Brad Campbell @ 2011-06-08 17:02 UTC (permalink / raw)
  To: Eric Dumazet
  Cc: Patrick McHardy, Bart De Schuymer, kvm, linux-mm, linux-kernel,
	netdev, netfilter-devel

On 08/06/11 11:59, Eric Dumazet wrote:

> Well, a bisection definitely should help, but needs a lot of time in
> your case.

Yes. compile, test, crash, walk out to the other building to press 
reset, lather, rinse, repeat.

I need a reset button on the end of a 50M wire, or a hardware watchdog!

Actually it's not so bad. If I turn off slub debugging the kernel panics 
and reboots itself.

This.. :
[    2.913034] netconsole: remote ethernet address 00:16:cb:a7:dd:d1
[    2.913066] netconsole: device eth0 not up yet, forcing it
[    3.660062] Refined TSC clocksource calibration: 3213.422 MHz.
[    3.660118] Switching to clocksource tsc
[   63.200273] r8169 0000:03:00.0: eth0: unable to load firmware patch 
rtl_nic/rtl8168e-1.fw (-2)
[   63.223513] r8169 0000:03:00.0: eth0: link down
[   63.223556] r8169 0000:03:00.0: eth0: link down

..is slowing down reboots considerably. 3.0-rc does _not_ like some 
timing hardware in my machine. Having said that, at least it does not 
randomly panic on SCSI like 2.6.39 does.

Ok, I've ruled out TCPMSS. Found out where it was being set and neutered 
it. I've replicated it with only the single DNAT rule.


> Could you try following patch, because this is the 'usual suspect' I had
> yesterday :
>
> diff --git a/net/core/skbuff.c b/net/core/skbuff.c
> index 46cbd28..9f548f9 100644
> --- a/net/core/skbuff.c
> +++ b/net/core/skbuff.c
> @@ -792,6 +792,7 @@ int pskb_expand_head(struct sk_buff *skb, int nhead, int ntail,
>   		fastpath = atomic_read(&skb_shinfo(skb)->dataref) == delta;
>   	}
>
> +#if 0
>   	if (fastpath&&
>   	size + sizeof(struct skb_shared_info)<= ksize(skb->head)) {
>   		memmove(skb->head + size, skb_shinfo(skb),
> @@ -802,7 +803,7 @@ int pskb_expand_head(struct sk_buff *skb, int nhead, int ntail,
>   		off = nhead;
>   		goto adjust_others;
>   	}
> -
> +#endif
>   	data = kmalloc(size + sizeof(struct skb_shared_info), gfp_mask);
>   	if (!data)
>   		goto nodata;
>
>
>

Nope.. that's not it. <sigh> That might have changed the characteristic 
of the fault slightly, but unfortunately I got caught with a couple of 
fsck's, so I only got to test it 3 times tonight.

It's unfortunate that this is a production system, so I can only take it 
down between about 9pm and 1am. That would normally be pretty 
productive, except that an fsck of a 14TB ext4 can take 30 minutes if it 
panics at the wrong time.

I'm out of time tonight, but I'll have a crack at some bisection 
tomorrow night. Now I just have to go back far enough that it works, and 
be near enough not to have to futz around with /proc /sys or drivers.

I really, really, really appreciate you guys helping me with this. It 
has been driving me absolutely bonkers. If I'm ever in the same town as 
any of you, dinner and drinks are on me.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
  2011-06-08 17:02                                                             ` Brad Campbell
  (?)
@ 2011-06-08 21:22                                                               ` Eric Dumazet
  -1 siblings, 0 replies; 111+ messages in thread
From: Eric Dumazet @ 2011-06-08 21:22 UTC (permalink / raw)
  To: Brad Campbell
  Cc: Patrick McHardy, Bart De Schuymer, kvm, linux-mm, linux-kernel,
	netdev, netfilter-devel

Le jeudi 09 juin 2011 à 01:02 +0800, Brad Campbell a écrit :
> On 08/06/11 11:59, Eric Dumazet wrote:
> 
> > Well, a bisection definitely should help, but needs a lot of time in
> > your case.
> 
> Yes. compile, test, crash, walk out to the other building to press 
> reset, lather, rinse, repeat.
> 
> I need a reset button on the end of a 50M wire, or a hardware watchdog!
> 
> Actually it's not so bad. If I turn off slub debugging the kernel panics 
> and reboots itself.
> 
> This.. :
> [    2.913034] netconsole: remote ethernet address 00:16:cb:a7:dd:d1
> [    2.913066] netconsole: device eth0 not up yet, forcing it
> [    3.660062] Refined TSC clocksource calibration: 3213.422 MHz.
> [    3.660118] Switching to clocksource tsc
> [   63.200273] r8169 0000:03:00.0: eth0: unable to load firmware patch 
> rtl_nic/rtl8168e-1.fw (-2)
> [   63.223513] r8169 0000:03:00.0: eth0: link down
> [   63.223556] r8169 0000:03:00.0: eth0: link down
> 
> ..is slowing down reboots considerably. 3.0-rc does _not_ like some 
> timing hardware in my machine. Having said that, at least it does not 
> randomly panic on SCSI like 2.6.39 does.
> 
> Ok, I've ruled out TCPMSS. Found out where it was being set and neutered 
> it. I've replicated it with only the single DNAT rule.
> 
> 
> > Could you try following patch, because this is the 'usual suspect' I had
> > yesterday :
> >
> > diff --git a/net/core/skbuff.c b/net/core/skbuff.c
> > index 46cbd28..9f548f9 100644
> > --- a/net/core/skbuff.c
> > +++ b/net/core/skbuff.c
> > @@ -792,6 +792,7 @@ int pskb_expand_head(struct sk_buff *skb, int nhead, int ntail,
> >   		fastpath = atomic_read(&skb_shinfo(skb)->dataref) == delta;
> >   	}
> >
> > +#if 0
> >   	if (fastpath&&
> >   	size + sizeof(struct skb_shared_info)<= ksize(skb->head)) {
> >   		memmove(skb->head + size, skb_shinfo(skb),
> > @@ -802,7 +803,7 @@ int pskb_expand_head(struct sk_buff *skb, int nhead, int ntail,
> >   		off = nhead;
> >   		goto adjust_others;
> >   	}
> > -
> > +#endif
> >   	data = kmalloc(size + sizeof(struct skb_shared_info), gfp_mask);
> >   	if (!data)
> >   		goto nodata;
> >
> >
> >
> 
> Nope.. that's not it. <sigh> That might have changed the characteristic 
> of the fault slightly, but unfortunately I got caught with a couple of 
> fsck's, so I only got to test it 3 times tonight.
> 
> It's unfortunate that this is a production system, so I can only take it 
> down between about 9pm and 1am. That would normally be pretty 
> productive, except that an fsck of a 14TB ext4 can take 30 minutes if it 
> panics at the wrong time.
> 
> I'm out of time tonight, but I'll have a crack at some bisection 
> tomorrow night. Now I just have to go back far enough that it works, and 
> be near enough not to have to futz around with /proc /sys or drivers.
> 
> I really, really, really appreciate you guys helping me with this. It 
> has been driving me absolutely bonkers. If I'm ever in the same town as 
> any of you, dinner and drinks are on me.

Hmm, I wonder if kmemcheck could help you, but its slow as hell, so not
appropriate for production :(




^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
@ 2011-06-08 21:22                                                               ` Eric Dumazet
  0 siblings, 0 replies; 111+ messages in thread
From: Eric Dumazet @ 2011-06-08 21:22 UTC (permalink / raw)
  To: Brad Campbell
  Cc: Patrick McHardy, Bart De Schuymer, kvm, linux-mm, linux-kernel,
	netdev, netfilter-devel

Le jeudi 09 juin 2011 à 01:02 +0800, Brad Campbell a écrit :
> On 08/06/11 11:59, Eric Dumazet wrote:
> 
> > Well, a bisection definitely should help, but needs a lot of time in
> > your case.
> 
> Yes. compile, test, crash, walk out to the other building to press 
> reset, lather, rinse, repeat.
> 
> I need a reset button on the end of a 50M wire, or a hardware watchdog!
> 
> Actually it's not so bad. If I turn off slub debugging the kernel panics 
> and reboots itself.
> 
> This.. :
> [    2.913034] netconsole: remote ethernet address 00:16:cb:a7:dd:d1
> [    2.913066] netconsole: device eth0 not up yet, forcing it
> [    3.660062] Refined TSC clocksource calibration: 3213.422 MHz.
> [    3.660118] Switching to clocksource tsc
> [   63.200273] r8169 0000:03:00.0: eth0: unable to load firmware patch 
> rtl_nic/rtl8168e-1.fw (-2)
> [   63.223513] r8169 0000:03:00.0: eth0: link down
> [   63.223556] r8169 0000:03:00.0: eth0: link down
> 
> ..is slowing down reboots considerably. 3.0-rc does _not_ like some 
> timing hardware in my machine. Having said that, at least it does not 
> randomly panic on SCSI like 2.6.39 does.
> 
> Ok, I've ruled out TCPMSS. Found out where it was being set and neutered 
> it. I've replicated it with only the single DNAT rule.
> 
> 
> > Could you try following patch, because this is the 'usual suspect' I had
> > yesterday :
> >
> > diff --git a/net/core/skbuff.c b/net/core/skbuff.c
> > index 46cbd28..9f548f9 100644
> > --- a/net/core/skbuff.c
> > +++ b/net/core/skbuff.c
> > @@ -792,6 +792,7 @@ int pskb_expand_head(struct sk_buff *skb, int nhead, int ntail,
> >   		fastpath = atomic_read(&skb_shinfo(skb)->dataref) == delta;
> >   	}
> >
> > +#if 0
> >   	if (fastpath&&
> >   	size + sizeof(struct skb_shared_info)<= ksize(skb->head)) {
> >   		memmove(skb->head + size, skb_shinfo(skb),
> > @@ -802,7 +803,7 @@ int pskb_expand_head(struct sk_buff *skb, int nhead, int ntail,
> >   		off = nhead;
> >   		goto adjust_others;
> >   	}
> > -
> > +#endif
> >   	data = kmalloc(size + sizeof(struct skb_shared_info), gfp_mask);
> >   	if (!data)
> >   		goto nodata;
> >
> >
> >
> 
> Nope.. that's not it. <sigh> That might have changed the characteristic 
> of the fault slightly, but unfortunately I got caught with a couple of 
> fsck's, so I only got to test it 3 times tonight.
> 
> It's unfortunate that this is a production system, so I can only take it 
> down between about 9pm and 1am. That would normally be pretty 
> productive, except that an fsck of a 14TB ext4 can take 30 minutes if it 
> panics at the wrong time.
> 
> I'm out of time tonight, but I'll have a crack at some bisection 
> tomorrow night. Now I just have to go back far enough that it works, and 
> be near enough not to have to futz around with /proc /sys or drivers.
> 
> I really, really, really appreciate you guys helping me with this. It 
> has been driving me absolutely bonkers. If I'm ever in the same town as 
> any of you, dinner and drinks are on me.

Hmm, I wonder if kmemcheck could help you, but its slow as hell, so not
appropriate for production :(



--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
@ 2011-06-08 21:22                                                               ` Eric Dumazet
  0 siblings, 0 replies; 111+ messages in thread
From: Eric Dumazet @ 2011-06-08 21:22 UTC (permalink / raw)
  To: Brad Campbell
  Cc: Patrick McHardy, Bart De Schuymer, kvm, linux-mm, linux-kernel,
	netdev, netfilter-devel

Le jeudi 09 juin 2011 A  01:02 +0800, Brad Campbell a A(C)crit :
> On 08/06/11 11:59, Eric Dumazet wrote:
> 
> > Well, a bisection definitely should help, but needs a lot of time in
> > your case.
> 
> Yes. compile, test, crash, walk out to the other building to press 
> reset, lather, rinse, repeat.
> 
> I need a reset button on the end of a 50M wire, or a hardware watchdog!
> 
> Actually it's not so bad. If I turn off slub debugging the kernel panics 
> and reboots itself.
> 
> This.. :
> [    2.913034] netconsole: remote ethernet address 00:16:cb:a7:dd:d1
> [    2.913066] netconsole: device eth0 not up yet, forcing it
> [    3.660062] Refined TSC clocksource calibration: 3213.422 MHz.
> [    3.660118] Switching to clocksource tsc
> [   63.200273] r8169 0000:03:00.0: eth0: unable to load firmware patch 
> rtl_nic/rtl8168e-1.fw (-2)
> [   63.223513] r8169 0000:03:00.0: eth0: link down
> [   63.223556] r8169 0000:03:00.0: eth0: link down
> 
> ..is slowing down reboots considerably. 3.0-rc does _not_ like some 
> timing hardware in my machine. Having said that, at least it does not 
> randomly panic on SCSI like 2.6.39 does.
> 
> Ok, I've ruled out TCPMSS. Found out where it was being set and neutered 
> it. I've replicated it with only the single DNAT rule.
> 
> 
> > Could you try following patch, because this is the 'usual suspect' I had
> > yesterday :
> >
> > diff --git a/net/core/skbuff.c b/net/core/skbuff.c
> > index 46cbd28..9f548f9 100644
> > --- a/net/core/skbuff.c
> > +++ b/net/core/skbuff.c
> > @@ -792,6 +792,7 @@ int pskb_expand_head(struct sk_buff *skb, int nhead, int ntail,
> >   		fastpath = atomic_read(&skb_shinfo(skb)->dataref) == delta;
> >   	}
> >
> > +#if 0
> >   	if (fastpath&&
> >   	size + sizeof(struct skb_shared_info)<= ksize(skb->head)) {
> >   		memmove(skb->head + size, skb_shinfo(skb),
> > @@ -802,7 +803,7 @@ int pskb_expand_head(struct sk_buff *skb, int nhead, int ntail,
> >   		off = nhead;
> >   		goto adjust_others;
> >   	}
> > -
> > +#endif
> >   	data = kmalloc(size + sizeof(struct skb_shared_info), gfp_mask);
> >   	if (!data)
> >   		goto nodata;
> >
> >
> >
> 
> Nope.. that's not it. <sigh> That might have changed the characteristic 
> of the fault slightly, but unfortunately I got caught with a couple of 
> fsck's, so I only got to test it 3 times tonight.
> 
> It's unfortunate that this is a production system, so I can only take it 
> down between about 9pm and 1am. That would normally be pretty 
> productive, except that an fsck of a 14TB ext4 can take 30 minutes if it 
> panics at the wrong time.
> 
> I'm out of time tonight, but I'll have a crack at some bisection 
> tomorrow night. Now I just have to go back far enough that it works, and 
> be near enough not to have to futz around with /proc /sys or drivers.
> 
> I really, really, really appreciate you guys helping me with this. It 
> has been driving me absolutely bonkers. If I'm ever in the same town as 
> any of you, dinner and drinks are on me.

Hmm, I wonder if kmemcheck could help you, but its slow as hell, so not
appropriate for production :(



--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
  2011-06-08 17:02                                                             ` Brad Campbell
@ 2011-06-10  2:52                                                               ` Simon Horman
  -1 siblings, 0 replies; 111+ messages in thread
From: Simon Horman @ 2011-06-10  2:52 UTC (permalink / raw)
  To: Brad Campbell
  Cc: Eric Dumazet, Patrick McHardy, Bart De Schuymer, kvm, linux-mm,
	linux-kernel, netdev, netfilter-devel

On Thu, Jun 09, 2011 at 01:02:13AM +0800, Brad Campbell wrote:
> On 08/06/11 11:59, Eric Dumazet wrote:
> 
> >Well, a bisection definitely should help, but needs a lot of time in
> >your case.
> 
> Yes. compile, test, crash, walk out to the other building to press
> reset, lather, rinse, repeat.
> 
> I need a reset button on the end of a 50M wire, or a hardware watchdog!

Not strictly on-topic, but in situations where I have machines
that either don't have lights-out facilities or have broken ones
I find that network controlled power switches to be very useful.

At one point I would have need an 8000km long wire to the reset switch :-)

^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
@ 2011-06-10  2:52                                                               ` Simon Horman
  0 siblings, 0 replies; 111+ messages in thread
From: Simon Horman @ 2011-06-10  2:52 UTC (permalink / raw)
  To: Brad Campbell
  Cc: Eric Dumazet, Patrick McHardy, Bart De Schuymer, kvm, linux-mm,
	linux-kernel, netdev, netfilter-devel

On Thu, Jun 09, 2011 at 01:02:13AM +0800, Brad Campbell wrote:
> On 08/06/11 11:59, Eric Dumazet wrote:
> 
> >Well, a bisection definitely should help, but needs a lot of time in
> >your case.
> 
> Yes. compile, test, crash, walk out to the other building to press
> reset, lather, rinse, repeat.
> 
> I need a reset button on the end of a 50M wire, or a hardware watchdog!

Not strictly on-topic, but in situations where I have machines
that either don't have lights-out facilities or have broken ones
I find that network controlled power switches to be very useful.

At one point I would have need an 8000km long wire to the reset switch :-)

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
  2011-06-10  2:52                                                               ` Simon Horman
@ 2011-06-10 12:37                                                                 ` Mark Lord
  -1 siblings, 0 replies; 111+ messages in thread
From: Mark Lord @ 2011-06-10 12:37 UTC (permalink / raw)
  To: Simon Horman
  Cc: Brad Campbell, Eric Dumazet, Patrick McHardy, Bart De Schuymer,
	kvm, linux-mm, linux-kernel, netdev, netfilter-devel

On 11-06-09 10:52 PM, Simon Horman wrote:
> On Thu, Jun 09, 2011 at 01:02:13AM +0800, Brad Campbell wrote:
>> On 08/06/11 11:59, Eric Dumazet wrote:
>>
>>> Well, a bisection definitely should help, but needs a lot of time in
>>> your case.
>>
>> Yes. compile, test, crash, walk out to the other building to press
>> reset, lather, rinse, repeat.
>>
>> I need a reset button on the end of a 50M wire, or a hardware watchdog!


Something many of us don't realize is that nearly all Intel chipsets
have a built-in hardware watchdog timer.  This includes chipset for
consumer desktop boards as well as the big iron server stuff.

It's the "i8xx_tco" driver in the kernel enables use of them:

    modprobe i8xx_tco

Cheers

^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
@ 2011-06-10 12:37                                                                 ` Mark Lord
  0 siblings, 0 replies; 111+ messages in thread
From: Mark Lord @ 2011-06-10 12:37 UTC (permalink / raw)
  To: Simon Horman
  Cc: Brad Campbell, Eric Dumazet, Patrick McHardy, Bart De Schuymer,
	kvm, linux-mm, linux-kernel, netdev, netfilter-devel

On 11-06-09 10:52 PM, Simon Horman wrote:
> On Thu, Jun 09, 2011 at 01:02:13AM +0800, Brad Campbell wrote:
>> On 08/06/11 11:59, Eric Dumazet wrote:
>>
>>> Well, a bisection definitely should help, but needs a lot of time in
>>> your case.
>>
>> Yes. compile, test, crash, walk out to the other building to press
>> reset, lather, rinse, repeat.
>>
>> I need a reset button on the end of a 50M wire, or a hardware watchdog!


Something many of us don't realize is that nearly all Intel chipsets
have a built-in hardware watchdog timer.  This includes chipset for
consumer desktop boards as well as the big iron server stuff.

It's the "i8xx_tco" driver in the kernel enables use of them:

    modprobe i8xx_tco

Cheers

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
  2011-06-10 12:37                                                                 ` Mark Lord
@ 2011-06-10 16:43                                                                   ` Henrique de Moraes Holschuh
  -1 siblings, 0 replies; 111+ messages in thread
From: Henrique de Moraes Holschuh @ 2011-06-10 16:43 UTC (permalink / raw)
  To: Mark Lord
  Cc: Simon Horman, Brad Campbell, Eric Dumazet, Patrick McHardy,
	Bart De Schuymer, kvm, linux-mm, linux-kernel, netdev,
	netfilter-devel

On Fri, 10 Jun 2011, Mark Lord wrote:
> Something many of us don't realize is that nearly all Intel chipsets
> have a built-in hardware watchdog timer.  This includes chipset for
> consumer desktop boards as well as the big iron server stuff.
> 
> It's the "i8xx_tco" driver in the kernel enables use of them:

That's the old module name, but yes, it is very useful in desktops and
laptops (when it works).   Server-class hardware will have a baseboard
management unit that can really power-cycle the system instead of just
rebooting.

And test it first before you depend on it triggering at a remote location,
as the firmware might cause the Intel chipset watchdog to actually hang the
box instead of causing a proper reboot (happens on the IBM thinkpad T43, for
example).

-- 
  "One disk to rule them all, One disk to find them. One disk to bring
  them all and in the darkness grind them. In the Land of Redmond
  where the shadows lie." -- The Silicon Valley Tarot
  Henrique Holschuh

^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
@ 2011-06-10 16:43                                                                   ` Henrique de Moraes Holschuh
  0 siblings, 0 replies; 111+ messages in thread
From: Henrique de Moraes Holschuh @ 2011-06-10 16:43 UTC (permalink / raw)
  To: Mark Lord
  Cc: Simon Horman, Brad Campbell, Eric Dumazet, Patrick McHardy,
	Bart De Schuymer, kvm, linux-mm, linux-kernel, netdev,
	netfilter-devel

On Fri, 10 Jun 2011, Mark Lord wrote:
> Something many of us don't realize is that nearly all Intel chipsets
> have a built-in hardware watchdog timer.  This includes chipset for
> consumer desktop boards as well as the big iron server stuff.
> 
> It's the "i8xx_tco" driver in the kernel enables use of them:

That's the old module name, but yes, it is very useful in desktops and
laptops (when it works).   Server-class hardware will have a baseboard
management unit that can really power-cycle the system instead of just
rebooting.

And test it first before you depend on it triggering at a remote location,
as the firmware might cause the Intel chipset watchdog to actually hang the
box instead of causing a proper reboot (happens on the IBM thinkpad T43, for
example).

-- 
  "One disk to rule them all, One disk to find them. One disk to bring
  them all and in the darkness grind them. In the Land of Redmond
  where the shadows lie." -- The Silicon Valley Tarot
  Henrique Holschuh

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
  2011-06-10  2:52                                                               ` Simon Horman
@ 2011-06-12 15:38                                                                 ` Avi Kivity
  -1 siblings, 0 replies; 111+ messages in thread
From: Avi Kivity @ 2011-06-12 15:38 UTC (permalink / raw)
  To: Simon Horman
  Cc: Brad Campbell, Eric Dumazet, Patrick McHardy, Bart De Schuymer,
	kvm, linux-mm, linux-kernel, netdev, netfilter-devel

On 06/10/2011 05:52 AM, Simon Horman wrote:
> At one point I would have need an 8000km long wire to the reset switch :-)

Even more off-topic, there has been a case when a 200,000,000 km long 
wire to the reset button was needed.  IIRC they got away with a watchdog.

-- 
error compiling committee.c: too many arguments to function


^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
@ 2011-06-12 15:38                                                                 ` Avi Kivity
  0 siblings, 0 replies; 111+ messages in thread
From: Avi Kivity @ 2011-06-12 15:38 UTC (permalink / raw)
  To: Simon Horman
  Cc: Brad Campbell, Eric Dumazet, Patrick McHardy, Bart De Schuymer,
	kvm, linux-mm, linux-kernel, netdev, netfilter-devel

On 06/10/2011 05:52 AM, Simon Horman wrote:
> At one point I would have need an 8000km long wire to the reset switch :-)

Even more off-topic, there has been a case when a 200,000,000 km long 
wire to the reset button was needed.  IIRC they got away with a watchdog.

-- 
error compiling committee.c: too many arguments to function

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
  2011-06-07 13:37                                             ` Eric Dumazet
  2011-06-07 15:15                                               ` Brad Campbell
@ 2011-08-20 13:16                                               ` Brad Campbell
  2011-08-22  6:36                                                 ` Avi Kivity
  1 sibling, 1 reply; 111+ messages in thread
From: Brad Campbell @ 2011-08-20 13:16 UTC (permalink / raw)
  To: Eric Dumazet; +Cc: Avi Kivity, CaT, Borislav Petkov, linux-kernel, kvm, netdev

On 07/06/11 21:37, Eric Dumazet wrote:
> Le mardi 07 juin 2011 à 21:27 +0800, Brad Campbell a écrit :
>> On 07/06/11 04:22, Eric Dumazet wrote:
>>
>>> Could you please try latest linux-2.6 tree ?
>>>
>>> We fixed many networking bugs that could explain your crash.
>>>
>>>
>>>
>>>
>>
>> No good I'm afraid.
>>
>> [  543.040056]
>> =============================================================================
>> [  543.040136] BUG ip_dst_cache: Padding overwritten.
>> 0xffff8803e4217ffe-0xffff8803e4217fff
>> [  543.040194]
>
> Thats pretty strange : These are the last two bytes of a page, set to
> 0x0000 (a 16 bit value)
>
> There is no way a dst field could actually sit on this location (its a
> padding), since a dst is a bit less than 256 bytes (0xe8), and each
> entry is aligned on a 64byte address.
>
> grep dst /proc/slabinfo
>
> ip_dst_cache       32823  62944    256   32    2 : tunables    0    0
> 0 : slabdata   1967   1967      0
>
> sizeof(struct rtable)=0xe8
>
>
>> -----------------------------------------------------------------------------
>> [  543.040198]
>> [  543.040298] INFO: Slab 0xffffea000d9e74d0 objects=25 used=25 fp=0x
>>          (null) flags=0x8000000000004081
>> [  543.040364] Pid: 4576, comm: kworker/1:2 Not tainted 3.0.0-rc2 #1
>> [  543.040415] Call Trace:
>> [  543.040472]  [<ffffffff810b9c1d>] ? slab_err+0xad/0xd0
>> [  543.040528]  [<ffffffff8102e034>] ? check_preempt_wakeup+0xa4/0x160
>> [  543.040595]  [<ffffffff810ba206>] ? slab_pad_check+0x126/0x170
>> [  543.040650]  [<ffffffff8133045b>] ? dst_destroy+0x8b/0x110
>> [  543.040701]  [<ffffffff810ba29a>] ? check_slab+0x4a/0xc0
>> [  543.040753]  [<ffffffff810baf2d>] ? free_debug_processing+0x2d/0x250
>> [  543.040808]  [<ffffffff810bb27b>] ? __slab_free+0x12b/0x140
>> [  543.040862]  [<ffffffff810bbe99>] ? kmem_cache_free+0x99/0xa0
>> [  543.040915]  [<ffffffff8133045b>] ? dst_destroy+0x8b/0x110
>> [  543.040967]  [<ffffffff813307f6>] ? dst_gc_task+0x196/0x1f0
>> [  543.041021]  [<ffffffff8104e954>] ? queue_delayed_work_on+0x154/0x160
>> [  543.041081]  [<ffffffff813066fe>] ? do_dbs_timer+0x20e/0x3d0
>> [  543.041133]  [<ffffffff81330660>] ? dst_alloc+0x180/0x180
>> [  543.041187]  [<ffffffff8104f28b>] ? process_one_work+0xfb/0x3b0
>> [  543.041242]  [<ffffffff8104f964>] ? worker_thread+0x144/0x3d0
>> [  543.041296]  [<ffffffff8102cc10>] ? __wake_up_common+0x50/0x80
>> [  543.041678]  [<ffffffff8104f820>] ? rescuer_thread+0x2e0/0x2e0
>> [  543.041729]  [<ffffffff8104f820>] ? rescuer_thread+0x2e0/0x2e0
>> [  543.041782]  [<ffffffff81053436>] ? kthread+0x96/0xa0
>> [  543.041835]  [<ffffffff813e1d14>] ? kernel_thread_helper+0x4/0x10
>> [  543.041890]  [<ffffffff810533a0>] ? kthread_worker_fn+0x120/0x120
>> [  543.041944]  [<ffffffff813e1d10>] ? gs_change+0xb/0xb
>> [  543.041993]  Padding 0xffff8803e4217f40:  5a 5a 5a 5a 5a 5a 5a 5a 5a
>> 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
>> [  543.042718]  Padding 0xffff8803e4217f50:  5a 5a 5a 5a 5a 5a 5a 5a 5a
>> 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
>> [  543.043433]  Padding 0xffff8803e4217f60:  5a 5a 5a 5a 5a 5a 5a 5a 5a
>> 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
>> [  543.044155]  Padding 0xffff8803e4217f70:  5a 5a 5a 5a 5a 5a 5a 5a 5a
>> 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
>> [  543.044866]  Padding 0xffff8803e4217f80:  5a 5a 5a 5a 5a 5a 5a 5a 5a
>> 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
>> [  543.045590]  Padding 0xffff8803e4217f90:  5a 5a 5a 5a 5a 5a 5a 5a 5a
>> 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
>> [  543.046311]  Padding 0xffff8803e4217fa0:  5a 5a 5a 5a 5a 5a 5a 5a 5a
>> 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
>> [  543.047034]  Padding 0xffff8803e4217fb0:  5a 5a 5a 5a 5a 5a 5a 5a 5a
>> 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
>> [  543.047755]  Padding 0xffff8803e4217fc0:  5a 5a 5a 5a 5a 5a 5a 5a 5a
>> 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
>> [  543.048474]  Padding 0xffff8803e4217fd0:  5a 5a 5a 5a 5a 5a 5a 5a 5a
>> 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
>> [  543.049203]  Padding 0xffff8803e4217fe0:  5a 5a 5a 5a 5a 5a 5a 5a 5a
>> 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
>> [  543.049909]  Padding 0xffff8803e4217ff0:  5a 5a 5a 5a 5a 5a 5a 5a 5a
>> 5a 5a 5a 5a 5a 00 00 ZZZZZZZZZZZZZZ..
>> [  543.050021] FIX ip_dst_cache: Restoring
>> 0xffff8803e4217f40-0xffff8803e4217fff=0x5a
>> [  543.050021]
>>
>> Dropped -mm, Hugh and Andrea from CC as this does not appear to be mm or
>> ksm related.
>>
>> I'll pare down the firewall and see if I can make it break easier with a
>> smaller test set.
>
> Hmm, not sure now :(
>
> Could you reproduce another bug please ?

I know this is an old one, but I recently purchased a second system to 
allow me to test and bisect this off-line (the live system is too much 
of a headache to bisect on).

brad@test:/raid10/src/linux-2.6$ git bisect log
git bisect start
# good: [9fe6206f400646a2322096b56c59891d530e8d51] Linux 2.6.35
git bisect good 9fe6206f400646a2322096b56c59891d530e8d51
# bad: [da5cabf80e2433131bf0ed8993abc0f7ea618c73] Linux 2.6.36-rc1
git bisect bad da5cabf80e2433131bf0ed8993abc0f7ea618c73
# bad: [0f477dd0851bdcee82923da66a7fc4a44cb1bc3d] Merge branch 
'x86-cpu-for-linus' of 
git://git.kernel.org/pub/scm/linux/kernel/git/tip/linux-2.6-tip
git bisect bad 0f477dd0851bdcee82923da66a7fc4a44cb1bc3d
# bad: [3ff1c25927e3af61c6bf0e4ed959504058ae4565] phy/marvell: add 
88ec048 support
git bisect bad 3ff1c25927e3af61c6bf0e4ed959504058ae4565
# good: [05318bc905467237d4aa68a701f6e92a2b332218] Merge branch 'master' 
of git://git.kernel.org/pub/scm/linux/kernel/git/linville/wireless-next-2.6
git bisect good 05318bc905467237d4aa68a701f6e92a2b332218
# bad: [2ba13ed678775195e8255b4e503c59d48b615bd8] Bluetooth: Remove 
check for supported mode
git bisect bad 2ba13ed678775195e8255b4e503c59d48b615bd8
# bad: [1e2cfeef060fa0270f9a2d66b1218c12c05062e0] Revert "tc35815: fix 
iomap leak"
git bisect bad 1e2cfeef060fa0270f9a2d66b1218c12c05062e0
# bad: [d9bed6bbd4f2a0120c93fed68605950651e1f225] isdn/gigaset: remove 
EXPERIMENTAL tag from GIGASET_CAPI
git bisect bad d9bed6bbd4f2a0120c93fed68605950651e1f225
# bad: [d117b6665847084cfe8a44b870f771153e18991d] fealnx: Use the 
instance of net_device_stats from net_device.
git bisect bad d117b6665847084cfe8a44b870f771153e18991d
# bad: [e490c1defec4236a6a131fe2d13bf7ba787c02f8] Merge branch 'master' 
of git://git.kernel.org/pub/scm/linux/kernel/git/kaber/nf-next-2.6
git bisect bad e490c1defec4236a6a131fe2d13bf7ba787c02f8
# bad: [0a17d8c744e44617a3c22e7af68b4c5c9c1c5dba] ixgbe: use NETIF_F_LRO
git bisect bad 0a17d8c744e44617a3c22e7af68b4c5c9c1c5dba
# bad: [ede3ef0d940ef052466f42c849390b23c6859abc] igb: fix PHY config 
access on 82580
git bisect bad ede3ef0d940ef052466f42c849390b23c6859abc
# good: [ee3cb6295144b0adfa75ccaca307643a6998b1e2] be2net: changes to 
properly provide phy details
git bisect good ee3cb6295144b0adfa75ccaca307643a6998b1e2
# bad: [7475271004b66e9c22e1bb28f240a38c5d6fe76e] x86: Drop 
CONFIG_MCORE2 check around setting of NET_IP_ALIGN
git bisect bad 7475271004b66e9c22e1bb28f240a38c5d6fe76e
brad@test:/raid10/src/linux-2.6$ git bisect log
git bisect start
# good: [9fe6206f400646a2322096b56c59891d530e8d51] Linux 2.6.35
git bisect good 9fe6206f400646a2322096b56c59891d530e8d51
# bad: [da5cabf80e2433131bf0ed8993abc0f7ea618c73] Linux 2.6.36-rc1
git bisect bad da5cabf80e2433131bf0ed8993abc0f7ea618c73
# bad: [0f477dd0851bdcee82923da66a7fc4a44cb1bc3d] Merge branch 
'x86-cpu-for-linus' of 
git://git.kernel.org/pub/scm/linux/kernel/git/tip/linux-2.6-tip
git bisect bad 0f477dd0851bdcee82923da66a7fc4a44cb1bc3d
# bad: [3ff1c25927e3af61c6bf0e4ed959504058ae4565] phy/marvell: add 
88ec048 support
git bisect bad 3ff1c25927e3af61c6bf0e4ed959504058ae4565
# good: [05318bc905467237d4aa68a701f6e92a2b332218] Merge branch 'master' 
of git://git.kernel.org/pub/scm/linux/kernel/git/linville/wireless-next-2.6
git bisect good 05318bc905467237d4aa68a701f6e92a2b332218
# bad: [2ba13ed678775195e8255b4e503c59d48b615bd8] Bluetooth: Remove 
check for supported mode
git bisect bad 2ba13ed678775195e8255b4e503c59d48b615bd8
# bad: [1e2cfeef060fa0270f9a2d66b1218c12c05062e0] Revert "tc35815: fix 
iomap leak"
git bisect bad 1e2cfeef060fa0270f9a2d66b1218c12c05062e0
# bad: [d9bed6bbd4f2a0120c93fed68605950651e1f225] isdn/gigaset: remove 
EXPERIMENTAL tag from GIGASET_CAPI
git bisect bad d9bed6bbd4f2a0120c93fed68605950651e1f225
# bad: [d117b6665847084cfe8a44b870f771153e18991d] fealnx: Use the 
instance of net_device_stats from net_device.
git bisect bad d117b6665847084cfe8a44b870f771153e18991d
# bad: [e490c1defec4236a6a131fe2d13bf7ba787c02f8] Merge branch 'master' 
of git://git.kernel.org/pub/scm/linux/kernel/git/kaber/nf-next-2.6
git bisect bad e490c1defec4236a6a131fe2d13bf7ba787c02f8
# bad: [0a17d8c744e44617a3c22e7af68b4c5c9c1c5dba] ixgbe: use NETIF_F_LRO
git bisect bad 0a17d8c744e44617a3c22e7af68b4c5c9c1c5dba
# bad: [ede3ef0d940ef052466f42c849390b23c6859abc] igb: fix PHY config 
access on 82580
git bisect bad ede3ef0d940ef052466f42c849390b23c6859abc
# good: [ee3cb6295144b0adfa75ccaca307643a6998b1e2] be2net: changes to 
properly provide phy details
git bisect good ee3cb6295144b0adfa75ccaca307643a6998b1e2
# bad: [7475271004b66e9c22e1bb28f240a38c5d6fe76e] x86: Drop 
CONFIG_MCORE2 check around setting of NET_IP_ALIGN
git bisect bad 7475271004b66e9c22e1bb28f240a38c5d6fe76e
brad@test:/raid10/src/linux-2.6$ git bisect good
7475271004b66e9c22e1bb28f240a38c5d6fe76e is the first bad commit
commit 7475271004b66e9c22e1bb28f240a38c5d6fe76e
Author: Alexander Duyck <alexander.h.duyck@intel.com>
Date:   Thu Jul 1 13:28:27 2010 +0000

     x86: Drop CONFIG_MCORE2 check around setting of NET_IP_ALIGN

     This patch removes the CONFIG_MCORE2 check from around 
NET_IP_ALIGN.  It is
     based on a suggestion from Andi Kleen.  The assumption is that 
there are
     not any x86 cores where unaligned access is really slow, and this 
change
     would allow for a performance improvement to still exist on 
configurations
     that are not necessarily optimized for Core 2.

     Cc: Andi Kleen <ak@linux.intel.com>
     Cc: Thomas Gleixner <tglx@linutronix.de>
     Cc: Ingo Molnar <mingo@redhat.com>
     Cc: "H. Peter Anvin" <hpa@zytor.com>
     Cc: x86@kernel.org
     Signed-off-by: Alexander Duyck <alexander.h.duyck@intel.com>
     Signed-off-by: Jeff Kirsher <jeffrey.t.kirsher@intel.com>
     Acked-by: H. Peter Anvin <hpa@zytor.com>
     Signed-off-by: David S. Miller <davem@davemloft.net>

:040000 040000 5a15867789080a2f67a74b17c4422f85b7a9fb4a 
b98769348bd765731ca3ff03b33764257e23226c M	arch

I can confirm this bug exists in the 3.0 kernel, however I'm unable to 
reproduce it on todays git.

So anyone using netfilter, kvm and bridge on kernels between 2.6.36-rc1 
and 3.0 may hit this bug, but it looks like it is fixed in the current 
3.1-rc kernels.


^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
  2011-08-20 13:16                                               ` Brad Campbell
@ 2011-08-22  6:36                                                 ` Avi Kivity
  2011-08-22  6:45                                                   ` Eric Dumazet
  0 siblings, 1 reply; 111+ messages in thread
From: Avi Kivity @ 2011-08-22  6:36 UTC (permalink / raw)
  To: Brad Campbell
  Cc: Eric Dumazet, CaT, Borislav Petkov, linux-kernel, kvm, netdev

On 08/20/2011 04:16 PM, Brad Campbell wrote:
> Author: Alexander Duyck <alexander.h.duyck@intel.com>
> Date:   Thu Jul 1 13:28:27 2010 +0000
>
>     x86: Drop CONFIG_MCORE2 check around setting of NET_IP_ALIGN
>
>     This patch removes the CONFIG_MCORE2 check from around 
> NET_IP_ALIGN.  It is
>     based on a suggestion from Andi Kleen.  The assumption is that 
> there are
>     not any x86 cores where unaligned access is really slow, and this 
> change
>     would allow for a performance improvement to still exist on 
> configurations
>     that are not necessarily optimized for Core 2.
>
>     Cc: Andi Kleen <ak@linux.intel.com>
>     Cc: Thomas Gleixner <tglx@linutronix.de>
>     Cc: Ingo Molnar <mingo@redhat.com>
>     Cc: "H. Peter Anvin" <hpa@zytor.com>
>     Cc: x86@kernel.org
>     Signed-off-by: Alexander Duyck <alexander.h.duyck@intel.com>
>     Signed-off-by: Jeff Kirsher <jeffrey.t.kirsher@intel.com>
>     Acked-by: H. Peter Anvin <hpa@zytor.com>
>     Signed-off-by: David S. Miller <davem@davemloft.net>
>
> :040000 040000 5a15867789080a2f67a74b17c4422f85b7a9fb4a 
> b98769348bd765731ca3ff03b33764257e23226c M    arch
>
> I can confirm this bug exists in the 3.0 kernel, however I'm unable to 
> reproduce it on todays git.
>
> So anyone using netfilter, kvm and bridge on kernels between 
> 2.6.36-rc1 and 3.0 may hit this bug, but it looks like it is fixed in 
> the current 3.1-rc kernels.
>

Thanks for this effort.  I don't think this patch is buggy in itself, it 
merely exposed another bug which was fixed later on.

-- 
I have a truly marvellous patch that fixes the bug which this
signature is too narrow to contain.


^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
  2011-08-22  6:36                                                 ` Avi Kivity
@ 2011-08-22  6:45                                                   ` Eric Dumazet
  2011-08-22 11:45                                                     ` Brad Campbell
  0 siblings, 1 reply; 111+ messages in thread
From: Eric Dumazet @ 2011-08-22  6:45 UTC (permalink / raw)
  To: Avi Kivity; +Cc: Brad Campbell, CaT, Borislav Petkov, linux-kernel, kvm, netdev

Le lundi 22 août 2011 à 09:36 +0300, Avi Kivity a écrit :
> On 08/20/2011 04:16 PM, Brad Campbell wrote:
> > Author: Alexander Duyck <alexander.h.duyck@intel.com>
> > Date:   Thu Jul 1 13:28:27 2010 +0000
> >
> >     x86: Drop CONFIG_MCORE2 check around setting of NET_IP_ALIGN
> >
> >     This patch removes the CONFIG_MCORE2 check from around 
> > NET_IP_ALIGN.  It is
> >     based on a suggestion from Andi Kleen.  The assumption is that 
> > there are
> >     not any x86 cores where unaligned access is really slow, and this 
> > change
> >     would allow for a performance improvement to still exist on 
> > configurations
> >     that are not necessarily optimized for Core 2.
> >
> >     Cc: Andi Kleen <ak@linux.intel.com>
> >     Cc: Thomas Gleixner <tglx@linutronix.de>
> >     Cc: Ingo Molnar <mingo@redhat.com>
> >     Cc: "H. Peter Anvin" <hpa@zytor.com>
> >     Cc: x86@kernel.org
> >     Signed-off-by: Alexander Duyck <alexander.h.duyck@intel.com>
> >     Signed-off-by: Jeff Kirsher <jeffrey.t.kirsher@intel.com>
> >     Acked-by: H. Peter Anvin <hpa@zytor.com>
> >     Signed-off-by: David S. Miller <davem@davemloft.net>
> >
> > :040000 040000 5a15867789080a2f67a74b17c4422f85b7a9fb4a 
> > b98769348bd765731ca3ff03b33764257e23226c M    arch
> >
> > I can confirm this bug exists in the 3.0 kernel, however I'm unable to 
> > reproduce it on todays git.
> >
> > So anyone using netfilter, kvm and bridge on kernels between 
> > 2.6.36-rc1 and 3.0 may hit this bug, but it looks like it is fixed in 
> > the current 3.1-rc kernels.
> >
> 
> Thanks for this effort.  I don't think this patch is buggy in itself, it 
> merely exposed another bug which was fixed later on.
> 

Some piece of hardware has a 2-byte offset requirement, and driver
incorrectly assumed NET_IP_ALIGN was 2 on x86.

Brad, could you post your config (lsmod, dmesg) again ?

tg3.c code for example uses a private value, not related to NET_IP_ALIGN

#define TG3_RAW_IP_ALIGN 2




^ permalink raw reply	[flat|nested] 111+ messages in thread

* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
  2011-08-22  6:45                                                   ` Eric Dumazet
@ 2011-08-22 11:45                                                     ` Brad Campbell
  0 siblings, 0 replies; 111+ messages in thread
From: Brad Campbell @ 2011-08-22 11:45 UTC (permalink / raw)
  To: Eric Dumazet; +Cc: Avi Kivity, CaT, Borislav Petkov, linux-kernel, kvm, netdev

On 22/08/11 14:45, Eric Dumazet wrote:
> Le lundi 22 août 2011 à 09:36 +0300, Avi Kivity a écrit :
>> On 08/20/2011 04:16 PM, Brad Campbell wrote:
>>> Author: Alexander Duyck<alexander.h.duyck@intel.com>
>>> Date:   Thu Jul 1 13:28:27 2010 +0000
>>>
>>>      x86: Drop CONFIG_MCORE2 check around setting of NET_IP_ALIGN
>>>
>>>      This patch removes the CONFIG_MCORE2 check from around
>>> NET_IP_ALIGN.  It is
>>>      based on a suggestion from Andi Kleen.  The assumption is that
>>> there are
>>>      not any x86 cores where unaligned access is really slow, and this
>>> change
>>>      would allow for a performance improvement to still exist on
>>> configurations
>>>      that are not necessarily optimized for Core 2.
>>>
>>>      Cc: Andi Kleen<ak@linux.intel.com>
>>>      Cc: Thomas Gleixner<tglx@linutronix.de>
>>>      Cc: Ingo Molnar<mingo@redhat.com>
>>>      Cc: "H. Peter Anvin"<hpa@zytor.com>
>>>      Cc: x86@kernel.org
>>>      Signed-off-by: Alexander Duyck<alexander.h.duyck@intel.com>
>>>      Signed-off-by: Jeff Kirsher<jeffrey.t.kirsher@intel.com>
>>>      Acked-by: H. Peter Anvin<hpa@zytor.com>
>>>      Signed-off-by: David S. Miller<davem@davemloft.net>
>>>
>>> :040000 040000 5a15867789080a2f67a74b17c4422f85b7a9fb4a
>>> b98769348bd765731ca3ff03b33764257e23226c M    arch
>>>
>>> I can confirm this bug exists in the 3.0 kernel, however I'm unable to
>>> reproduce it on todays git.
>>>
>>> So anyone using netfilter, kvm and bridge on kernels between
>>> 2.6.36-rc1 and 3.0 may hit this bug, but it looks like it is fixed in
>>> the current 3.1-rc kernels.
>>>
>>
>> Thanks for this effort.  I don't think this patch is buggy in itself, it
>> merely exposed another bug which was fixed later on.
>>
>
> Some piece of hardware has a 2-byte offset requirement, and driver
> incorrectly assumed NET_IP_ALIGN was 2 on x86.
>
> Brad, could you post your config (lsmod, dmesg) again ?
>
> tg3.c code for example uses a private value, not related to NET_IP_ALIGN
>
> #define TG3_RAW_IP_ALIGN 2

Both sets of hardware were using Marvell 88e1116 chips (the r8169 driver).

Interestingly on the hardware I just used to reproduce the fault the 
network path is VM1 -> br1 -> eth0 -> DNAT -> br1 -> VM2

On the machine that originally exhibited the fault the network path was
VM1 -> br1 -> ppp0 -> DNAT -> br1 -> VM2

So on the original machine the packet never even goes near a physical 
interface.

The problem only exhibited itself if I tried to use the address of eth0 
or ppp0 to get to VM2 from VM1. If I went dirctly from VM1 to VM2 using 
local addresses and never traversed netfilter it never paniced.

I also agree the patch I identified simply unmasked an underlying issue, 
and of course on anything later than the linus 3.0 kernel I can't 
reproduce it now anyway.

It's nice to know its fixed, but it'd be nice to know *what* fixed it 
rather than it just went away after some code re-factoring.

I did have a perverse thought about doing a bisect between 3.0.0 and 
3.1-rc2 to see precisely *what* fixed the problem, but the thought of 
another 7 hours of crash and burn is not overly appealing.

[    0.000000] Linux version 3.0.0 (brad@srv) (gcc version 4.4.5 (Debian 
4.4.5-8) ) #5 SMP Mon Jul 25 22:43:34 WST 2011
[    0.000000] Command line: BOOT_IMAGE=/bzImage 
root=UUID=9162858d-df5e-43d6-8331-6d2b65082b4c ro panic=5 
netconsole=6666@192.168.2.1/eth0,5050@192.168.2.87/00:16:cb:a7:dd:d1
[    0.000000] BIOS-provided physical RAM map:
[    0.000000]  BIOS-e820: 0000000000000000 - 000000000009c400 (usable)
[    0.000000]  BIOS-e820: 000000000009c400 - 00000000000a0000 (reserved)
[    0.000000]  BIOS-e820: 00000000000e4000 - 0000000000100000 (reserved)
[    0.000000]  BIOS-e820: 0000000000100000 - 00000000dbf80000 (usable)
[    0.000000]  BIOS-e820: 00000000dbf80000 - 00000000dffa0000 (reserved)
[    0.000000]  BIOS-e820: 00000000dffa0000 - 00000000dffb0000 (ACPI data)
[    0.000000]  BIOS-e820: 00000000dffb0000 - 00000000dffe0000 (ACPI NVS)
[    0.000000]  BIOS-e820: 00000000dffe0000 - 00000000e0000000 (reserved)
[    0.000000]  BIOS-e820: 00000000fff00000 - 0000000100000000 (reserved)
[    0.000000]  BIOS-e820: 0000000100000000 - 0000000420000000 (usable)
[    0.000000] NX (Execute Disable) protection: active
[    0.000000] DMI present.
[    0.000000] DMI: To Be Filled By O.E.M. To Be Filled By O.E.M./880G 
Extreme3, BIOS P1.70 06/21/2010
[    0.000000] e820 update range: 0000000000000000 - 0000000000010000 
(usable) ==> (reserved)
[    0.000000] e820 remove range: 00000000000a0000 - 0000000000100000 
(usable)
[    0.000000] No AGP bridge found
[    0.000000] last_pfn = 0x420000 max_arch_pfn = 0x400000000
[    0.000000] MTRR default type: uncachable
[    0.000000] MTRR fixed ranges enabled:
[    0.000000]   00000-9FFFF write-back
[    0.000000]   A0000-EFFFF uncachable
[    0.000000]   F0000-FFFFF write-protect
[    0.000000] MTRR variable ranges enabled:
[    0.000000]   0 base 000000000000 mask FFFF80000000 write-back
[    0.000000]   1 base 000080000000 mask FFFFC0000000 write-back
[    0.000000]   2 base 0000C0000000 mask FFFFE0000000 write-back
[    0.000000]   3 disabled
[    0.000000]   4 disabled
[    0.000000]   5 disabled
[    0.000000]   6 disabled
[    0.000000]   7 disabled
[    0.000000] TOM2: 0000000420000000 aka 16896M
[    0.000000] x86 PAT enabled: cpu 0, old 0x7040600070406, new 
0x7010600070106
[    0.000000] e820 update range: 00000000e0000000 - 0000000100000000 
(usable) ==> (reserved)
[    0.000000] last_pfn = 0xdbf80 max_arch_pfn = 0x400000000
[    0.000000] initial memory mapped : 0 - 20000000
[    0.000000] Base memory trampoline at [ffff88000009a000] 9a000 size 8192
[    0.000000] Using GB pages for direct mapping
[    0.000000] init_memory_mapping: 0000000000000000-00000000dbf80000
[    0.000000]  0000000000 - 00c0000000 page 1G
[    0.000000]  00c0000000 - 00dbe00000 page 2M
[    0.000000]  00dbe00000 - 00dbf80000 page 4k
[    0.000000] kernel direct mapping tables up to dbf80000 @ 
dbf7d000-dbf80000
[    0.000000] init_memory_mapping: 0000000100000000-0000000420000000
[    0.000000]  0100000000 - 0400000000 page 1G
[    0.000000]  0400000000 - 0420000000 page 2M
[    0.000000] kernel direct mapping tables up to 420000000 @ 
41fffe000-420000000
[    0.000000] RAMDISK: 37cd8000 - 37ff0000
[    0.000000] ACPI: RSDP 00000000000fa940 00014 (v00 ACPIAM)
[    0.000000] ACPI: RSDT 00000000dffa0000 00044 (v01 062110 RSDT2249 
20100621 MSFT 00000097)
[    0.000000] ACPI: FACP 00000000dffa0200 00084 (v01 A_M_I  OEMFACP 
12000601 MSFT 00000097)
[    0.000000] ACPI: DSDT 00000000dffa0450 07D9F (v01  AS368 AS368164 
00000164 INTL 20051117)
[    0.000000] ACPI: FACS 00000000dffb0000 00040
[    0.000000] ACPI: APIC 00000000dffa0390 0007C (v01 062110 APIC2249 
20100621 MSFT 00000097)
[    0.000000] ACPI: MCFG 00000000dffa0410 0003C (v01 062110 OEMMCFG 
20100621 MSFT 00000097)
[    0.000000] ACPI: OEMB 00000000dffb0040 00072 (v01 062110 OEMB2249 
20100621 MSFT 00000097)
[    0.000000] ACPI: SRAT 00000000dffaa450 00108 (v03 AMD    FAM_F_10 
00000002 AMD  00000001)
[    0.000000] ACPI: AAFT 00000000dffaa560 00027 (v01 062110 OEMAAFT 
20100621 MSFT 00000097)
[    0.000000] ACPI: HPET 00000000dffaa590 00038 (v01 062110 OEMHPET 
20100621 MSFT 00000097)
[    0.000000] ACPI: SSDT 00000000dffaa5d0 00DA4 (v01 A M I  POWERNOW 
00000001 AMD  00000001)
[    0.000000] ACPI: Local APIC address 0xfee00000
[    0.000000]  [ffffea0000000000-ffffea000e7fffff] PMD -> 
[ffff88040f600000-ffff88041d7fffff] on node 0
[    0.000000] Zone PFN ranges:
[    0.000000]   DMA      0x00000010 -> 0x00001000
[    0.000000]   DMA32    0x00001000 -> 0x00100000
[    0.000000]   Normal   0x00100000 -> 0x00420000
[    0.000000] Movable zone start PFN for each node
[    0.000000] early_node_map[3] active PFN ranges
[    0.000000]     0: 0x00000010 -> 0x0000009c
[    0.000000]     0: 0x00000100 -> 0x000dbf80
[    0.000000]     0: 0x00100000 -> 0x00420000
[    0.000000] On node 0 totalpages: 4177676
[    0.000000]   DMA zone: 56 pages used for memmap
[    0.000000]   DMA zone: 2 pages reserved
[    0.000000]   DMA zone: 3922 pages, LIFO batch:0
[    0.000000]   DMA32 zone: 14280 pages used for memmap
[    0.000000]   DMA32 zone: 882616 pages, LIFO batch:31
[    0.000000]   Normal zone: 44800 pages used for memmap
[    0.000000]   Normal zone: 3232000 pages, LIFO batch:31
[    0.000000] ACPI: PM-Timer IO Port: 0x808
[    0.000000] ACPI: Local APIC address 0xfee00000
[    0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x02] lapic_id[0x01] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x03] lapic_id[0x02] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x04] lapic_id[0x03] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x05] lapic_id[0x04] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x06] lapic_id[0x05] enabled)
[    0.000000] ACPI: IOAPIC (id[0x06] address[0xfec00000] gsi_base[0])
[    0.000000] IOAPIC[0]: apic_id 6, version 33, address 0xfec00000, GSI 
0-23
[    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
[    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 low level)
[    0.000000] ACPI: IRQ0 used by override.
[    0.000000] ACPI: IRQ2 used by override.
[    0.000000] ACPI: IRQ9 used by override.
[    0.000000] Using ACPI (MADT) for SMP configuration information
[    0.000000] ACPI: HPET id: 0x8300 base: 0xfed00000
[    0.000000] SMP: Allowing 6 CPUs, 0 hotplug CPUs
[    0.000000] nr_irqs_gsi: 40
[    0.000000] Allocating PCI resources starting at e0000000 (gap: 
e0000000:1ff00000)
[    0.000000] setup_percpu: NR_CPUS:8 nr_cpumask_bits:8 nr_cpu_ids:6 
nr_node_ids:1
[    0.000000] PERCPU: Embedded 24 pages/cpu @ffff88041fc00000 s69248 
r8192 d20864 u262144
[    0.000000] pcpu-alloc: s69248 r8192 d20864 u262144 alloc=1*2097152
[    0.000000] pcpu-alloc: [0] 0 1 2 3 4 5 - -
[    0.000000] Built 1 zonelists in Zone order, mobility grouping on. 
Total pages: 4118538
[    0.000000] Kernel command line: BOOT_IMAGE=/bzImage 
root=UUID=9162858d-df5e-43d6-8331-6d2b65082b4c ro panic=5 
netconsole=6666@192.168.2.1/eth0,5050@192.168.2.87/00:16:cb:a7:dd:d1
[    0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes)
[    0.000000] Dentry cache hash table entries: 2097152 (order: 12, 
16777216 bytes)
[    0.000000] Inode-cache hash table entries: 1048576 (order: 11, 
8388608 bytes)
[    0.000000] Checking aperture...
[    0.000000] No AGP bridge found
[    0.000000] Node 0: aperture @ d21e000000 size 32 MB
[    0.000000] Aperture beyond 4GB. Ignoring.
[    0.000000] Your BIOS doesn't leave a aperture memory hole
[    0.000000] Please enable the IOMMU option in the BIOS setup
[    0.000000] This costs you 64 MB of RAM
[    0.000000] Mapping aperture over 65536 KB of RAM @ d0000000
[    0.000000] Memory: 16310976k/17301504k available (4036k kernel code, 
590800k absent, 399728k reserved, 2027k data, 468k init)
[    0.000000] SLUB: Genslabs=15, HWalign=64, Order=0-3, MinObjects=0, 
CPUs=6, Nodes=1
[    0.000000] Hierarchical RCU implementation.
[    0.000000] NR_IRQS:512
[    0.000000] spurious 8259A interrupt: IRQ7.
[    0.000000] Console: colour VGA+ 80x25
[    0.000000] console [tty0] enabled
[    0.000000] hpet clockevent registered
[    0.000000] Fast TSC calibration using PIT
[    0.000000] Detected 3213.157 MHz processor.
[    0.000003] Calibrating delay loop (skipped), value calculated using 
timer frequency.. 6426.31 BogoMIPS (lpj=32131570)
[    0.000064] pid_max: default: 32768 minimum: 301
[    0.000126] Mount-cache hash table entries: 256
[    0.000267] tseg: 0000000000
[    0.000272] CPU: Physical Processor ID: 0
[    0.000304] CPU: Processor Core ID: 0
[    0.000334] mce: CPU supports 6 MCE banks
[    0.000420] Freeing SMP alternatives: 20k freed
[    0.000456] ACPI: Core revision 20110413
[    0.012256] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
[    0.112297] CPU0: AMD Phenom(tm) II X6 1090T Processor stepping 00
[    0.120000] Performance Events: AMD PMU driver.
[    0.120000] ... version:                0
[    0.120000] ... bit width:              48
[    0.120000] ... generic registers:      4
[    0.120000] ... value mask:             0000ffffffffffff
[    0.120000] ... max period:             00007fffffffffff
[    0.120000] ... fixed-purpose events:   0
[    0.120000] ... event mask:             000000000000000f
[    0.120000] Booting Node   0, Processors  #1
[    0.120000] smpboot cpu 1: start_ip = 9a000
[    0.280028]  #2
[    0.280056] smpboot cpu 2: start_ip = 9a000
[    0.450026]  #3
[    0.450054] smpboot cpu 3: start_ip = 9a000
[    0.620025]  #4
[    0.620066] smpboot cpu 4: start_ip = 9a000
[    0.790027]  #5 Ok.
[    0.790057] smpboot cpu 5: start_ip = 9a000
[    0.960003] Brought up 6 CPUs
[    0.960033] Total of 6 processors activated (38560.60 BogoMIPS).
[    0.963391] devtmpfs: initialized
[    0.963391] xor: automatically using best checksumming function: 
generic_sse
[    1.009895]    generic_sse: 14408.400 MB/sec
[    1.009928] xor: using function: generic_sse (14408.400 MB/sec)
[    1.010040] NET: Registered protocol family 16
[    1.010079] node 0 link 0: io port [1000, ffffff]
[    1.010079] TOM: 00000000e0000000 aka 3584M
[    1.010079] Fam 10h mmconf [e0000000, efffffff]
[    1.010079] node 0 link 0: mmio [a0000, bffff]
[    1.010079] node 0 link 0: mmio [e0000000, efffffff] ==> none
[    1.010079] node 0 link 0: mmio [f0000000, fe1fffff]
[    1.010079] node 0 link 0: mmio [fe200000, fe3fffff]
[    1.010079] node 0 link 0: mmio [fe400000, ffefffff]
[    1.010079] TOM2: 0000000420000000 aka 16896M
[    1.010079] bus: [00, 07] on node 0 link 0
[    1.010079] bus: 00 index 0 [io  0x0000-0xffff]
[    1.010080] bus: 00 index 1 [mem 0x000a0000-0x000bffff]
[    1.010081] bus: 00 index 2 [mem 0xf0000000-0xffffffff]
[    1.010082] bus: 00 index 3 [mem 0x420000000-0xfcffffffff]
[    1.010089] Extended Config Space enabled on 1 nodes
[    1.010151] ACPI: bus type pci registered
[    1.010185] PCI: MMCONFIG for domain 0000 [bus 00-ff] at [mem 
0xe0000000-0xefffffff] (base 0xe0000000)
[    1.010185] PCI: not using MMCONFIG
[    1.010185] PCI: Using configuration type 1 for base access
[    1.010185] PCI: Using configuration type 1 for extended access
[    1.010185] bio: create slab <bio-0> at 0
[    1.179907] raid6: int64x1   3168 MB/s
[    1.349907] raid6: int64x2   4039 MB/s
[    1.519903] raid6: int64x4   3205 MB/s
[    1.689900] raid6: int64x8   3017 MB/s
[    1.859895] raid6: sse2x1    4891 MB/s
[    2.029898] raid6: sse2x2    7971 MB/s
[    2.199900] raid6: sse2x4    9353 MB/s
[    2.199934] raid6: using algorithm sse2x4 (9353 MB/s)
[    2.200432] ACPI: EC: Look up EC in DSDT
[    2.200983] ACPI: Executed 4 blocks of module-level executable AML code
[    2.230239] ACPI: Interpreter enabled
[    2.230271] ACPI: (supports S0 S5)
[    2.230358] ACPI: Using IOAPIC for interrupt routing
[    2.230475] PCI: MMCONFIG for domain 0000 [bus 00-ff] at [mem 
0xe0000000-0xefffffff] (base 0xe0000000)
[    2.231052] PCI: MMCONFIG at [mem 0xe0000000-0xefffffff] reserved in 
ACPI motherboard resources
[    2.245403] ACPI: No dock devices found.
[    2.245403] PCI: Using host bridge windows from ACPI; if necessary, 
use "pci=nocrs" and report a bug
[    2.245403] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])
[    2.245403] pci_root PNP0A03:00: host bridge window [io  0x0000-0x0cf7]
[    2.245403] pci_root PNP0A03:00: host bridge window [io  0x0d00-0xffff]
[    2.245403] pci_root PNP0A03:00: host bridge window [mem 
0x000a0000-0x000bffff]
[    2.245403] pci_root PNP0A03:00: host bridge window [mem 
0x000d0000-0x000dffff]
[    2.245403] pci_root PNP0A03:00: host bridge window [mem 
0xf0000000-0xfebfffff]
[    2.245403] pci_root PNP0A03:00: address space collision: host bridge 
window [mem 0x000d0000-0x000dffff] conflicts with Adapter ROM [mem 
0x000cf000-0x000d53ff]
[    2.245403] pci 0000:00:00.0: [1022:9601] type 0 class 0x000600
[    2.245403] pci 0000:00:01.0: [1849:9602] type 1 class 0x000604
[    2.245403] pci 0000:00:02.0: [1022:9603] type 1 class 0x000604
[    2.245403] pci 0000:00:02.0: PME# supported from D0 D3hot D3cold
[    2.245403] pci 0000:00:02.0: PME# disabled
[    2.245403] pci 0000:00:03.0: [1022:960b] type 1 class 0x000604
[    2.245403] pci 0000:00:03.0: PME# supported from D0 D3hot D3cold
[    2.245403] pci 0000:00:03.0: PME# disabled
[    2.245403] pci 0000:00:04.0: [1022:9604] type 1 class 0x000604
[    2.245403] pci 0000:00:04.0: PME# supported from D0 D3hot D3cold
[    2.245403] pci 0000:00:04.0: PME# disabled
[    2.245403] pci 0000:00:09.0: [1022:9608] type 1 class 0x000604
[    2.245403] pci 0000:00:09.0: PME# supported from D0 D3hot D3cold
[    2.245403] pci 0000:00:09.0: PME# disabled
[    2.245403] pci 0000:00:0a.0: [1022:9609] type 1 class 0x000604
[    2.245403] pci 0000:00:0a.0: PME# supported from D0 D3hot D3cold
[    2.245403] pci 0000:00:0a.0: PME# disabled
[    2.245403] pci 0000:00:11.0: [1002:4391] type 0 class 0x000106
[    2.245403] pci 0000:00:11.0: reg 10: [io  0x9000-0x9007]
[    2.245403] pci 0000:00:11.0: reg 14: [io  0x8000-0x8003]
[    2.245403] pci 0000:00:11.0: reg 18: [io  0x7000-0x7007]
[    2.245403] pci 0000:00:11.0: reg 1c: [io  0x6000-0x6003]
[    2.245403] pci 0000:00:11.0: reg 20: [io  0x5000-0x500f]
[    2.245403] pci 0000:00:11.0: reg 24: [mem 0xfe1ffc00-0xfe1fffff]
[    2.245403] pci 0000:00:12.0: [1002:4397] type 0 class 0x000c03
[    2.245403] pci 0000:00:12.0: reg 10: [mem 0xfe1fe000-0xfe1fefff]
[    2.245403] pci 0000:00:12.2: [1002:4396] type 0 class 0x000c03
[    2.245403] pci 0000:00:12.2: reg 10: [mem 0xfe1ff800-0xfe1ff8ff]
[    2.245403] pci 0000:00:12.2: supports D1 D2
[    2.245403] pci 0000:00:12.2: PME# supported from D0 D1 D2 D3hot
[    2.245403] pci 0000:00:12.2: PME# disabled
[    2.245403] pci 0000:00:13.0: [1002:4397] type 0 class 0x000c03
[    2.245403] pci 0000:00:13.0: reg 10: [mem 0xfe1fd000-0xfe1fdfff]
[    2.245403] pci 0000:00:13.2: [1002:4396] type 0 class 0x000c03
[    2.245403] pci 0000:00:13.2: reg 10: [mem 0xfe1ff400-0xfe1ff4ff]
[    2.245403] pci 0000:00:13.2: supports D1 D2
[    2.245403] pci 0000:00:13.2: PME# supported from D0 D1 D2 D3hot
[    2.245403] pci 0000:00:13.2: PME# disabled
[    2.245403] pci 0000:00:14.0: [1002:4385] type 0 class 0x000c05
[    2.245403] pci 0000:00:14.2: [1002:4383] type 0 class 0x000403
[    2.245403] pci 0000:00:14.2: reg 10: [mem 0xfe1f8000-0xfe1fbfff 64bit]
[    2.245403] pci 0000:00:14.2: PME# supported from D0 D3hot D3cold
[    2.245403] pci 0000:00:14.2: PME# disabled
[    2.245403] pci 0000:00:14.3: [1002:439d] type 0 class 0x000601
[    2.245403] pci 0000:00:14.4: [1002:4384] type 1 class 0x000604
[    2.245403] pci 0000:00:14.5: [1002:4399] type 0 class 0x000c03
[    2.245403] pci 0000:00:14.5: reg 10: [mem 0xfe1fc000-0xfe1fcfff]
[    2.245403] pci 0000:00:16.0: [1002:4397] type 0 class 0x000c03
[    2.245403] pci 0000:00:16.0: reg 10: [mem 0xfe1f7000-0xfe1f7fff]
[    2.245403] pci 0000:00:16.2: [1002:4396] type 0 class 0x000c03
[    2.245403] pci 0000:00:16.2: reg 10: [mem 0xfe1ff000-0xfe1ff0ff]
[    2.245403] pci 0000:00:16.2: supports D1 D2
[    2.245403] pci 0000:00:16.2: PME# supported from D0 D1 D2 D3hot
[    2.245403] pci 0000:00:16.2: PME# disabled
[    2.245403] pci 0000:00:18.0: [1022:1200] type 0 class 0x000600
[    2.245403] pci 0000:00:18.1: [1022:1201] type 0 class 0x000600
[    2.245403] pci 0000:00:18.2: [1022:1202] type 0 class 0x000600
[    2.245403] pci 0000:00:18.3: [1022:1203] type 0 class 0x000600
[    2.250003] pci 0000:00:18.4: [1022:1204] type 0 class 0x000600
[    2.250040] pci 0000:01:05.0: [1002:9715] type 0 class 0x000300
[    2.250046] pci 0000:01:05.0: reg 10: [mem 0xf0000000-0xf7ffffff pref]
[    2.250050] pci 0000:01:05.0: reg 14: [io  0xa000-0xa0ff]
[    2.250054] pci 0000:01:05.0: reg 18: [mem 0xfe3f0000-0xfe3fffff]
[    2.250061] pci 0000:01:05.0: reg 24: [mem 0xfe200000-0xfe2fffff]
[    2.250070] pci 0000:01:05.0: supports D1 D2
[    2.250097] pci 0000:00:01.0: PCI bridge to [bus 01-01]
[    2.250129] pci 0000:00:01.0:   bridge window [io  0xa000-0xafff]
[    2.250131] pci 0000:00:01.0:   bridge window [mem 0xfe200000-0xfe3fffff]
[    2.250134] pci 0000:00:01.0:   bridge window [mem 
0xf0000000-0xf7ffffff 64bit pref]
[    2.250158] pci 0000:04:00.0: [1000:0073] type 0 class 0x000104
[    2.250165] pci 0000:04:00.0: reg 10: [io  0xc000-0xc0ff]
[    2.250172] pci 0000:04:00.0: reg 14: [mem 0xfe6fc000-0xfe6fffff 64bit]
[    2.250180] pci 0000:04:00.0: reg 1c: [mem 0xfe680000-0xfe6bffff 64bit]
[    2.250190] pci 0000:04:00.0: reg 30: [mem 0xfe6c0000-0xfe6dffff pref]
[    2.250204] pci 0000:04:00.0: supports D1 D2
[    2.270016] pci 0000:00:02.0: PCI bridge to [bus 04-04]
[    2.270057] pci 0000:00:02.0:   bridge window [io  0xc000-0xcfff]
[    2.270059] pci 0000:00:02.0:   bridge window [mem 0xfe600000-0xfe6fffff]
[    2.270062] pci 0000:00:02.0:   bridge window [mem 
0xfff00000-0x000fffff pref] (disabled)
[    2.270088] pci 0000:05:00.0: [1000:0073] type 0 class 0x000104
[    2.270096] pci 0000:05:00.0: reg 10: [io  0xd000-0xd0ff]
[    2.270104] pci 0000:05:00.0: reg 14: [mem 0xfe7fc000-0xfe7fffff 64bit]
[    2.270112] pci 0000:05:00.0: reg 1c: [mem 0xfe780000-0xfe7bffff 64bit]
[    2.270121] pci 0000:05:00.0: reg 30: [mem 0xfe7c0000-0xfe7dffff pref]
[    2.270136] pci 0000:05:00.0: supports D1 D2
[    2.290016] pci 0000:00:03.0: PCI bridge to [bus 05-05]
[    2.290054] pci 0000:00:03.0:   bridge window [io  0xd000-0xdfff]
[    2.290057] pci 0000:00:03.0:   bridge window [mem 0xfe700000-0xfe7fffff]
[    2.290060] pci 0000:00:03.0:   bridge window [mem 
0xfff00000-0x000fffff pref] (disabled)
[    2.290091] pci 0000:07:00.0: [11ab:7042] type 0 class 0x000100
[    2.290104] pci 0000:07:00.0: reg 10: [mem 0xfe900000-0xfe9fffff 64bit]
[    2.290110] pci 0000:07:00.0: reg 18: [io  0xe800-0xe8ff]
[    2.290133] pci 0000:07:00.0: reg 30: [mem 0xfe800000-0xfe8fffff pref]
[    2.290161] pci 0000:07:00.0: disabling ASPM on pre-1.1 PCIe device. 
  You can enable it with 'pcie_aspm=force'
[    2.290198] pci 0000:00:04.0: PCI bridge to [bus 07-07]
[    2.290230] pci 0000:00:04.0:   bridge window [io  0xe000-0xefff]
[    2.290232] pci 0000:00:04.0:   bridge window [mem 0xfe800000-0xfe9fffff]
[    2.290235] pci 0000:00:04.0:   bridge window [mem 
0xfff00000-0x000fffff pref] (disabled)
[    2.290264] pci 0000:02:00.0: [1033:0194] type 0 class 0x000c03
[    2.290277] pci 0000:02:00.0: reg 10: [mem 0xfe4fe000-0xfe4fffff 64bit]
[    2.290327] pci 0000:02:00.0: PME# supported from D0 D3hot D3cold
[    2.290330] pci 0000:02:00.0: PME# disabled
[    2.310021] pci 0000:00:09.0: PCI bridge to [bus 02-02]
[    2.310056] pci 0000:00:09.0:   bridge window [io  0xf000-0x0000] 
(disabled)
[    2.310058] pci 0000:00:09.0:   bridge window [mem 0xfe400000-0xfe4fffff]
[    2.310061] pci 0000:00:09.0:   bridge window [mem 
0xfff00000-0x000fffff pref] (disabled)
[    2.310092] pci 0000:03:00.0: [10ec:8168] type 0 class 0x000200
[    2.310104] pci 0000:03:00.0: reg 10: [io  0xb800-0xb8ff]
[    2.310122] pci 0000:03:00.0: reg 18: [mem 0xfdfff000-0xfdffffff 
64bit pref]
[    2.310134] pci 0000:03:00.0: reg 20: [mem 0xfdff8000-0xfdffbfff 
64bit pref]
[    2.310142] pci 0000:03:00.0: reg 30: [mem 0xfe5e0000-0xfe5fffff pref]
[    2.310167] pci 0000:03:00.0: supports D1 D2
[    2.310169] pci 0000:03:00.0: PME# supported from D0 D1 D2 D3hot D3cold
[    2.310172] pci 0000:03:00.0: PME# disabled
[    2.330022] pci 0000:00:0a.0: PCI bridge to [bus 03-03]
[    2.330056] pci 0000:00:0a.0:   bridge window [io  0xb000-0xbfff]
[    2.330058] pci 0000:00:0a.0:   bridge window [mem 0xfe500000-0xfe5fffff]
[    2.330061] pci 0000:00:0a.0:   bridge window [mem 
0xfdf00000-0xfdffffff 64bit pref]
[    2.330109] pci 0000:00:14.4: PCI bridge to [bus 06-06] (subtractive 
decode)
[    2.330142] pci 0000:00:14.4:   bridge window [io  0xf000-0x0000] 
(disabled)
[    2.330145] pci 0000:00:14.4:   bridge window [mem 
0xfff00000-0x000fffff] (disabled)
[    2.330148] pci 0000:00:14.4:   bridge window [mem 
0xfff00000-0x000fffff pref] (disabled)
[    2.330150] pci 0000:00:14.4:   bridge window [io  0x0000-0x0cf7] 
(subtractive decode)
[    2.330152] pci 0000:00:14.4:   bridge window [io  0x0d00-0xffff] 
(subtractive decode)
[    2.330153] pci 0000:00:14.4:   bridge window [mem 
0x000a0000-0x000bffff] (subtractive decode)
[    2.330155] pci 0000:00:14.4:   bridge window [mem 
0xf0000000-0xfebfffff] (subtractive decode)
[    2.330168] pci_bus 0000:00: on NUMA node 0
[    2.330171] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0._PRT]
[    2.330251] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.P0P1._PRT]
[    2.330268] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.PCE2._PRT]
[    2.330283] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.PCE3._PRT]
[    2.330296] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.PCE4._PRT]
[    2.330310] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.PCE9._PRT]
[    2.330324] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.PCEA._PRT]
[    2.330343] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.P0PC._PRT]
[    2.330382]  pci0000:00: Requesting ACPI _OSC control (0x1d)
[    2.330420]  pci0000:00: ACPI _OSC request failed (AE_NOT_FOUND), 
returned control mask: 0x1d
[    2.330453] ACPI _OSC control for PCIe not granted, disabling ASPM
[    2.330576] ACPI: PCI Interrupt Link [LNKA] (IRQs 10 11 14 15) *0
[    2.330576] ACPI: PCI Interrupt Link [LNKB] (IRQs 10 11 14 15) *0
[    2.330576] ACPI: PCI Interrupt Link [LNKC] (IRQs 10 11 14 15) *0
[    2.330763] ACPI: PCI Interrupt Link [LNKD] (IRQs 10 11 14 15) *0
[    2.331002] ACPI: PCI Interrupt Link [LNKE] (IRQs 10 11 14 15) *0
[    2.331234] ACPI: PCI Interrupt Link [LNKF] (IRQs 10 11 14 15) *0
[    2.331466] ACPI: PCI Interrupt Link [LNKG] (IRQs 10 11 14 15) *0
[    2.331698] ACPI: PCI Interrupt Link [LNKH] (IRQs 10 11 14 15) *0
[    2.331926] vgaarb: device added: 
PCI:0000:01:05.0,decodes=io+mem,owns=io+mem,locks=none
[    2.331926] vgaarb: loaded
[    2.331926] vgaarb: bridge control possible 0000:01:05.0
[    2.331926] SCSI subsystem initialized
[    2.331926] libata version 3.00 loaded.
[    2.331926] PCI: Using ACPI for IRQ routing
[    2.341813] PCI: pci_cache_line_size set to 64 bytes
[    2.341867] reserve RAM buffer: 000000000009c400 - 000000000009ffff
[    2.341869] reserve RAM buffer: 00000000dbf80000 - 00000000dbffffff
[    2.341881] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0
[    2.341881] hpet0: 3 comparators, 32-bit 14.318180 MHz counter
[    2.342205] Switching to clocksource hpet
[    2.342293] pnp: PnP ACPI init
[    2.342293] ACPI: bus type pnp registered
[    2.342293] pnp 00:00: [bus 00-ff]
[    2.342293] pnp 00:00: [io  0x0cf8-0x0cff]
[    2.342293] pnp 00:00: [io  0x0000-0x0cf7 window]
[    2.342293] pnp 00:00: [io  0x0d00-0xffff window]
[    2.342293] pnp 00:00: [mem 0x000a0000-0x000bffff window]
[    2.342293] pnp 00:00: [mem 0x000d0000-0x000dffff window]
[    2.342293] pnp 00:00: [mem 0xe0000000-0xdfffffff window disabled]
[    2.342293] pnp 00:00: [mem 0xf0000000-0xfebfffff window]
[    2.342293] pnp 00:00: Plug and Play ACPI device, IDs PNP0a03 (active)
[    2.342293] pnp 00:01: [mem 0x00000000-0xffffffffffffffff disabled]
[    2.342293] pnp 00:01: [mem 0x00000000-0xffffffffffffffff disabled]
[    2.342293] system 00:01: Plug and Play ACPI device, IDs PNP0c02 (active)
[    2.342293] pnp 00:02: [dma 4]
[    2.342293] pnp 00:02: [io  0x0000-0x000f]
[    2.342293] pnp 00:02: [io  0x0081-0x0083]
[    2.342293] pnp 00:02: [io  0x0087]
[    2.342293] pnp 00:02: [io  0x0089-0x008b]
[    2.342293] pnp 00:02: [io  0x008f]
[    2.342293] pnp 00:02: [io  0x00c0-0x00df]
[    2.342293] pnp 00:02: Plug and Play ACPI device, IDs PNP0200 (active)
[    2.342293] pnp 00:03: [io  0x0070-0x0071]
[    2.342293] pnp 00:03: [irq 8]
[    2.342293] pnp 00:03: Plug and Play ACPI device, IDs PNP0b00 (active)
[    2.342293] pnp 00:04: [io  0x0061]
[    2.342293] pnp 00:04: Plug and Play ACPI device, IDs PNP0800 (active)
[    2.342293] pnp 00:05: [io  0x00f0-0x00ff]
[    2.342293] pnp 00:05: [irq 13]
[    2.342293] pnp 00:05: Plug and Play ACPI device, IDs PNP0c04 (active)
[    2.342293] pnp 00:06: [io  0x02f8-0x02ff]
[    2.342293] pnp 00:06: [irq 3]
[    2.342293] pnp 00:06: [dma 0 disabled]
[    2.342293] pnp 00:06: Plug and Play ACPI device, IDs PNP0510 (active)
[    2.342293] pnp 00:07: [mem 0xfed00000-0xfed003ff]
[    2.342293] pnp 00:07: Plug and Play ACPI device, IDs PNP0103 (active)
[    2.342293] pnp 00:08: [mem 0xfec00000-0xfec00fff]
[    2.342293] pnp 00:08: [mem 0xfee00000-0xfee00fff]
[    2.342293] system 00:08: [mem 0xfec00000-0xfec00fff] could not be 
reserved
[    2.342293] system 00:08: [mem 0xfee00000-0xfee00fff] has been reserved
[    2.342293] system 00:08: Plug and Play ACPI device, IDs PNP0c02 (active)
[    2.342293] pnp 00:09: [io  0x0010-0x001f]
[    2.342293] pnp 00:09: [io  0x0022-0x003f]
[    2.342293] pnp 00:09: [io  0x0062-0x0063]
[    2.342293] pnp 00:09: [io  0x0065-0x006f]
[    2.342293] pnp 00:09: [io  0x0072-0x007f]
[    2.342293] pnp 00:09: [io  0x0080]
[    2.342293] pnp 00:09: [io  0x0084-0x0086]
[    2.342293] pnp 00:09: [io  0x0088]
[    2.342293] pnp 00:09: [io  0x008c-0x008e]
[    2.342293] pnp 00:09: [io  0x0090-0x009f]
[    2.342293] pnp 00:09: [io  0x00a2-0x00bf]
[    2.342293] pnp 00:09: [io  0x00b1]
[    2.342293] pnp 00:09: [io  0x00e0-0x00ef]
[    2.342293] pnp 00:09: [io  0x04d0-0x04d1]
[    2.342293] pnp 00:09: [io  0x040b]
[    2.342293] pnp 00:09: [io  0x04d6]
[    2.342293] pnp 00:09: [io  0x0c00-0x0c01]
[    2.342293] pnp 00:09: [io  0x0c14]
[    2.342293] pnp 00:09: [io  0x0c50-0x0c51]
[    2.342293] pnp 00:09: [io  0x0c52]
[    2.342293] pnp 00:09: [io  0x0c6c]
[    2.342293] pnp 00:09: [io  0x0c6f]
[    2.342293] pnp 00:09: [io  0x0cd0-0x0cd1]
[    2.342293] pnp 00:09: [io  0x0cd2-0x0cd3]
[    2.342293] pnp 00:09: [io  0x0cd4-0x0cd5]
[    2.342293] pnp 00:09: [io  0x0cd6-0x0cd7]
[    2.342293] pnp 00:09: [io  0x0cd8-0x0cdf]
[    2.342293] pnp 00:09: [io  0x0800-0x089f]
[    2.342293] pnp 00:09: [io  0x0000-0xffffffffffffffff disabled]
[    2.342293] pnp 00:09: [io  0x0b00-0x0b1f]
[    2.342293] pnp 00:09: [io  0x0b20-0x0b3f]
[    2.342293] pnp 00:09: [io  0x0900-0x090f]
[    2.342293] pnp 00:09: [io  0x0910-0x091f]
[    2.342293] pnp 00:09: [io  0xfe00-0xfefe]
[    2.342293] pnp 00:09: [io  0x0060-0x005f disabled]
[    2.342293] pnp 00:09: [io  0x0064-0x0063 disabled]
[    2.342293] pnp 00:09: [mem 0xffb80000-0xffbfffff]
[    2.342293] pnp 00:09: [mem 0xfec10000-0xfec1001f]
[    2.342293] pnp 00:09: [mem 0xfed80000-0xfed80fff]
[    2.342293] pnp 00:09: [mem 0x00000000-0xffffffffffffffff disabled]
[    2.342293] system 00:09: [io  0x04d0-0x04d1] has been reserved
[    2.342293] system 00:09: [io  0x040b] has been reserved
[    2.342293] system 00:09: [io  0x04d6] has been reserved
[    2.342293] system 00:09: [io  0x0c00-0x0c01] has been reserved
[    2.342293] system 00:09: [io  0x0c14] has been reserved
[    2.342293] system 00:09: [io  0x0c50-0x0c51] has been reserved
[    2.342293] system 00:09: [io  0x0c52] has been reserved
[    2.342293] system 00:09: [io  0x0c6c] has been reserved
[    2.342293] system 00:09: [io  0x0c6f] has been reserved
[    2.342293] system 00:09: [io  0x0cd0-0x0cd1] has been reserved
[    2.342293] system 00:09: [io  0x0cd2-0x0cd3] has been reserved
[    2.342293] system 00:09: [io  0x0cd4-0x0cd5] has been reserved
[    2.342293] system 00:09: [io  0x0cd6-0x0cd7] has been reserved
[    2.342293] system 00:09: [io  0x0cd8-0x0cdf] has been reserved
[    2.342293] system 00:09: [io  0x0800-0x089f] has been reserved
[    2.342293] system 00:09: [io  0x0b00-0x0b1f] has been reserved
[    2.342293] system 00:09: [io  0x0b20-0x0b3f] has been reserved
[    2.342293] system 00:09: [io  0x0900-0x090f] has been reserved
[    2.342293] system 00:09: [io  0x0910-0x091f] has been reserved
[    2.342293] system 00:09: [io  0xfe00-0xfefe] has been reserved
[    2.342293] system 00:09: [mem 0xffb80000-0xffbfffff] has been reserved
[    2.342293] system 00:09: [mem 0xfec10000-0xfec1001f] has been reserved
[    2.342293] system 00:09: [mem 0xfed80000-0xfed80fff] has been reserved
[    2.342293] system 00:09: Plug and Play ACPI device, IDs PNP0c02 (active)
[    2.342293] pnp 00:0a: [io  0x0060]
[    2.342293] pnp 00:0a: [io  0x0064]
[    2.342293] pnp 00:0a: [irq 1]
[    2.342293] pnp 00:0a: Plug and Play ACPI device, IDs PNP0303 PNP030b 
(active)
[    2.342293] pnp 00:0b: [io  0x03f8-0x03ff]
[    2.342293] pnp 00:0b: [irq 4]
[    2.342293] pnp 00:0b: [dma 0 disabled]
[    2.342293] pnp 00:0b: Plug and Play ACPI device, IDs PNP0501 (active)
[    2.342316] pnp 00:0c: [io  0x0000-0xffffffffffffffff disabled]
[    2.342317] pnp 00:0c: [io  0x0000-0xffffffffffffffff disabled]
[    2.342319] pnp 00:0c: [io  0x0290-0x029f]
[    2.342365] system 00:0c: [io  0x0290-0x029f] has been reserved
[    2.342396] system 00:0c: Plug and Play ACPI device, IDs PNP0c02 (active)
[    2.342412] pnp 00:0d: [mem 0xe0000000-0xefffffff]
[    2.342457] system 00:0d: [mem 0xe0000000-0xefffffff] has been reserved
[    2.342489] system 00:0d: Plug and Play ACPI device, IDs PNP0c02 (active)
[    2.342549] pnp 00:0e: [mem 0x00000000-0x0009ffff]
[    2.342550] pnp 00:0e: [mem 0x000c0000-0x000cffff]
[    2.342551] pnp 00:0e: [mem 0x000e0000-0x000fffff]
[    2.342553] pnp 00:0e: [mem 0x00100000-0xdfffffff]
[    2.342554] pnp 00:0e: [mem 0xfec00000-0xffffffff]
[    2.342629] system 00:0e: [mem 0x00000000-0x0009ffff] could not be 
reserved
[    2.342660] system 00:0e: [mem 0x000c0000-0x000cffff] could not be 
reserved
[    2.342691] system 00:0e: [mem 0x000e0000-0x000fffff] could not be 
reserved
[    2.342723] system 00:0e: [mem 0x00100000-0xdfffffff] could not be 
reserved
[    2.342754] system 00:0e: [mem 0xfec00000-0xffffffff] could not be 
reserved
[    2.342785] system 00:0e: Plug and Play ACPI device, IDs PNP0c01 (active)
[    2.342835] pnp: PnP ACPI: found 15 devices
[    2.342865] ACPI: ACPI bus type pnp unregistered
[    2.345616] Switched to NOHz mode on CPU #0
[    2.345673] Switched to NOHz mode on CPU #3
[    2.345731] Switched to NOHz mode on CPU #1
[    2.345734] Switched to NOHz mode on CPU #2
[    2.345746] Switched to NOHz mode on CPU #4
[    2.345749] Switched to NOHz mode on CPU #5
[    2.349869] PCI: max bus depth: 1 pci_try_num: 2
[    2.349888] pci 0000:00:01.0: PCI bridge to [bus 01-01]
[    2.349919] pci 0000:00:01.0:   bridge window [io  0xa000-0xafff]
[    2.349951] pci 0000:00:01.0:   bridge window [mem 0xfe200000-0xfe3fffff]
[    2.349982] pci 0000:00:01.0:   bridge window [mem 
0xf0000000-0xf7ffffff 64bit pref]
[    2.350059] pci 0000:00:02.0: PCI bridge to [bus 04-04]
[    2.350090] pci 0000:00:02.0:   bridge window [io  0xc000-0xcfff]
[    2.350122] pci 0000:00:02.0:   bridge window [mem 0xfe600000-0xfe6fffff]
[    2.350153] pci 0000:00:02.0:   bridge window [mem pref disabled]
[    2.350185] pci 0000:00:03.0: PCI bridge to [bus 05-05]
[    2.350216] pci 0000:00:03.0:   bridge window [io  0xd000-0xdfff]
[    2.350249] pci 0000:00:03.0:   bridge window [mem 0xfe700000-0xfe7fffff]
[    2.350280] pci 0000:00:03.0:   bridge window [mem pref disabled]
[    2.350312] pci 0000:00:04.0: PCI bridge to [bus 07-07]
[    2.350343] pci 0000:00:04.0:   bridge window [io  0xe000-0xefff]
[    2.350375] pci 0000:00:04.0:   bridge window [mem 0xfe800000-0xfe9fffff]
[    2.350411] pci 0000:00:04.0:   bridge window [mem pref disabled]
[    2.350443] pci 0000:00:09.0: PCI bridge to [bus 02-02]
[    2.350473] pci 0000:00:09.0:   bridge window [io  disabled]
[    2.350504] pci 0000:00:09.0:   bridge window [mem 0xfe400000-0xfe4fffff]
[    2.350535] pci 0000:00:09.0:   bridge window [mem pref disabled]
[    2.350567] pci 0000:00:0a.0: PCI bridge to [bus 03-03]
[    2.350598] pci 0000:00:0a.0:   bridge window [io  0xb000-0xbfff]
[    2.350629] pci 0000:00:0a.0:   bridge window [mem 0xfe500000-0xfe5fffff]
[    2.350661] pci 0000:00:0a.0:   bridge window [mem 
0xfdf00000-0xfdffffff 64bit pref]
[    2.350694] pci 0000:00:14.4: PCI bridge to [bus 06-06]
[    2.350724] pci 0000:00:14.4:   bridge window [io  disabled]
[    2.350757] pci 0000:00:14.4:   bridge window [mem disabled]
[    2.350789] pci 0000:00:14.4:   bridge window [mem pref disabled]
[    2.350829] pci 0000:00:02.0: PCI INT A -> GSI 18 (level, low) -> IRQ 18
[    2.350861] pci 0000:00:02.0: setting latency timer to 64
[    2.350865] pci 0000:00:03.0: PCI INT A -> GSI 19 (level, low) -> IRQ 19
[    2.350897] pci 0000:00:03.0: setting latency timer to 64
[    2.350901] pci 0000:00:04.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
[    2.350932] pci 0000:00:04.0: setting latency timer to 64
[    2.350936] pci 0000:00:09.0: PCI INT A -> GSI 17 (level, low) -> IRQ 17
[    2.350967] pci 0000:00:09.0: setting latency timer to 64
[    2.350970] pci 0000:00:0a.0: PCI INT A -> GSI 18 (level, low) -> IRQ 18
[    2.351001] pci 0000:00:0a.0: setting latency timer to 64
[    2.351006] pci_bus 0000:00: resource 4 [io  0x0000-0x0cf7]
[    2.351007] pci_bus 0000:00: resource 5 [io  0x0d00-0xffff]
[    2.351009] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff]
[    2.351010] pci_bus 0000:00: resource 7 [mem 0xf0000000-0xfebfffff]
[    2.351012] pci_bus 0000:01: resource 0 [io  0xa000-0xafff]
[    2.351013] pci_bus 0000:01: resource 1 [mem 0xfe200000-0xfe3fffff]
[    2.351014] pci_bus 0000:01: resource 2 [mem 0xf0000000-0xf7ffffff 
64bit pref]
[    2.351016] pci_bus 0000:04: resource 0 [io  0xc000-0xcfff]
[    2.351017] pci_bus 0000:04: resource 1 [mem 0xfe600000-0xfe6fffff]
[    2.351019] pci_bus 0000:05: resource 0 [io  0xd000-0xdfff]
[    2.351020] pci_bus 0000:05: resource 1 [mem 0xfe700000-0xfe7fffff]
[    2.351021] pci_bus 0000:07: resource 0 [io  0xe000-0xefff]
[    2.351023] pci_bus 0000:07: resource 1 [mem 0xfe800000-0xfe9fffff]
[    2.351024] pci_bus 0000:02: resource 1 [mem 0xfe400000-0xfe4fffff]
[    2.351025] pci_bus 0000:03: resource 0 [io  0xb000-0xbfff]
[    2.351027] pci_bus 0000:03: resource 1 [mem 0xfe500000-0xfe5fffff]
[    2.351028] pci_bus 0000:03: resource 2 [mem 0xfdf00000-0xfdffffff 
64bit pref]
[    2.351030] pci_bus 0000:06: resource 4 [io  0x0000-0x0cf7]
[    2.351031] pci_bus 0000:06: resource 5 [io  0x0d00-0xffff]
[    2.351032] pci_bus 0000:06: resource 6 [mem 0x000a0000-0x000bffff]
[    2.351034] pci_bus 0000:06: resource 7 [mem 0xf0000000-0xfebfffff]
[    2.351057] NET: Registered protocol family 2
[    2.351135] IP route cache hash table entries: 524288 (order: 10, 
4194304 bytes)
[    2.352087] TCP established hash table entries: 262144 (order: 10, 
4194304 bytes)
[    2.353248] TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)
[    2.353567] TCP: Hash tables configured (established 262144 bind 65536)
[    2.353601] TCP reno registered
[    2.353632] UDP hash table entries: 8192 (order: 6, 262144 bytes)
[    2.353743] UDP-Lite hash table entries: 8192 (order: 6, 262144 bytes)
[    2.353926] NET: Registered protocol family 1
[    2.354056] RPC: Registered named UNIX socket transport module.
[    2.354087] RPC: Registered udp transport module.
[    2.354117] RPC: Registered tcp transport module.
[    2.354147] RPC: Registered tcp NFSv4.1 backchannel transport module.
[    2.354182] pci 0000:00:01.0: MSI quirk detected; subordinate MSI 
disabled
[    2.500089] pci 0000:01:05.0: Boot video device
[    2.500118] PCI: CLS 64 bytes, default 64
[    2.500152] Unpacking initramfs...
[    2.539130] Freeing initrd memory: 3168k freed
[    2.539863] PCI-DMA: Disabling AGP.
[    2.540036] PCI-DMA: aperture base @ d0000000 size 65536 KB
[    2.540067] PCI-DMA: using GART IOMMU.
[    2.540098] PCI-DMA: Reserving 64MB of IOMMU area in the AGP aperture
[    2.574189] HugeTLB registered 2 MB page size, pre-allocated 0 pages
[    2.636100] Installing knfsd (copyright (C) 1996 okir@monad.swb.de).
[    2.636310] msgmni has been set to 31992
[    2.636457] Block layer SCSI generic (bsg) driver version 0.4 loaded 
(major 254)
[    2.636490] io scheduler noop registered
[    2.636519] io scheduler deadline registered
[    2.636606] io scheduler cfq registered (default)
[    2.636711] pcieport 0000:00:02.0: setting latency timer to 64
[    2.636730] pcieport 0000:00:02.0: irq 40 for MSI/MSI-X
[    2.636781] pcieport 0000:00:03.0: setting latency timer to 64
[    2.636795] pcieport 0000:00:03.0: irq 41 for MSI/MSI-X
[    2.636839] pcieport 0000:00:04.0: setting latency timer to 64
[    2.636854] pcieport 0000:00:04.0: irq 42 for MSI/MSI-X
[    2.636898] pcieport 0000:00:09.0: setting latency timer to 64
[    2.636912] pcieport 0000:00:09.0: irq 43 for MSI/MSI-X
[    2.636956] pcieport 0000:00:0a.0: setting latency timer to 64
[    2.636971] pcieport 0000:00:0a.0: irq 44 for MSI/MSI-X
[    2.637246] input: Power Button as 
/devices/LNXSYSTM:00/device:00/PNP0C0C:00/input/input0
[    2.637280] ACPI: Power Button [PWRB]
[    2.637353] input: Power Button as 
/devices/LNXSYSTM:00/LNXPWRBN:00/input/input1
[    2.637386] ACPI: Power Button [PWRF]
[    2.637446] ACPI: acpi_idle registered with cpuidle
[    2.658560] Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled
[    2.679110] serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
[    2.699710] serial8250: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
[    2.720581] 00:0b: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
[    2.720829] Real Time Clock Driver v1.12b
[    2.721594] loop: module loaded
[    2.721752] tun: Universal TUN/TAP device driver, 1.6
[    2.721782] tun: (C) 1999-2004 Max Krasnyansky <maxk@qualcomm.com>
[    2.721866] r8169 Gigabit Ethernet driver 2.3LK-NAPI loaded
[    2.721909] r8169 0000:03:00.0: PCI INT A -> GSI 18 (level, low) -> 
IRQ 18
[    2.721993] r8169 0000:03:00.0: setting latency timer to 64
[    2.722028] r8169 0000:03:00.0: irq 45 for MSI/MSI-X
[    2.722217] r8169 0000:03:00.0: eth0: RTL8168e/8111e at 
0xffffc9000000e000, 00:25:22:55:3e:2d, XID 0c100000 IRQ 45
[    2.722289] netconsole: local port 6666
[    2.722332] netconsole: local IP 192.168.2.1
[    2.722361] netconsole: interface 'eth0'
[    2.722391] netconsole: remote port 5050
[    2.722420] netconsole: remote IP 192.168.2.87
[    2.722451] netconsole: remote ethernet address 00:16:cb:a7:dd:d1
[    2.722482] netconsole: device eth0 not up yet, forcing it
[    3.540028] Refined TSC clocksource calibration: 3213.418 MHz.
[    3.540067] Switching to clocksource tsc
[   63.200106] r8169 0000:03:00.0: eth0: unable to load firmware patch 
rtl_nic/rtl8168e-1.fw (-2)
[   63.223480] r8169 0000:03:00.0: eth0: link down
[   63.223523] r8169 0000:03:00.0: eth0: link down
[   65.179915] r8169 0000:03:00.0: eth0: link up
[   65.182530] console [netcon0] enabled
[   65.182562] netconsole: network logging started
[   65.182738] i8042: PNP: PS/2 Controller [PNP0303:PS2K] at 0x60,0x64 irq 1
[   65.182771] i8042: PNP: PS/2 appears to have AUX port disabled, if 
this is incorrect please boot with i8042.nopnp
[   65.183304] serio: i8042 KBD port at 0x60,0x64 irq 1
[   65.183474] mousedev: PS/2 mouse device common for all mice
[   65.183532] i2c /dev entries driver
[   65.183588] md: linear personality registered for level -1
[   65.183619] md: raid0 personality registered for level 0
[   65.183651] md: raid1 personality registered for level 1
[   65.183683] md: raid10 personality registered for level 10
[   65.183714] md: raid6 personality registered for level 6
[   65.183746] md: raid5 personality registered for level 5
[   65.183779] md: raid4 personality registered for level 4
[   65.183968] device-mapper: ioctl: 4.20.0-ioctl (2011-02-02) 
initialised: dm-devel@redhat.com
[   65.184375] cpuidle: using governor ladder
[   65.185144] cpuidle: using governor menu
[   65.185339] TCP cubic registered
[   65.185377] NET: Registered protocol family 17
[   65.185431] Bridge firewalling registered
[   65.185468] 802.1Q VLAN Support v1.8
[   65.185502] Registering the dns_resolver key type
[   65.185730] registered taskstats version 1
[   65.216255] Freeing unused kernel memory: 468k freed
[   65.226393] udev[1318]: starting version 164
[   65.237596] input: AT Translated Set 2 keyboard as 
/devices/platform/i8042/serio0/input/input2
[   65.243627] megasas: 00.00.05.38-rc1 Wed. May. 11 17:00:00 PDT 2011
[   65.243694] megasas: 0x1000:0x0073:0x1014:0x03b1: bus 4:slot 0:func 0
[   65.243765] megaraid_sas 0000:04:00.0: PCI INT A -> GSI 18 (level, 
low) -> IRQ 18
[   65.243803] megaraid_sas 0000:04:00.0: setting latency timer to 64
[   65.244850] megasas: FW now in Ready state
[   65.245232] megaraid_sas 0000:04:00.0: irq 46 for MSI/MSI-X
[   65.246538] sata_mv 0000:07:00.0: version 1.28
[   65.246610] sata_mv 0000:07:00.0: PCI INT A -> GSI 16 (level, low) -> 
IRQ 16
[   65.246785] sata_mv 0000:07:00.0: Gen-IIE 32 slots 4 ports SCSI mode 
IRQ via INTx
[   65.246825] sata_mv 0000:07:00.0: setting latency timer to 64
[   65.247046] ahci 0000:00:11.0: version 3.0
[   65.247063] ahci 0000:00:11.0: PCI INT A -> GSI 19 (level, low) -> IRQ 19
[   65.247146] ahci 0000:00:11.0: irq 47 for MSI/MSI-X
[   65.247256] ahci 0000:00:11.0: AHCI 0001.0200 32 slots 6 ports 6 Gbps 
0x3f impl SATA mode
[   65.247293] ahci 0000:00:11.0: flags: 64bit ncq sntf ilck pm led clo 
pmp pio slum part
[   65.247385] usbcore: registered new interface driver usbfs
[   65.247443] usbcore: registered new interface driver hub
[   65.248070] scsi1 : sata_mv
[   65.248189] usbcore: registered new device driver usb
[   65.248450] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[   65.248497] scsi3 : sata_mv
[   65.248728] scsi2 : ahci
[   65.249183] scsi4 : sata_mv
[   65.249270] scsi6 : sata_mv
[   65.249318] ata1: SATA max UDMA/133 mmio m1048576@0xfe900000 port 
0xfe922000 irq 16
[   65.249354] ata2: SATA max UDMA/133 mmio m1048576@0xfe900000 port 
0xfe924000 irq 16
[   65.249390] ata3: SATA max UDMA/133 mmio m1048576@0xfe900000 port 
0xfe926000 irq 16
[   65.249425] ata4: SATA max UDMA/133 mmio m1048576@0xfe900000 port 
0xfe928000 irq 16
[   65.249981] scsi5 : ahci
[   65.250068] scsi7 : ahci
[   65.250436] scsi8 : ahci
[   65.250534] scsi9 : ahci
[   65.250685] scsi10 : ahci
[   65.250746] ata5: SATA max UDMA/133 abar m1024@0xfe1ffc00 port 
0xfe1ffd00 irq 47
[   65.250783] ata6: SATA max UDMA/133 abar m1024@0xfe1ffc00 port 
0xfe1ffd80 irq 47
[   65.250820] ata7: SATA max UDMA/133 abar m1024@0xfe1ffc00 port 
0xfe1ffe00 irq 47
[   65.250857] ata8: SATA max UDMA/133 abar m1024@0xfe1ffc00 port 
0xfe1ffe80 irq 47
[   65.250895] ata9: SATA max UDMA/133 abar m1024@0xfe1ffc00 port 
0xfe1fff00 irq 47
[   65.250933] ata10: SATA max UDMA/133 abar m1024@0xfe1ffc00 port 
0xfe1fff80 irq 47
[   65.250996] ehci_hcd 0000:00:12.2: PCI INT B -> GSI 17 (level, low) 
-> IRQ 17
[   65.251056] ehci_hcd 0000:00:12.2: EHCI Host Controller
[   65.251101] ehci_hcd 0000:00:12.2: new USB bus registered, assigned 
bus number 1
[   65.251148] ehci_hcd 0000:00:12.2: applying AMD 
SB700/SB800/Hudson-2/3 EHCI dummy qh workaround
[   65.251199] QUIRK: Enable AMD PLL fix
[   65.251208] ehci_hcd 0000:00:12.2: debug port 1
[   65.251257] ehci_hcd 0000:00:12.2: irq 17, io mem 0xfe1ff800
[   65.270037] ehci_hcd 0000:00:12.2: USB 2.0 started, EHCI 1.00
[   65.270214] hub 1-0:1.0: USB hub found
[   65.270254] hub 1-0:1.0: 5 ports detected
[   65.270351] ehci_hcd 0000:00:13.2: PCI INT B -> GSI 17 (level, low) 
-> IRQ 17
[   65.270404] ehci_hcd 0000:00:13.2: EHCI Host Controller
[   65.270454] ehci_hcd 0000:00:13.2: new USB bus registered, assigned 
bus number 2
[   65.270496] ehci_hcd 0000:00:13.2: applying AMD 
SB700/SB800/Hudson-2/3 EHCI dummy qh workaround
[   65.270551] ehci_hcd 0000:00:13.2: debug port 1
[   65.270595] ehci_hcd 0000:00:13.2: irq 17, io mem 0xfe1ff400
[   65.270605] megasas_init_mfi: fw_support_ieee=67108864
[   65.270606] megasas: INIT adapter done
[   65.290144] ehci_hcd 0000:00:13.2: USB 2.0 started, EHCI 1.00
[   65.290258] hub 2-0:1.0: USB hub found
[   65.290292] hub 2-0:1.0: 5 ports detected
[   65.290362] ehci_hcd 0000:00:16.2: PCI INT B -> GSI 17 (level, low) 
-> IRQ 17
[   65.290407] ehci_hcd 0000:00:16.2: EHCI Host Controller
[   65.290440] ehci_hcd 0000:00:16.2: new USB bus registered, assigned 
bus number 3
[   65.290478] ehci_hcd 0000:00:16.2: applying AMD 
SB700/SB800/Hudson-2/3 EHCI dummy qh workaround
[   65.290533] ehci_hcd 0000:00:16.2: debug port 1
[   65.290573] ehci_hcd 0000:00:16.2: irq 17, io mem 0xfe1ff000
[   65.320044] ehci_hcd 0000:00:16.2: USB 2.0 started, EHCI 1.00
[   65.320151] hub 3-0:1.0: USB hub found
[   65.320184] hub 3-0:1.0: 4 ports detected
[   65.348414] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
[   65.348479] ohci_hcd 0000:00:12.0: PCI INT A -> GSI 18 (level, low) 
-> IRQ 18
[   65.348539] ohci_hcd 0000:00:12.0: OHCI Host Controller
[   65.348574] ohci_hcd 0000:00:12.0: new USB bus registered, assigned 
bus number 4
[   65.348629] ohci_hcd 0000:00:12.0: irq 18, io mem 0xfe1fe000
[   65.390058] scsi0 : LSI SAS based MegaRAID driver
[   65.402010] scsi 0:0:10:0: Direct-Access     ATA      WDC 
WD20EARS-60M AB51 PQ: 0 ANSI: 5
[   65.402165] sd 0:0:10:0: Attached scsi generic sg0 type 0
[   65.404137] hub 4-0:1.0: USB hub found
[   65.404173] hub 4-0:1.0: 5 ports detected
[   65.404224] ohci_hcd 0000:00:13.0: PCI INT A -> GSI 18 (level, low) 
-> IRQ 18
[   65.404265] ohci_hcd 0000:00:13.0: OHCI Host Controller
[   65.404298] ohci_hcd 0000:00:13.0: new USB bus registered, assigned 
bus number 5
[   65.404342] ohci_hcd 0000:00:13.0: irq 18, io mem 0xfe1fd000
[   65.408855] sd 0:0:10:0: [sda] physical block alignment offset: 4096
[   65.408893] sd 0:0:10:0: [sda] 3907029168 512-byte logical blocks: 
(2.00 TB/1.81 TiB)
[   65.408928] sd 0:0:10:0: [sda] 4096-byte physical blocks
[   65.410683] scsi 0:0:11:0: Direct-Access     ATA      WDC 
WD20EARS-00M AB51 PQ: 0 ANSI: 5
[   65.410807] sd 0:0:11:0: Attached scsi generic sg1 type 0
[   65.415759] sd 0:0:11:0: [sdb] 3907029168 512-byte logical blocks: 
(2.00 TB/1.81 TiB)
[   65.423407] scsi 0:0:12:0: Direct-Access     ATA      WDC 
WD20EARS-60M AB51 PQ: 0 ANSI: 5
[   65.423536] sd 0:0:12:0: Attached scsi generic sg2 type 0
[   65.430537] sd 0:0:12:0: [sdc] physical block alignment offset: 4096
[   65.430574] sd 0:0:12:0: [sdc] 3907029168 512-byte logical blocks: 
(2.00 TB/1.81 TiB)
[   65.430608] sd 0:0:12:0: [sdc] 4096-byte physical blocks
[   65.437120] sd 0:0:11:0: [sdb] Write Protect is off
[   65.437156] sd 0:0:11:0: [sdb] Mode Sense: 73 00 00 08
[   65.437730] sd 0:0:10:0: [sda] Write Protect is off
[   65.437766] sd 0:0:10:0: [sda] Mode Sense: 73 00 00 08
[   65.441716] scsi 0:0:13:0: Direct-Access     ATA      WDC 
WD20EARS-60M AB51 PQ: 0 ANSI: 5
[   65.441835] sd 0:0:13:0: Attached scsi generic sg3 type 0
[   65.444683] sd 0:0:11:0: [sdb] Write cache: enabled, read cache: 
enabled, doesn't support DPO or FUA
[   65.445330] scsi 0:0:18:0: Direct-Access     SEAGATE  ST3300655SS 
   S515 PQ: 0 ANSI: 5
[   65.445467] sd 0:0:18:0: Attached scsi generic sg4 type 0
[   65.451625] sd 0:0:10:0: [sda] Write cache: enabled, read cache: 
enabled, doesn't support DPO or FUA
[   65.451711] sd 0:0:13:0: [sdd] physical block alignment offset: 4096
[   65.451747] sd 0:0:13:0: [sdd] 3907029168 512-byte logical blocks: 
(2.00 TB/1.81 TiB)
[   65.451790] sd 0:0:13:0: [sdd] 4096-byte physical blocks
[   65.452285] sd 0:0:18:0: [sde] 585937500 512-byte logical blocks: 
(300 GB/279 GiB)
[   65.453321] scsi 0:0:19:0: Direct-Access     SEAGATE  ST3300655SS 
   A005 PQ: 0 ANSI: 5
[   65.453456] sd 0:0:19:0: Attached scsi generic sg5 type 0
[   65.458128] sd 0:0:18:0: [sde] Write Protect is off
[   65.458163] sd 0:0:18:0: [sde] Mode Sense: b3 00 10 08
[   65.459325] sd 0:0:19:0: [sdf] 585937500 512-byte logical blocks: 
(300 GB/279 GiB)
[   65.459448] scsi 0:0:20:0: Direct-Access     SEAGATE  ST3300655SS 
   A005 PQ: 0 ANSI: 5
[   65.459583] sd 0:0:20:0: Attached scsi generic sg6 type 0
[   65.464128] hub 5-0:1.0: USB hub found
[   65.464165] hub 5-0:1.0: 5 ports detected
[   65.464214] ohci_hcd 0000:00:14.5: PCI INT C -> GSI 18 (level, low) 
-> IRQ 18
[   65.464255] ohci_hcd 0000:00:14.5: OHCI Host Controller
[   65.464288] ohci_hcd 0000:00:14.5: new USB bus registered, assigned 
bus number 6
[   65.464331] ohci_hcd 0000:00:14.5: irq 18, io mem 0xfe1fc000
[   65.464611] sd 0:0:18:0: [sde] Write cache: enabled, read cache: 
enabled, supports DPO and FUA
[   65.465337] sd 0:0:20:0: [sdg] 585937500 512-byte logical blocks: 
(300 GB/279 GiB)
[   65.465748] scsi 0:0:21:0: Direct-Access     SEAGATE  ST3300655SS 
   S515 PQ: 0 ANSI: 5
[   65.465906] sd 0:0:21:0: Attached scsi generic sg7 type 0
[   65.470327] sd 0:0:19:0: [sdf] Write Protect is off
[   65.470362] sd 0:0:19:0: [sdf] Mode Sense: b3 00 10 08
[   65.476731] sd 0:0:21:0: [sdh] 585937500 512-byte logical blocks: 
(300 GB/279 GiB)
[   65.476846] sd 0:0:20:0: [sdg] Write Protect is off
[   65.476892] sd 0:0:20:0: [sdg] Mode Sense: b3 00 10 08
[   65.476916] sd 0:0:19:0: [sdf] Write cache: enabled, read cache: 
enabled, supports DPO and FUA
[   65.482548] sd 0:0:21:0: [sdh] Write Protect is off
[   65.482586] sd 0:0:21:0: [sdh] Mode Sense: b3 00 10 08
[   65.483436] sd 0:0:20:0: [sdg] Write cache: enabled, read cache: 
enabled, supports DPO and FUA
[   65.488248] sd 0:0:21:0: [sdh] Write cache: enabled, read cache: 
enabled, supports DPO and FUA
[   65.494395]  sde: unknown partition table
[   65.501262]  sdb: unknown partition table
[   65.503732] sd 0:0:12:0: [sdc] Write Protect is off
[   65.503776] sd 0:0:12:0: [sdc] Mode Sense: 73 00 00 08
[   65.505389]  sdf: unknown partition table
[   65.506463] sd 0:0:18:0: [sde] Attached SCSI disk
[   65.511538]  sdh: unknown partition table
[   65.512582]  sda: unknown partition table
[   65.519104] sd 0:0:12:0: [sdc] Write cache: enabled, read cache: 
enabled, doesn't support DPO or FUA
[   65.521490]  sdg: unknown partition table
[   65.524125] hub 6-0:1.0: USB hub found
[   65.524162] hub 6-0:1.0: 2 ports detected
[   65.524211] ohci_hcd 0000:00:16.0: PCI INT A -> GSI 18 (level, low) 
-> IRQ 18
[   65.524252] ohci_hcd 0000:00:16.0: OHCI Host Controller
[   65.524285] ohci_hcd 0000:00:16.0: new USB bus registered, assigned 
bus number 7
[   65.524332] ohci_hcd 0000:00:16.0: irq 18, io mem 0xfe1f7000
[   65.526953] sd 0:0:19:0: [sdf] Attached SCSI disk
[   65.529657] sd 0:0:21:0: [sdh] Attached SCSI disk
[   65.531531] sd 0:0:13:0: [sdd] Write Protect is off
[   65.531569] sd 0:0:13:0: [sdd] Mode Sense: 73 00 00 08
[   65.532606] sd 0:0:20:0: [sdg] Attached SCSI disk
[   65.537818] megasas: 0x1000:0x0073:0x1014:0x03b1: bus 5:slot 0:func 0
[   65.537887] megaraid_sas 0000:05:00.0: PCI INT A -> GSI 19 (level, 
low) -> IRQ 19
[   65.537923] megaraid_sas 0000:05:00.0: setting latency timer to 64
[   65.537974] megasas: FW now in Ready state
[   65.538034] megaraid_sas 0000:05:00.0: irq 48 for MSI/MSI-X
[   65.543918] sd 0:0:13:0: [sdd] Write cache: enabled, read cache: 
enabled, doesn't support DPO or FUA
[   65.545106] sd 0:0:11:0: [sdb] Attached SCSI disk
[   65.560038] megasas_init_mfi: fw_support_ieee=67108864
[   65.560079] megasas: INIT adapter done
[   65.563406] sd 0:0:10:0: [sda] Attached SCSI disk
[   65.570045]  sdc: unknown partition table
[   65.584124] hub 7-0:1.0: USB hub found
[   65.584161] hub 7-0:1.0: 4 ports detected
[   65.593514]  sdd: unknown partition table
[   65.600047] ata9: SATA link down (SStatus 0 SControl 300)
[   65.600107] ata7: SATA link down (SStatus 0 SControl 300)
[   65.600170] ata8: SATA link down (SStatus 0 SControl 300)
[   65.600261] ata10: SATA link down (SStatus 0 SControl 300)
[   65.623341] sd 0:0:12:0: [sdc] Attached SCSI disk
[   65.642020] sd 0:0:13:0: [sdd] Attached SCSI disk
[   65.650037] scsi11 : LSI SAS based MegaRAID driver
[   65.661988] scsi 11:0:9:0: Direct-Access     ATA      WDC 
WD20EARS-60M AB51 PQ: 0 ANSI: 5
[   65.662113] sd 11:0:9:0: Attached scsi generic sg8 type 0
[   65.668949] sd 11:0:9:0: [sdi] physical block alignment offset: 4096
[   65.668986] sd 11:0:9:0: [sdi] 3907029168 512-byte logical blocks: 
(2.00 TB/1.81 TiB)
[   65.669020] sd 11:0:9:0: [sdi] 4096-byte physical blocks
[   65.675059] scsi 11:0:10:0: Direct-Access     ATA      WDC 
WD20EARS-60M AB51 PQ: 0 ANSI: 5
[   65.675182] sd 11:0:10:0: Attached scsi generic sg9 type 0
[   65.681812] sd 11:0:10:0: [sdj] physical block alignment offset: 4096
[   65.681849] sd 11:0:10:0: [sdj] 3907029168 512-byte logical blocks: 
(2.00 TB/1.81 TiB)
[   65.681884] sd 11:0:10:0: [sdj] 4096-byte physical blocks
[   65.698957] sd 11:0:9:0: [sdi] Write Protect is off
[   65.698993] sd 11:0:9:0: [sdi] Mode Sense: 73 00 00 08
[   65.709639] scsi 11:0:13:0: Direct-Access     ATA      WDC 
WD20EARS-60M AB51 PQ: 0 ANSI: 5
[   65.709759] sd 11:0:13:0: Attached scsi generic sg10 type 0
[   65.711168] sd 11:0:9:0: [sdi] Write cache: enabled, read cache: 
enabled, doesn't support DPO or FUA
[   65.716961] sd 11:0:13:0: [sdk] physical block alignment offset: 4096
[   65.716998] sd 11:0:13:0: [sdk] 3907029168 512-byte logical blocks: 
(2.00 TB/1.81 TiB)
[   65.717035] sd 11:0:13:0: [sdk] 4096-byte physical blocks
[   65.742927] scsi 11:0:14:0: Direct-Access     ATA      WDC 
WD20EARS-60M AB51 PQ: 0 ANSI: 5
[   65.743040] sd 11:0:14:0: Attached scsi generic sg11 type 0
[   65.747118] sd 11:0:13:0: [sdk] Write Protect is off
[   65.747153] sd 11:0:13:0: [sdk] Mode Sense: 73 00 00 08
[   65.749113] sd 11:0:14:0: [sdl] physical block alignment offset: 4096
[   65.749150] sd 11:0:14:0: [sdl] 3907029168 512-byte logical blocks: 
(2.00 TB/1.81 TiB)
[   65.749184] sd 11:0:14:0: [sdl] 4096-byte physical blocks
[   65.749947] sd 11:0:10:0: [sdj] Write Protect is off
[   65.749982] sd 11:0:10:0: [sdj] Mode Sense: 73 00 00 08
[   65.750068] ata1: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
[   65.759269] sd 11:0:13:0: [sdk] Write cache: enabled, read cache: 
enabled, doesn't support DPO or FUA
[   65.759275]  sdi: unknown partition table
[   65.761631] sd 11:0:10:0: [sdj] Write cache: enabled, read cache: 
enabled, doesn't support DPO or FUA
[   65.777351] sd 11:0:14:0: [sdl] Write Protect is off
[   65.777386] sd 11:0:14:0: [sdl] Mode Sense: 73 00 00 08
[   65.787708] sd 11:0:14:0: [sdl] Write cache: enabled, read cache: 
enabled, doesn't support DPO or FUA
[   65.790285] ata1.00: ATA-8: FM-25S2S-60GBP2, 2.1, max UDMA/133
[   65.790321] ata1.00: 117231408 sectors, multi 1: LBA48 NCQ (depth 31/32)
[   65.800049] ata6: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[   65.800102] ata5: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[   65.803492] scsi 11:0:15:0: Direct-Access     ATA      WDC 
WD20EARS-60M AB51 PQ: 0 ANSI: 5
[   65.803611] sd 11:0:15:0: Attached scsi generic sg12 type 0
[   65.807891] sd 11:0:9:0: [sdi] Attached SCSI disk
[   65.808134]  sdj: unknown partition table
[   65.808954] ata5.00: ATA-8: WDC WD1002FAEX-00Z3A0, 05.01D05, max UDMA/133
[   65.808989] ata5.00: 1953525168 sectors, multi 0: LBA48 NCQ (depth 
31/32), AA
[   65.809689]  sdk: unknown partition table
[   65.810820] sd 11:0:15:0: [sdm] physical block alignment offset: 4096
[   65.810865] sd 11:0:15:0: [sdm] 3907029168 512-byte logical blocks: 
(2.00 TB/1.81 TiB)
[   65.810902] sd 11:0:15:0: [sdm] 4096-byte physical blocks
[   65.811042] ata5.00: configured for UDMA/133
[   65.812159] ata6.00: ATA-8: WDC WD1002FAEX-00Z3A0, 05.01D05, max UDMA/133
[   65.812194] ata6.00: 1953525168 sectors, multi 0: LBA48 NCQ (depth 
31/32), AA
[   65.814325] ata6.00: configured for UDMA/133
[   65.833497]  sdl: unknown partition table
[   65.841554] sd 11:0:15:0: [sdm] Write Protect is off
[   65.841590] sd 11:0:15:0: [sdm] Mode Sense: 73 00 00 08
[   65.850283] ata1.00: configured for UDMA/133
[   65.850387] scsi 1:0:0:0: Direct-Access     ATA      FM-25S2S-60GBP2 
  2.1  PQ: 0 ANSI: 5
[   65.850502] sd 1:0:0:0: Attached scsi generic sg13 type 0
[   65.850529] sd 1:0:0:0: [sdn] 117231408 512-byte logical blocks: 
(60.0 GB/55.8 GiB)
[   65.850653] sd 1:0:0:0: [sdn] Write Protect is off
[   65.850687] sd 1:0:0:0: [sdn] Mode Sense: 00 3a 00 00
[   65.850712] sd 1:0:0:0: [sdn] Write cache: enabled, read cache: 
enabled, doesn't support DPO or FUA
[   65.851123]  sdn: unknown partition table
[   65.851320] sd 1:0:0:0: [sdn] Attached SCSI disk
[   65.853501] sd 11:0:15:0: [sdm] Write cache: enabled, read cache: 
enabled, doesn't support DPO or FUA
[   65.853583] sd 11:0:10:0: [sdj] Attached SCSI disk
[   65.860607] sd 11:0:13:0: [sdk] Attached SCSI disk
[   65.876691] sd 11:0:14:0: [sdl] Attached SCSI disk
[   65.890059] usb 4-2: new full speed USB device number 2 using ohci_hcd
[   65.902817]  sdm: unknown partition table
[   65.921026] scsi 11:0:16:0: Direct-Access     ATA      WDC 
WD20EARS-60M AB51 PQ: 0 ANSI: 5
[   65.921149] sd 11:0:16:0: Attached scsi generic sg14 type 0
[   65.928106] sd 11:0:16:0: [sdo] physical block alignment offset: 4096
[   65.928143] sd 11:0:16:0: [sdo] 3907029168 512-byte logical blocks: 
(2.00 TB/1.81 TiB)
[   65.928177] sd 11:0:16:0: [sdo] 4096-byte physical blocks
[   65.953117] sd 11:0:15:0: [sdm] Attached SCSI disk
[   66.030173] sd 11:0:16:0: [sdo] Write Protect is off
[   66.030211] sd 11:0:16:0: [sdo] Mode Sense: 73 00 00 08
[   66.043679] sd 11:0:16:0: [sdo] Write cache: enabled, read cache: 
enabled, doesn't support DPO or FUA
[   66.092356]  sdo: unknown partition table
[   66.182327] sd 11:0:16:0: [sdo] Attached SCSI disk
[   66.200059] ata2: SATA link down (SStatus 0 SControl 300)
[   66.550061] ata3: SATA link down (SStatus 0 SControl 300)
[   66.900053] ata4: SATA link down (SStatus 0 SControl 300)
[   66.900198] scsi 2:0:0:0: Direct-Access     ATA      WDC WD1002FAEX-0 
05.0 PQ: 0 ANSI: 5
[   66.900325] sd 2:0:0:0: [sdp] 1953525168 512-byte logical blocks: 
(1.00 TB/931 GiB)
[   66.900327] sd 2:0:0:0: Attached scsi generic sg15 type 0
[   66.900427] scsi 5:0:0:0: Direct-Access     ATA      WDC WD1002FAEX-0 
05.0 PQ: 0 ANSI: 5
[   66.900464] sd 2:0:0:0: [sdp] Write Protect is off
[   66.900467] sd 2:0:0:0: [sdp] Mode Sense: 00 3a 00 00
[   66.900491] sd 2:0:0:0: [sdp] Write cache: enabled, read cache: 
enabled, doesn't support DPO or FUA
[   66.900643] sd 5:0:0:0: [sdq] 1953525168 512-byte logical blocks: 
(1.00 TB/931 GiB)
[   66.900645] sd 5:0:0:0: Attached scsi generic sg16 type 0
[   66.900773] sd 5:0:0:0: [sdq] Write Protect is off
[   66.900808] sd 5:0:0:0: [sdq] Mode Sense: 00 3a 00 00
[   66.900829] sd 5:0:0:0: [sdq] Write cache: enabled, read cache: 
enabled, doesn't support DPO or FUA
[   66.904434]  sdq: sdq1 sdq2
[   66.904681] sd 5:0:0:0: [sdq] Attached SCSI disk
[   66.909577]  sdp: sdp1 sdp2
[   66.909810] sd 2:0:0:0: [sdp] Attached SCSI disk
[   67.191179] md: md1 stopped.
[   67.191764] md: bind<sdq1>
[   67.191895] md: bind<sdp1>
[   67.192591] bio: create slab <bio-1> at 1
[   67.192670] md/raid1:md1: active with 2 out of 2 mirrors
[   67.192714] md1: detected capacity change from 0 to 524222464
[   67.200588]  md1: unknown partition table
[   67.207511] md: md2 stopped.
[   67.208237] md: bind<sdq2>
[   67.209779] md: bind<sdp2>
[   67.210543] md/raid10:md2: active with 2 out of 2 devices
[   67.210673] created bitmap (8 pages) for device md2
[   67.210934] md2: bitmap initialized from disk: read 1/1 pages, set 2 
of 14897 bits
[   67.219474] md2: detected capacity change from 0 to 999677755392
[   67.224116]  md2: p1 p2 p3
[   67.230750] md: md10 stopped.
[   67.244399] md: bind<sde>
[   67.244619] md: bind<sdf>
[   67.244823] md: bind<sdg>
[   67.248205] md: bind<sdh>
[   67.248955] md/raid10:md10: active with 4 out of 4 devices
[   67.252191] created bitmap (5 pages) for device md10
[   67.252505] md10: bitmap initialized from disk: read 1/1 pages, set 0 
of 8941 bits
[   67.264422] md10: detected capacity change from 0 to 599997284352
[   67.271735]  md10: unknown partition table
[   67.296795] md: md0 stopped.
[   67.398685] md: bind<sdd>
[   67.404514] md: bind<sdm>
[   67.404703] md: bind<sdb>
[   67.408534] md: bind<sdj>
[   67.411132] md: bind<sdi>
[   67.411724] md: bind<sdl>
[   67.430316] md: bind<sda>
[   67.430607] md: bind<sdc>
[   67.430770] md: bind<sdk>
[   67.430922] md: bind<sdo>
[   67.431657] md/raid:md0: device sdo operational as raid disk 0
[   67.431694] md/raid:md0: device sdk operational as raid disk 9
[   67.431726] md/raid:md0: device sdc operational as raid disk 8
[   67.431758] md/raid:md0: device sda operational as raid disk 7
[   67.431793] md/raid:md0: device sdl operational as raid disk 6
[   67.431824] md/raid:md0: device sdi operational as raid disk 5
[   67.431856] md/raid:md0: device sdj operational as raid disk 4
[   67.431891] md/raid:md0: device sdb operational as raid disk 3
[   67.431925] md/raid:md0: device sdm operational as raid disk 2
[   67.431956] md/raid:md0: device sdd operational as raid disk 1
[   67.432356] md/raid:md0: allocated 10572kB
[   67.432418] md/raid:md0: raid level 6 active with 10 out of 10 
devices, algorithm 2
[   67.432457] RAID conf printout:
[   67.432458]  --- level:6 rd:10 wd:10
[   67.432459]  disk 0, o:1, dev:sdo
[   67.432461]  disk 1, o:1, dev:sdd
[   67.432462]  disk 2, o:1, dev:sdm
[   67.432463]  disk 3, o:1, dev:sdb
[   67.432463]  disk 4, o:1, dev:sdj
[   67.432464]  disk 5, o:1, dev:sdi
[   67.432465]  disk 6, o:1, dev:sdl
[   67.432466]  disk 7, o:1, dev:sda
[   67.432467]  disk 8, o:1, dev:sdc
[   67.432468]  disk 9, o:1, dev:sdk
[   67.466604] created bitmap (15 pages) for device md0
[   67.466953] md0: bitmap initialized from disk: read 1/1 pages, set 0 
of 29809 bits
[   67.472708] md0: detected capacity change from 0 to 16003181314048
[   67.511792]  md0: unknown partition table
[   67.680098] EXT4-fs (md2p1): INFO: recovery required on readonly 
filesystem
[   67.680135] EXT4-fs (md2p1): write access will be enabled during recovery
[   69.810256] EXT4-fs (md2p1): recovery complete
[   69.810604] EXT4-fs (md2p1): mounted filesystem with ordered data 
mode. Opts: (null)
[   70.548329] udev[1803]: starting version 164
[   70.676012] usbcore: registered new interface driver usbserial
[   70.676059] USB Serial support registered for generic
[   70.676103] usbcore: registered new interface driver usbserial_generic
[   70.676147] usbserial: USB Serial Driver core
[   70.683832] USB Serial support registered for pl2303
[   70.683881] pl2303 4-2:1.0: pl2303 converter detected
[   70.698118] piix4_smbus 0000:00:14.0: SMBus Host Controller at 0xb00, 
revision 0
[   70.701760] xhci_hcd 0000:02:00.0: PCI INT A -> GSI 17 (level, low) 
-> IRQ 17
[   70.701834] xhci_hcd 0000:02:00.0: setting latency timer to 64
[   70.701837] xhci_hcd 0000:02:00.0: xHCI Host Controller
[   70.701877] xhci_hcd 0000:02:00.0: new USB bus registered, assigned 
bus number 8
[   70.702046] xhci_hcd 0000:02:00.0: irq 17, io mem 0xfe4fe000
[   70.702134] xhci_hcd 0000:02:00.0: irq 49 for MSI/MSI-X
[   70.702138] xhci_hcd 0000:02:00.0: irq 50 for MSI/MSI-X
[   70.702140] xhci_hcd 0000:02:00.0: irq 51 for MSI/MSI-X
[   70.702142] xhci_hcd 0000:02:00.0: irq 52 for MSI/MSI-X
[   70.702144] xhci_hcd 0000:02:00.0: irq 53 for MSI/MSI-X
[   70.702146] xhci_hcd 0000:02:00.0: irq 54 for MSI/MSI-X
[   70.702148] xhci_hcd 0000:02:00.0: irq 55 for MSI/MSI-X
[   70.702325] xHCI xhci_add_endpoint called for root hub
[   70.702327] xHCI xhci_check_bandwidth called for root hub
[   70.702347] hub 8-0:1.0: USB hub found
[   70.702384] hub 8-0:1.0: 2 ports detected
[   70.702526] xhci_hcd 0000:02:00.0: xHCI Host Controller
[   70.702567] xhci_hcd 0000:02:00.0: new USB bus registered, assigned 
bus number 9
[   70.704246] usb 4-2: pl2303 converter now attached to ttyUSB0
[   70.704320] usbcore: registered new interface driver pl2303
[   70.704367] pl2303: Prolific PL2303 USB to serial adaptor driver
[   70.705493] xHCI xhci_add_endpoint called for root hub
[   70.705495] xHCI xhci_check_bandwidth called for root hub
[   70.705526] hub 9-0:1.0: USB hub found
[   70.705569] hub 9-0:1.0: 2 ports detected
[   71.350688] EXT4-fs (md2p1): re-mounted. Opts: (null)
[   71.586339] EXT4-fs (md2p1): re-mounted. Opts: 
errors=remount-ro,commit=30
[   71.688584] kvm: Nested Virtualization enabled
[   71.688624] kvm: Nested Paging enabled
[   71.715947] powernow-k8: Found 1 AMD Phenom(tm) II X6 1090T Processor 
(6 cpu cores) (version 2.20.00)
[   71.715994] powernow-k8: Core Performance Boosting: on.
[   71.716050] powernow-k8:    0 : pstate 0 (3200 MHz)
[   71.716081] powernow-k8:    1 : pstate 1 (2400 MHz)
[   71.716112] powernow-k8:    2 : pstate 2 (1600 MHz)
[   71.716144] powernow-k8:    3 : pstate 3 (800 MHz)
[   71.741872] w83627ehf: Found NCT6775F chip at 0x290
[  113.081227] fuse init (API version 7.16)
[  113.463322] EXT4-fs (md2p3): mounted filesystem with ordered data 
mode. Opts: commit=30
[  113.523249] EXT4-fs (md2p2): mounted filesystem with ordered data 
mode. Opts: commit=30
[  113.603188] EXT4-fs (md10): mounted filesystem with ordered data 
mode. Opts: commit=30
[  114.034563] EXT4-fs (md0): mounted filesystem with ordered data mode. 
Opts: commit=30,acl
[  116.104972] Adding 20971516k swap on /raid10/SWAP/srv.swap. 
Priority:-1 extents:13 across:22421500k
[  116.578585] NET: Registered protocol family 15
[  138.220721] device br1_dummy entered promiscuous mode
[  180.833074] u32 classifier
[  180.833129]     Actions configured
[  191.222339] PPP generic driver version 2.4.2
[  192.068503] NET: Registered protocol family 24
[  192.759974] ip_tables: (C) 2000-2006 Netfilter Core Team
[  192.772071] nf_conntrack version 0.5.0 (16384 buckets, 65536 max)
[  193.055922] HTB: quantum of class 10040 is small. Consider r2q change.
[  204.360085] sshd (3856): /proc/3856/oom_adj is deprecated, please use 
/proc/3856/oom_score_adj instead.
[  212.869525] NFSD: Using /var/lib/nfs/v4recovery as the NFSv4 state 
recovery directory
[  212.875925] NFSD: starting 90-second grace period

brad@srv:~$ lsmod
Module                  Size  Used by
nfs                   294438  1
xt_length               1200  2
xt_CLASSIFY             1075  5
sch_sfq                 6676  2
xt_CHECKSUM             1213  2
iptable_mangle          1471  1
xt_TCPMSS               2551  2
ipt_REJECT              2229  1
ipt_MASQUERADE          1743  10
ipt_REDIRECT            1117  1
xt_recent               8207  2
xt_state                1210  6
xt_tcpudp               2451  28
iptable_filter          1426  1
iptable_nat             3977  1
nf_nat                 16789  3 ipt_MASQUERADE,ipt_REDIRECT,iptable_nat
nf_conntrack_ipv4      11852  9 iptable_nat,nf_nat
nf_conntrack           60821  5 
ipt_MASQUERADE,xt_state,iptable_nat,nf_nat,nf_conntrack_ipv4
nf_defrag_ipv4          1401  1 nf_conntrack_ipv4
ip_tables              16538  3 iptable_mangle,iptable_filter,iptable_nat
x_tables               20342  14 
xt_length,xt_CLASSIFY,xt_CHECKSUM,iptable_mangle,xt_TCPMSS,ipt_REJECT,ipt_MASQUERADE,ipt_REDIRECT,xt_recent,xt_state,xt_tcpudp,iptable_filter,iptable_nat,ip_tables
pppoe                   9558  2
pppox                   2156  1 pppoe
ppp_generic            24227  6 pppoe,pppox
slhc                    5277  1 ppp_generic
cls_u32                 6356  6
sch_htb                14352  2
deflate                 1921  0
zlib_deflate           21196  1 deflate
des_generic            16119  0
cbc                     2705  0
ecb                     1959  0
crypto_blkcipher       13629  2 cbc,ecb
sha1_generic            2079  0
md5                     3985  0
hmac                    2961  0
crypto_hash            14471  3 sha1_generic,md5,hmac
cryptomgr               2652  0
aead                    6121  1 cryptomgr
crypto_algapi          15273  9 
deflate,des_generic,cbc,ecb,crypto_blkcipher,hmac,crypto_hash,cryptomgr,aead
af_key                 27356  0
fuse                   66684  1
w83627ehf              32036  0
hwmon_vid               2851  1 w83627ehf
vhost_net              17239  5
powernow_k8            12884  0
mperf                   1247  1 powernow_k8
kvm_amd                54109  20
kvm                   238605  1 kvm_amd
xhci_hcd               66960  0
k10temp                 3199  0
i2c_piix4               8375  0
pl2303                 12716  1
usbserial              34420  3 pl2303
ohci_hcd               17897  0
ehci_hcd               33721  0
usbcore               131192  5 xhci_hcd,pl2303,usbserial,ohci_hcd,ehci_hcd
ahci                   20732  4
libahci                21186  1 ahci
sata_mv                26923  0
megaraid_sas           71785  14

^ permalink raw reply	[flat|nested] 111+ messages in thread

end of thread, other threads:[~2011-08-22 11:46 UTC | newest]

Thread overview: 111+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2011-05-31  1:24 KVM induced panic on 2.6.38[2367] & 2.6.39 Brad Campbell
2011-05-31  5:47 ` Borislav Petkov
2011-05-31  5:47   ` Borislav Petkov
2011-05-31  9:26   ` Brad Campbell
2011-05-31  9:26     ` Brad Campbell
2011-05-31 10:38     ` Borislav Petkov
2011-05-31 10:38       ` Borislav Petkov
2011-05-31 14:24       ` Brad Campbell
2011-05-31 14:24         ` Brad Campbell
2011-05-31 14:24         ` Brad Campbell
2011-05-31 22:31         ` Hugh Dickins
2011-05-31 22:31           ` Hugh Dickins
2011-06-01  0:18           ` Brad Campbell
2011-06-01  0:18             ` Brad Campbell
2011-06-01  0:37           ` Brad Campbell
2011-06-01  0:37             ` Brad Campbell
2011-06-01  1:15             ` Andrea Arcangeli
2011-06-01  1:15               ` Andrea Arcangeli
2011-06-01  2:03               ` Brad Campbell
2011-06-01  2:03                 ` Brad Campbell
2011-06-01  4:52               ` Hugh Dickins
2011-06-01  4:52                 ` Hugh Dickins
2011-06-01  6:31                 ` Brad Campbell
2011-06-01  6:31                   ` Brad Campbell
2011-06-01  6:56                   ` Avi Kivity
2011-06-01  6:56                     ` Avi Kivity
2011-06-01  9:29                     ` Brad Campbell
2011-06-01  9:29                       ` Brad Campbell
2011-06-01  9:29                       ` Brad Campbell
2011-06-01  9:40                       ` Avi Kivity
2011-06-01  9:40                         ` Avi Kivity
2011-06-01  9:41                         ` Avi Kivity
2011-06-01  9:41                           ` Avi Kivity
2011-06-01 10:53                           ` Brad Campbell
2011-06-01 10:53                             ` Brad Campbell
2011-06-01 11:09                             ` Avi Kivity
2011-06-01 11:09                               ` Avi Kivity
2011-06-01 11:18                             ` CaT
2011-06-01 11:18                               ` CaT
2011-06-01 11:52                               ` Brad Campbell
2011-06-01 11:52                                 ` Brad Campbell
2011-06-01 23:03                                 ` CaT
2011-06-01 23:03                                   ` CaT
2011-06-03 13:38                                   ` Brad Campbell
2011-06-03 13:38                                     ` Brad Campbell
2011-06-03 15:50                                     ` Bernhard Held
2011-06-03 15:50                                       ` Bernhard Held
2011-06-03 15:50                                       ` Bernhard Held
2011-06-03 16:07                                       ` Brad Campbell
2011-06-03 16:07                                         ` Brad Campbell
2011-06-06 20:10                                         ` Bart De Schuymer
2011-06-06 20:10                                           ` Bart De Schuymer
2011-06-06 20:23                                           ` Eric Dumazet
2011-06-06 20:23                                             ` Eric Dumazet
2011-06-06 20:23                                             ` Eric Dumazet
2011-06-07  3:33                                           ` Brad Campbell
2011-06-07  3:33                                             ` Brad Campbell
2011-06-07 13:30                                             ` Patrick McHardy
2011-06-07 13:30                                               ` Patrick McHardy
2011-06-07 14:40                                               ` Brad Campbell
2011-06-07 14:40                                                 ` Brad Campbell
2011-06-07 15:35                                                 ` Patrick McHardy
2011-06-07 15:35                                                   ` Patrick McHardy
2011-06-07 18:31                                                   ` Eric Dumazet
2011-06-07 18:31                                                     ` Eric Dumazet
2011-06-07 18:31                                                     ` Eric Dumazet
2011-06-07 22:57                                                     ` Patrick McHardy
2011-06-07 22:57                                                       ` Patrick McHardy
2011-06-07 22:57                                                       ` Patrick McHardy
2011-06-08  0:18                                                       ` Brad Campbell
2011-06-08  0:18                                                         ` Brad Campbell
2011-06-08  0:18                                                         ` Brad Campbell
2011-06-08  3:59                                                         ` Eric Dumazet
2011-06-08  3:59                                                           ` Eric Dumazet
2011-06-08  3:59                                                           ` Eric Dumazet
2011-06-08 17:02                                                           ` Brad Campbell
2011-06-08 17:02                                                             ` Brad Campbell
2011-06-08 17:02                                                             ` Brad Campbell
2011-06-08 21:22                                                             ` Eric Dumazet
2011-06-08 21:22                                                               ` Eric Dumazet
2011-06-08 21:22                                                               ` Eric Dumazet
2011-06-10  2:52                                                             ` Simon Horman
2011-06-10  2:52                                                               ` Simon Horman
2011-06-10 12:37                                                               ` Mark Lord
2011-06-10 12:37                                                                 ` Mark Lord
2011-06-10 16:43                                                                 ` Henrique de Moraes Holschuh
2011-06-10 16:43                                                                   ` Henrique de Moraes Holschuh
2011-06-12 15:38                                                               ` Avi Kivity
2011-06-12 15:38                                                                 ` Avi Kivity
2011-06-07 23:43                                                   ` Brad Campbell
2011-06-07 23:43                                                     ` Brad Campbell
2011-06-07 18:04                                                 ` Bart De Schuymer
2011-06-07 18:04                                                   ` Bart De Schuymer
2011-06-08  0:15                                                   ` Brad Campbell
2011-06-08  0:15                                                     ` Brad Campbell
2011-06-05  8:14                                     ` Avi Kivity
2011-06-05  8:14                                       ` Avi Kivity
2011-06-05 13:45                                       ` Brad Campbell
2011-06-05 13:45                                         ` Brad Campbell
2011-06-05 13:58                                         ` Avi Kivity
2011-06-05 13:58                                           ` Avi Kivity
2011-06-06 20:22                                         ` Eric Dumazet
2011-06-06 20:22                                           ` Eric Dumazet
2011-06-06 20:22                                           ` Eric Dumazet
2011-06-07 13:27                                           ` Brad Campbell
2011-06-07 13:37                                             ` Eric Dumazet
2011-06-07 15:15                                               ` Brad Campbell
2011-08-20 13:16                                               ` Brad Campbell
2011-08-22  6:36                                                 ` Avi Kivity
2011-08-22  6:45                                                   ` Eric Dumazet
2011-08-22 11:45                                                     ` Brad Campbell

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.