* KVM induced panic on 2.6.38[2367] & 2.6.39
@ 2011-05-31 1:24 Brad Campbell
2011-05-31 5:47 ` Borislav Petkov
0 siblings, 1 reply; 111+ messages in thread
From: Brad Campbell @ 2011-05-31 1:24 UTC (permalink / raw)
To: linux-kernel; +Cc: kvm
G'day all,
I'm running a pretty standard home server
x86_64 Phenom-II 6 Core
16GB DDR 3
I run some virtual machines under that. 3 x Debian 64 Bit, 1 x XP 32
Bit. These run at boot.
When I fire up another XP 32 bit instance and play with it for more than
about 2 minutes, I get the panics included in this mail.
I've included three of them here. The first and third are as booted. The
second was with ksmd disabled just as a data point.
The machine passes every load test and memory test I can throw at it,
but I still can't rule out this being a hardware issue.
Provided I don't start this XP VM the machine is quite stable, but
running this VM will kill it within minutes.
This was tested with qemu-kvm.
The last commit in the git tree was
commit c007db193eb6b2557acb5caf2dc4d7023639e6f3
Author: Avi Kivity <avi@redhat.com>
Date: Sun May 29 09:00:42 2011 -0400
(I pulled it yesterday)
These panics were captured with netconsole to a remote syslog daemon,
and the formatting was ruined, so I've reformatted them by hand prior to
posting.
I've tested and reproduced this on 2.6.38.[2,3,6 & 7] and 2.6.39 obviously.
Can anyone help shed some light on this?
Regards,
Brad
[ 438.632061] general protection fault: 0000 [#1] SMP
[ 438.632196] last sysfs file: /sys/module/x_tables/initstate
[ 438.632242] CPU 4
[ 438.632282] Modules linked in: xt_iprange xt_DSCP xt_length
xt_CLASSIFY sch_sfq xt_CHECKSUM ipt_REJECT ipt_MASQUERADE ipt_REDIRECT
xt_recent xt_state iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack
nf_defrag_ipv4 iptable_filter xt_TCPMSS xt_tcpmss xt_tcpudp
iptable_mangle ip_tables x_tables pppoe pppox ppp_generic slhc cls_u32
sch_htb deflate zlib_deflate des_generic cbc ecb crypto_blkcipher
sha1_generic md5 hmac crypto_hash cryptomgr aead crypto_algapi af_key
fuse netconsole configfs= vhost_net powernow_k8 mperf i2c_nforce2
kvm_amd kvm pl2303 usbserial xhci_hcd k10temp i2c_piix4 usb_storage
usb_libusual ohci_hcd ehci_hcd usbcore ahci libahci r8169 mii sata_mv
megaraid_sas [last unloaded: scsi_wait_scan]
[ 438.634960]
[ 438.635006] Pid: 551, comm: ksmd Not tainted 2.6.39 #3 To Be Filled By
O.E.M. To Be Filled By O.E.M. /880G Extreme3
[ 438.635170] RIP: 0010:[<ffffffff810b4596>] [<ffffffff810b4596>]
remove_rmap_item_from_tree+0x96/0x150
[ 438.635268] RSP: 0018:ffff88041c065e20 EFLAGS: 00010282
[ 438.635314] RAX: ffff8804153bd8b0 RBX: ffff8804176c3fc0 RCX:
0000000000057754
[ 438.635362] RDX: 0000880415418030 RSI: ffff880414b65003 RDI:
ffffea000dde6030
[ 438.635410] RBP: ffff880414b65000 R08: 0000000000057755 R09:
000000001bbde1bd
[ 438.635458] R10: 000000001c28e0c3 R11: 0000000000000002 R12:
ffffea000dde6030
[ 438.635506] R13: ffff8804176c3f80 R14: ffff8804151ed7b0 R15:
ffff88041bf23be0
[ 438.635555] FS: 00007f617772e700(0000) GS:ffff88041fd00000(0000)
knlGS:0000000000000000
[ 438.635607] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 438.635654] CR2: 0000000000e70000 CR3: 0000000001583000 CR4:
00000000000006e0
[ 438.635703] DR0: 0000000000000045 DR1: 0000000000000000 DR2:
0000000000000000
[ 438.635750] DR3: 0000000000000005 DR6: 00000000ffff0ff0 DR7:
0000000000000400
[ 438.635799] Process ksmd (pid: 551, threadinfo ffff88041c064000, task
ffff88041d8caa70)
[ 438.635851] Stack:
[ 438.635893] ffffea000c43c408 ffff8804176c3fc0 000000000000036c
ffffffff810b58f2
[ 438.636088] ffff88041c782a00 00007fc2d81b5000 ffff88041c064000
00000000003808ff
[ 438.636281] ffff88041c064000 ffff88041c065e98 ffff88041d8caa70
ffff8804165d4480
[ 438.636479] Call Trace:
[ 438.636528] [<ffffffff810b58f2>] ? ksm_scan_thread+0x4e2/0xc20
[ 438.636580] [<ffffffff81052a20>] ? wake_up_bit+0x40/0x40
[ 438.636628] [<ffffffff810b5410>] ? try_to_merge_with_ksm_page+0x570/0x570
[ 438.636679] [<ffffffff810b5410>] ? try_to_merge_with_ksm_page+0x570/0x570
[ 438.636730] [<ffffffff810525b6>] ? kthread+0x96/0xa0
[ 438.636781] [<ffffffff813d1794>] ? kernel_thread_helper+0x4/0x10
[ 438.636832] [<ffffffff81052520>] ? kthread_worker_fn+0x120/0x120
[ 438.636882] [<ffffffff813d1790>] ? gs_change+0xb/0xb
[ 438.636926] Code: 28 48 89 ef e8 6c fe ff ff 48 85 c0 49 89 c4 74 d2
f0 0f ba 28 00 19 c0 85 c0 0f 85 ae 00 00 00 00 48 8b 43 30 48 8b 53 38
48 85 c0
[ 438.638504] 89 02 74 04 48 89 50 08 48 ba 00 01 10 00 00 00 00 ad de 48 b8
[ 438.639329] RIP [<ffffffff810b4596>] remove_rmap_item_from_tree+0x96/0x150
[ 438.639414] RSP <ffff88041c065e20>
[ 438.639460] ---[ end trace c29fb871f6b874e3 ]---
[ 438.639506] Kernel panic - not syncing: Fatal exception
[ 438.639553] Pid: 551, comm: ksmd Tainted: G D 2.6.39 #3
[ 438.639598] Call Trace:
[ 438.639644] [<ffffffff813cd6f5>] ? panic+0x92/0x18a
[ 438.639693] [<ffffffff81038b61>] ? kmsg_dump+0x41/0xf0
[ 438.639745] [<ffffffff810050ad>] ? oops_end+0x8d/0xa0
[ 438.639794] [<ffffffff813d05ef>] ? general_protection+0x1f/0x30
[ 438.639842] [<ffffffff810b4596>] ? remove_rmap_item_from_tree+0x96/0x150
[ 438.639891] [<ffffffff810b58f2>] ? ksm_scan_thread+0x4e2/0xc20
[ 438.639941] [<ffffffff81052a20>] ? wake_up_bit+0x40/0x40
[ 438.640045] [<ffffffff810b5410>] ? try_to_merge_with_ksm_page+0x570/0x570
[ 438.640096] [<ffffffff810b5410>] ? try_to_merge_with_ksm_page+0x570/0x570
[ 438.640147] [<ffffffff810525b6>] ? kthread+0x96/0xa0
[ 438.640196] [<ffffffff813d1794>] ? kernel_thread_helper+0x4/0x10
[ 438.640247] [<ffffffff81052520>] ? kthread_worker_fn+0x120/0x120
[ 438.640297] [<ffffffff813d1790>] ? gs_change+0xb/0xb
[ 438.640349] Rebooting in 60 seconds..
[ 469.706013] general protection fault: 0000 [#1] SMP
[ 469.706125] last sysfs file:
/sys/devices/pci0000:00/0000:00:18.3/temp1_input
[ 469.706162] CPU 1
[ 469.706195] Modules linked in: xt_iprange xt_DSCP xt_length
xt_CLASSIFY sch_sfq xt_CHECKSUM ipt_REJECT ipt_MASQUERADE ipt_REDIRECT
xt_recent xt_state iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack
nf_defrag_ipv4 iptable_filter xt_TCPMSS xt_tcpmss xt_tcpudp
iptable_mangle ip_tables x_tables pppoe pppox ppp_generic slhc cls_u32
sch_htb deflate zlib_deflate des_generi cbc ecb crypto_blkcipher
sha1_generic md5 hmac crypto_hash cryptomgr aead crypto_algapi af_key
fuse netconsole configfs vhost_net powernow_k8 mperf i2c_nforce2 kvm_amd
kvm pl2303 usbserial xhci_hcd i2c_piix4 k10temp ahci usb_storage
usb_libusual ohci_hcd ehci_hcd r8169 libahci usbcore mii sata_mv
megaraid_sas [last unloaded: scsi_wait_scan]
[ 469.708573]
[ 469.708608] Pid: 4942, comm: runbot Not tainted 2.6.39 #3 To Be Filled
By O.E.M. To Be Filled By O.E.M. /880G Extreme3
[ 469.708753] RIP: 0010:[<ffffffff810db878>] [<ffffffff810db878>]
dup_fd+0x168/0x300
[ 469.708829] RSP: 0018:ffff880417279dd0 EFLAGS: 00010202
[ 469.708870] RAX: 00000000000007f8 RBX: ffff88041c1658c0 RCX:
bfffffffffffffff
[ 469.708907] RDX: 0000880413155540 RSI: 00000000000000ff RDI:
0000000000000800
[ 469.708945] RBP: ffff88041bf22000 R08: ffff880412839dc0 R09:
0000000000000003
[ 469.708981] R10: 0000000000000001 R11: 4000000000000000 R12:
0000000000000100
[ 469.709018] R13: ffff88041d3cc000 R14: ffff880412839d40 R15:
ffff88041c72f440
[ 469.709054] FS: 00007f510c0b4700(0000) GS:ffff88041fc40000(0000)
knlGS:0000000000000000
[ 469.709094] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 469.709130] CR2: 00007f510ba56810 CR3: 0000000417c93000 CR4:
00000000000006e0
[ 469.709167] DR0: 00000000000000a0 DR1: 0000000000000000 DR2:
0000000000000003
[ 469.709204] DR3: 00000000000000b0 DR6: 00000000ffff0ff0 DR7:
0000000000000400
[ 469.709240] Process runbot (pid: 4942, threadinfo ffff880417278000,
task ffff880401351230)
[ 469.709279] Stack:
[ 469.709313] ffff880401351230 0000000000000020 ffff880412e9ca80
ffff88041c1658d0 ffff88041c1658d0
[ 469.709477] ffff88041723f330 0000000001200011 ffff88041723f330
0000000000000000
[ 469.709641] 00007f510c0b49d0 ffff880401351230 0000000000000000
ffffffff81037645
[ 469.709812] Call Trace:
[ 469.709849] [<ffffffff81037645>] ? copy_process+0xa75/0xfd0
[ 469.709885] [<ffffffff81037c0d>] ? do_fork+0x6d/0x2b0
[ 469.709921] [<ffffffff810dbc23>] ? alloc_fd+0x43/0x130
[ 469.709958] [<ffffffff810c08c5>] ? fd_install+0x35/0x70
[ 469.709998] [<ffffffff810457a9>] ? sigprocmask+0x69/0x100
[ 469.710042] [<ffffffff813d0ca3>] ? stub_clone+0x13/0x20
[ 469.710079] [<ffffffff813d0a3b>] ? system_call_fastpath+0x16/0x1b
[ 469.710114] Code: 4c 89 c2 e8 6b e5 0f 00 45 85 e4 74 78 41 8d 44 24
ff 31 f6 41 ba 01 00 00 00 00 48 8d 3c c5 08 00 00 00 00 31 c0 eb 1a 0f
1f 44 00 00 f0> 48 ff 42 30 48 89 54 05 00 48 83 c0
May 30 16:39:36 192.168.2.1 08 ff c6 48 39 f8 74 3b
[ 469.712247] RIP [<ffffffff810db878>] dup_fd+0x168/0x300
[ 469.712317] RSP <ffff880417279dd0>
[ 469.712391] ---[ end trace b80596608cd6bb6b ]---
[ 469.712432] Kernel panic - not syncing: Fatal exception
[ 469.712470] Pid: 4942, comm: runbot Tainted: G D 2.6.39 #3
[ 469.712508] Call Trace:
[ 469.712544] [<ffffffff813cd6f5>] ? panic+0x92/0x18a
[ 469.712581] [<ffffffff81038b61>] ? kmsg_dump+0x41/0xf0
[ 469.712618] [<ffffffff810050ad>] ? oops_end+0x8d/0xa0
[ 469.712654] [<ffffffff813d05ef>] ? general_protection+0x1f/0x30
[ 469.712691] [<ffffffff810db878>] ? dup_fd+0x168/0x300
[ 469.712727] [<ffffffff81037645>] ? copy_process+0xa75/0xfd0
[ 469.712767] [<ffffffff81037c0d>] ? do_fork+0x6d/0x2b0
[ 469.712803] [<ffffffff810dbc23>] ? alloc_fd+0x43/0x130
[ 469.712839] [<ffffffff810c08c5>] ? fd_install+0x35/0x70
[ 469.712875] [<ffffffff810457a9>] ? sigprocmask+0x69/0x100
[ 469.712912] [<ffffffff813d0ca3>] ? stub_clone+0x13/0x20
[ 469.712948] [<ffffffff813d0a3b>] ? system_call_fastpath+0x16/0x1b
[ 469.712989] Rebooting in 60 seconds..
[ 1100.473612] general protection fault: 0000 [#1] SMP
[ 1100.473718] last sysfs file:
/sys/devices/pci0000:00/0000:00:18.3/temp1_input
[ 1100.473755] CPU 0
[ 1100.473788] Modules linked in: xt_iprange xt_DSCP xt_length
xt_CLASSIFY sch_sfq xt_CHECKSUM ipt_REJECT ipt_MASQUERADE ipt_REDIRECT
xt_recent xt_state iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack
nf_defrag_ipv4 xt_TCPMSS xt_tcpmss xt_tcpudp iptable_mangle pppoe pppox
iptable_filter ip_tables x_tables ppp_generic slhc cls_u32 sch_htb
deflate zlib_deflate des_generic cbc ecb crypto_blkcipher sha1_generic
md5 hmac crypto_hash cryptomgr aead crypto_algapi af_key fuse netconsole
configfs vhost_net powernow_k8 mperf i2c_nforce2 kvm_amd kvm pl2303
xhci_hcd usbserial k10temp i2c_piix4 usb_storage usb_libusual ohci_hcd
ehci_hcd ahci usbcore sata_mv r8169 mii libahci megaraid_sas [last
unloaded: scsi_wait_scan]
[ 1100.476102]
[ 1100.476137] Pid: 551, comm: ksmd Not tainted 2.6.39 #3 To Be Filled
By O.E.M. To Be Filled By O.E.M. /880G Extreme3
[ 1100.476277] RIP: 0010:[<ffffffff810b5c71>] [<ffffffff810b5c71>]
ksm_scan_thread+0x861/0xc20
[ 1100.476351] RSP: 0018:ffff88041c065e40 EFLAGS: 00010282
[ 1100.476387] RAX: 0000000000000000 RBX: ffffea000a5c8468 RCX:
0000000000000f01
[ 1100.476424] RDX: 0000880403456828 RSI: ffff8802f5dcb002 RDI:
ffffea000ce7eac8
[ 1100.476460] RBP: ffff880418377fc0 R08: 6db6db6db6db6db7 R09:
0000160000000000
[ 1100.476498] R10: ffffea000ce7eac8 R11: 0000000080100073 R12:
00000000000000ab
[ 1100.476535] R13: ffff880418377ff8 R14: ffffea000ce7eac8 R15:
ffff880418377fe8
[ 1100.476572] FS: 00007f08c6498700(0000) GS:ffff88041fc00000(0000)
knlGS:0000000000000000
[ 1100.476611] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 1100.476646] CR2: 00000000e1dd3000 CR3: 00000004025f7000 CR4:
00000000000006f0
[ 1100.477007] DR0: 0000000000000045 DR1: 0000000000000000 DR2:
0000000000000000
[ 1100.477042] DR3: 0000000000000005 DR6: 00000000ffff0ff0 DR7:
0000000000000400
[ 1100.477078] Process ksmd (pid: 551, threadinfo ffff88041c064000, task
ffff88041d0454e0)
[ 1100.477115] Stack:
[ 1100.477148] ffff880403a8b100 00007fc1863cb000 ffff88041c064000
00000000002f5dcb
[ 1100.477317] ffff88041c064000 ffff88041c065e98 ffff88041d0454e0
ffff88041267a580
[ 1100.477480] 0000000000000000 ffff88041d0454e0 ffffffff81052a20
ffff88041c065e98
[ 1100.477644] Call Trace:
[ 1100.477681] [<ffffffff81052a20>] ? wake_up_bit+0x40/0x40
[ 1100.477717] [<ffffffff810b5410>] ?
try_to_merge_with_ksm_page+0x570/0x570
[ 1100.477754] [<ffffffff810b5410>] ?
try_to_merge_with_ksm_page+0x570/0x570
[ 1100.477791] [<ffffffff810525b6>] ? kthread+0x96/0xa0
[ 1100.477829] [<ffffffff813d1794>] ? kernel_thread_helper+0x4/0x10
[ 1100.477865] [<ffffffff81052520>] ? kthread_worker_fn+0x120/0x120
[ 1100.477902] [<ffffffff813d1790>] ? gs_change+0xb/0xb
[ 1100.477937] Code: 05 99 6e 6e 00 4c 89 ff e8 ed 12 fa ff e9 6c f9 ff
ff 48 89 6c 24 38 49 83 7d 00 00 0f 84 63 01 00 00 e8 b3 8b 31 00 49 8b
55 00
[ 1100.479306] 8b 42 e8 4c 8b 72 f0 48 8d 6a d8 4c 8d 78 60 48 89 04 24 4c
[ 1100.480012] RIP [<ffffffff810b5c71>] ksm_scan_thread+0x861/0xc20
[ 1100.480015] RSP <ffff88041c065e40>
[ 1100.480141] ---[ end trace 7382777d9e74ea23 ]---
[ 1100.480178] Kernel panic - not syncing: Fatal exception
[ 1100.480216] Pid: 551, comm: ksmd Tainted: G D 2.6.39 #3
[ 1100.480257] Call Trace:
[ 1100.480297] [<ffffffff813cd6f5>] ? panic+0x92/0x18a
[ 1100.480340] [<ffffffff81038b61>] ? kmsg_dump+0x41/0xf0
[ 1100.480384] [<ffffffff810050ad>] ? oops_end+0x8d/0xa0
[ 1100.480425] [<ffffffff813d05ef>] ? general_protection+0x1f/0x30
[ 1100.480465] [<ffffffff810b5c71>] ? ksm_scan_thread+0x861/0xc20
[ 1100.480507] [<ffffffff81052a20>] ? wake_up_bit+0x40/0x40
[ 1100.480545] [<ffffffff810b5410>] ?
try_to_merge_with_ksm_page+0x570/0x570
[ 1100.480584] [<ffffffff810b5410>] ?
try_to_merge_with_ksm_page+0x570/0x570
[ 1100.480624] [<ffffffff810525b6>] ? kthread+0x96/0xa0
[ 1100.480669] [<ffffffff813d1794>] ? kernel_thread_helper+0x4/0x10
[ 1100.480708] [<ffffffff81052520>] ? kthread_worker_fn+0x120/0x120
[ 1100.480748] [<ffffffff813d1790>] ? gs_change+0xb/0xb
[ 1100.480794] Rebooting in 60 seconds..
brad@srv:~$ zcat /proc/config.gz | grep -v "#"
CONFIG_64BIT=y
CONFIG_X86_64=y
CONFIG_X86=y
CONFIG_INSTRUCTION_DECODER=y
CONFIG_OUTPUT_FORMAT="elf64-x86-64"
CONFIG_ARCH_DEFCONFIG="arch/x86/configs/x86_64_defconfig"
CONFIG_GENERIC_CMOS_UPDATE=y
CONFIG_CLOCKSOURCE_WATCHDOG=y
CONFIG_GENERIC_CLOCKEVENTS=y
CONFIG_GENERIC_CLOCKEVENTS_BROADCAST=y
CONFIG_LOCKDEP_SUPPORT=y
CONFIG_STACKTRACE_SUPPORT=y
CONFIG_HAVE_LATENCYTOP_SUPPORT=y
CONFIG_MMU=y
CONFIG_ZONE_DMA=y
CONFIG_NEED_DMA_MAP_STATE=y
CONFIG_NEED_SG_DMA_LENGTH=y
CONFIG_GENERIC_ISA_DMA=y
CONFIG_GENERIC_IOMAP=y
CONFIG_GENERIC_BUG=y
CONFIG_GENERIC_BUG_RELATIVE_POINTERS=y
CONFIG_GENERIC_HWEIGHT=y
CONFIG_ARCH_MAY_HAVE_PC_FDC=y
CONFIG_RWSEM_XCHGADD_ALGORITHM=y
CONFIG_ARCH_HAS_CPU_IDLE_WAIT=y
CONFIG_GENERIC_CALIBRATE_DELAY=y
CONFIG_GENERIC_TIME_VSYSCALL=y
CONFIG_ARCH_HAS_CPU_RELAX=y
CONFIG_ARCH_HAS_DEFAULT_IDLE=y
CONFIG_ARCH_HAS_CACHE_LINE_SIZE=y
CONFIG_HAVE_SETUP_PER_CPU_AREA=y
CONFIG_NEED_PER_CPU_EMBED_FIRST_CHUNK=y
CONFIG_NEED_PER_CPU_PAGE_FIRST_CHUNK=y
CONFIG_HAVE_CPUMASK_OF_CPU_MAP=y
CONFIG_ARCH_HIBERNATION_POSSIBLE=y
CONFIG_ARCH_SUSPEND_POSSIBLE=y
CONFIG_ZONE_DMA32=y
CONFIG_ARCH_POPULATES_NODE_MAP=y
CONFIG_AUDIT_ARCH=y
CONFIG_ARCH_SUPPORTS_OPTIMIZED_INLINING=y
CONFIG_ARCH_SUPPORTS_DEBUG_PAGEALLOC=y
CONFIG_X86_64_SMP=y
CONFIG_X86_HT=y
CONFIG_ARCH_HWEIGHT_CFLAGS="-fcall-saved-rdi -fcall-saved-rsi
-fcall-saved-rdx -fcall-saved-rcx -fcall-saved-r8 -fcall-saved-r9
-fcall-saved-r10 -fcall-saved-r11"
CONFIG_DEFCONFIG_LIST="/lib/modules/$UNAME_RELEASE/.config"
CONFIG_CONSTRUCTORS=y
CONFIG_HAVE_IRQ_WORK=y
CONFIG_IRQ_WORK=y
CONFIG_EXPERIMENTAL=y
CONFIG_INIT_ENV_ARG_LIMIT=32
CONFIG_CROSS_COMPILE=""
CONFIG_LOCALVERSION=""
CONFIG_LOCALVERSION_AUTO=y
CONFIG_HAVE_KERNEL_GZIP=y
CONFIG_HAVE_KERNEL_BZIP2=y
CONFIG_HAVE_KERNEL_LZMA=y
CONFIG_HAVE_KERNEL_XZ=y
CONFIG_HAVE_KERNEL_LZO=y
CONFIG_KERNEL_GZIP=y
CONFIG_SWAP=y
CONFIG_SYSVIPC=y
CONFIG_SYSVIPC_SYSCTL=y
CONFIG_BSD_PROCESS_ACCT=y
CONFIG_TASKSTATS=y
CONFIG_TASK_DELAY_ACCT=y
CONFIG_TASK_XACCT=y
CONFIG_TASK_IO_ACCOUNTING=y
CONFIG_HAVE_GENERIC_HARDIRQS=y
CONFIG_GENERIC_HARDIRQS=y
CONFIG_HAVE_SPARSE_IRQ=y
CONFIG_GENERIC_IRQ_PROBE=y
CONFIG_GENERIC_IRQ_SHOW=y
CONFIG_GENERIC_PENDING_IRQ=y
CONFIG_IRQ_FORCED_THREADING=y
CONFIG_TREE_RCU=y
CONFIG_RCU_FANOUT=64
CONFIG_IKCONFIG=y
CONFIG_IKCONFIG_PROC=y
CONFIG_LOG_BUF_SHIFT=20
CONFIG_HAVE_UNSTABLE_SCHED_CLOCK=y
CONFIG_NAMESPACES=y
CONFIG_BLK_DEV_INITRD=y
CONFIG_INITRAMFS_SOURCE=""
CONFIG_RD_GZIP=y
CONFIG_RD_BZIP2=y
CONFIG_RD_LZMA=y
CONFIG_RD_XZ=y
CONFIG_RD_LZO=y
CONFIG_SYSCTL=y
CONFIG_ANON_INODES=y
CONFIG_UID16=y
CONFIG_SYSCTL_SYSCALL=y
CONFIG_KALLSYMS=y
CONFIG_KALLSYMS_ALL=y
CONFIG_KALLSYMS_EXTRA_PASS=y
CONFIG_HOTPLUG=y
CONFIG_PRINTK=y
CONFIG_BUG=y
CONFIG_ELF_CORE=y
CONFIG_PCSPKR_PLATFORM=y
CONFIG_BASE_FULL=y
CONFIG_FUTEX=y
CONFIG_EPOLL=y
CONFIG_SIGNALFD=y
CONFIG_TIMERFD=y
CONFIG_EVENTFD=y
CONFIG_SHMEM=y
CONFIG_AIO=y
CONFIG_HAVE_PERF_EVENTS=y
CONFIG_PERF_EVENTS=y
CONFIG_VM_EVENT_COUNTERS=y
CONFIG_PCI_QUIRKS=y
CONFIG_SLUB_DEBUG=y
CONFIG_SLUB=y
CONFIG_HAVE_OPROFILE=y
CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS=y
CONFIG_USER_RETURN_NOTIFIER=y
CONFIG_HAVE_IOREMAP_PROT=y
CONFIG_HAVE_KPROBES=y
CONFIG_HAVE_KRETPROBES=y
CONFIG_HAVE_OPTPROBES=y
CONFIG_HAVE_ARCH_TRACEHOOK=y
CONFIG_HAVE_DMA_ATTRS=y
CONFIG_USE_GENERIC_SMP_HELPERS=y
CONFIG_HAVE_REGS_AND_STACK_ACCESS_API=y
CONFIG_HAVE_DMA_API_DEBUG=y
CONFIG_HAVE_HW_BREAKPOINT=y
CONFIG_HAVE_MIXED_BREAKPOINTS_REGS=y
CONFIG_HAVE_USER_RETURN_NOTIFIER=y
CONFIG_HAVE_PERF_EVENTS_NMI=y
CONFIG_HAVE_ARCH_JUMP_LABEL=y
CONFIG_SLABINFO=y
CONFIG_RT_MUTEXES=y
CONFIG_BASE_SMALL=0
CONFIG_MODULES=y
CONFIG_MODULE_UNLOAD=y
CONFIG_STOP_MACHINE=y
CONFIG_BLOCK=y
CONFIG_BLK_DEV_BSG=y
CONFIG_BLOCK_COMPAT=y
CONFIG_IOSCHED_NOOP=y
CONFIG_IOSCHED_DEADLINE=y
CONFIG_IOSCHED_CFQ=y
CONFIG_DEFAULT_CFQ=y
CONFIG_DEFAULT_IOSCHED="cfq"
CONFIG_PREEMPT_NOTIFIERS=y
CONFIG_INLINE_SPIN_UNLOCK=y
CONFIG_INLINE_SPIN_UNLOCK_IRQ=y
CONFIG_INLINE_READ_UNLOCK=y
CONFIG_INLINE_READ_UNLOCK_IRQ=y
CONFIG_INLINE_WRITE_UNLOCK=y
CONFIG_INLINE_WRITE_UNLOCK_IRQ=y
CONFIG_MUTEX_SPIN_ON_OWNER=y
CONFIG_TICK_ONESHOT=y
CONFIG_NO_HZ=y
CONFIG_HIGH_RES_TIMERS=y
CONFIG_GENERIC_CLOCKEVENTS_BUILD=y
CONFIG_SMP=y
CONFIG_X86_SUPPORTS_MEMORY_FAILURE=y
CONFIG_SCHED_OMIT_FRAME_POINTER=y
CONFIG_NO_BOOTMEM=y
CONFIG_MK8=y
CONFIG_X86_INTERNODE_CACHE_SHIFT=6
CONFIG_X86_CMPXCHG=y
CONFIG_CMPXCHG_LOCAL=y
CONFIG_X86_L1_CACHE_SHIFT=6
CONFIG_X86_XADD=y
CONFIG_X86_WP_WORKS_OK=y
CONFIG_X86_INTEL_USERCOPY=y
CONFIG_X86_USE_PPRO_CHECKSUM=y
CONFIG_X86_TSC=y
CONFIG_X86_CMPXCHG64=y
CONFIG_X86_CMOV=y
CONFIG_X86_MINIMUM_CPU_FAMILY=64
CONFIG_X86_DEBUGCTLMSR=y
CONFIG_CPU_SUP_INTEL=y
CONFIG_CPU_SUP_AMD=y
CONFIG_CPU_SUP_CENTAUR=y
CONFIG_HPET_TIMER=y
CONFIG_HPET_EMULATE_RTC=y
CONFIG_DMI=y
CONFIG_GART_IOMMU=y
CONFIG_AMD_IOMMU=y
CONFIG_SWIOTLB=y
CONFIG_IOMMU_HELPER=y
CONFIG_IOMMU_API=y
CONFIG_NR_CPUS=8
CONFIG_SCHED_MC=y
CONFIG_PREEMPT_NONE=y
CONFIG_X86_LOCAL_APIC=y
CONFIG_X86_IO_APIC=y
CONFIG_X86_MCE=y
CONFIG_X86_MCE_AMD=y
CONFIG_X86_MCE_THRESHOLD=y
CONFIG_ARCH_PHYS_ADDR_T_64BIT=y
CONFIG_ARCH_DMA_ADDR_T_64BIT=y
CONFIG_DIRECT_GBPAGES=y
CONFIG_ARCH_SPARSEMEM_DEFAULT=y
CONFIG_ARCH_SPARSEMEM_ENABLE=y
CONFIG_ARCH_SELECT_MEMORY_MODEL=y
CONFIG_ARCH_MEMORY_PROBE=y
CONFIG_ILLEGAL_POINTER_VALUE=0xdead000000000000
CONFIG_SELECT_MEMORY_MODEL=y
CONFIG_SPARSEMEM_MANUAL=y
CONFIG_SPARSEMEM=y
CONFIG_HAVE_MEMORY_PRESENT=y
CONFIG_SPARSEMEM_EXTREME=y
CONFIG_SPARSEMEM_VMEMMAP_ENABLE=y
CONFIG_SPARSEMEM_ALLOC_MEM_MAP_TOGETHER=y
CONFIG_SPARSEMEM_VMEMMAP=y
CONFIG_HAVE_MEMBLOCK=y
CONFIG_MEMORY_HOTPLUG=y
CONFIG_MEMORY_HOTPLUG_SPARSE=y
CONFIG_PAGEFLAGS_EXTENDED=y
CONFIG_SPLIT_PTLOCK_CPUS=4
CONFIG_COMPACTION=y
CONFIG_MIGRATION=y
CONFIG_PHYS_ADDR_T_64BIT=y
CONFIG_ZONE_DMA_FLAG=1
CONFIG_BOUNCE=y
CONFIG_VIRT_TO_BUS=y
CONFIG_MMU_NOTIFIER=y
CONFIG_KSM=y
CONFIG_DEFAULT_MMAP_MIN_ADDR=4096
CONFIG_ARCH_SUPPORTS_MEMORY_FAILURE=y
CONFIG_TRANSPARENT_HUGEPAGE=y
CONFIG_TRANSPARENT_HUGEPAGE_ALWAYS=y
CONFIG_X86_RESERVE_LOW=64
CONFIG_MTRR=y
CONFIG_MTRR_SANITIZER=y
CONFIG_MTRR_SANITIZER_ENABLE_DEFAULT=0
CONFIG_MTRR_SANITIZER_SPARE_REG_NR_DEFAULT=1
CONFIG_X86_PAT=y
CONFIG_ARCH_USES_PG_UNCACHED=y
CONFIG_SECCOMP=y
CONFIG_HZ_100=y
CONFIG_HZ=100
CONFIG_SCHED_HRTICK=y
CONFIG_PHYSICAL_START=0x1000000
CONFIG_PHYSICAL_ALIGN=0x1000000
CONFIG_ARCH_ENABLE_MEMORY_HOTPLUG=y
CONFIG_ARCH_ENABLE_MEMORY_HOTREMOVE=y
CONFIG_ACPI=y
CONFIG_ACPI_BUTTON=y
CONFIG_ACPI_DOCK=y
CONFIG_ACPI_PROCESSOR=y
CONFIG_ACPI_THERMAL=y
CONFIG_ACPI_BLACKLIST_YEAR=0
CONFIG_X86_PM_TIMER=y
CONFIG_CPU_FREQ=y
CONFIG_CPU_FREQ_TABLE=y
CONFIG_CPU_FREQ_STAT=y
CONFIG_CPU_FREQ_STAT_DETAILS=y
CONFIG_CPU_FREQ_DEFAULT_GOV_ONDEMAND=y
CONFIG_CPU_FREQ_GOV_PERFORMANCE=y
CONFIG_CPU_FREQ_GOV_POWERSAVE=y
CONFIG_CPU_FREQ_GOV_USERSPACE=y
CONFIG_CPU_FREQ_GOV_ONDEMAND=y
CONFIG_CPU_FREQ_GOV_CONSERVATIVE=y
CONFIG_X86_ACPI_CPUFREQ=m
CONFIG_X86_POWERNOW_K8=m
CONFIG_CPU_IDLE=y
CONFIG_CPU_IDLE_GOV_LADDER=y
CONFIG_CPU_IDLE_GOV_MENU=y
CONFIG_PCI=y
CONFIG_PCI_DIRECT=y
CONFIG_PCI_MMCONFIG=y
CONFIG_PCI_DOMAINS=y
CONFIG_PCIEPORTBUS=y
CONFIG_PCIEAER=y
CONFIG_PCIEASPM=y
CONFIG_ARCH_SUPPORTS_MSI=y
CONFIG_PCI_MSI=y
CONFIG_HT_IRQ=y
CONFIG_PCI_IOAPIC=y
CONFIG_PCI_LABEL=y
CONFIG_ISA_DMA_API=y
CONFIG_AMD_NB=y
CONFIG_BINFMT_ELF=y
CONFIG_COMPAT_BINFMT_ELF=y
CONFIG_BINFMT_MISC=m
CONFIG_IA32_EMULATION=y
CONFIG_IA32_AOUT=m
CONFIG_COMPAT=y
CONFIG_COMPAT_FOR_U64_ALIGNMENT=y
CONFIG_SYSVIPC_COMPAT=y
CONFIG_HAVE_TEXT_POKE_SMP=y
CONFIG_NET=y
CONFIG_PACKET=y
CONFIG_UNIX=y
CONFIG_XFRM=y
CONFIG_XFRM_MIGRATE=y
CONFIG_XFRM_IPCOMP=m
CONFIG_NET_KEY=m
CONFIG_NET_KEY_MIGRATE=y
CONFIG_INET=y
CONFIG_IP_MULTICAST=y
CONFIG_IP_ROUTE_CLASSID=y
CONFIG_NET_IPGRE_DEMUX=m
CONFIG_INET_AH=m
CONFIG_INET_ESP=m
CONFIG_INET_IPCOMP=m
CONFIG_INET_XFRM_TUNNEL=m
CONFIG_INET_TUNNEL=m
CONFIG_INET_XFRM_MODE_TRANSPORT=m
CONFIG_INET_XFRM_MODE_TUNNEL=m
CONFIG_INET_XFRM_MODE_BEET=m
CONFIG_INET_DIAG=y
CONFIG_INET_TCP_DIAG=y
CONFIG_TCP_CONG_CUBIC=y
CONFIG_DEFAULT_TCP_CONG="cubic"
CONFIG_NETFILTER=y
CONFIG_NETFILTER_ADVANCED=y
CONFIG_BRIDGE_NETFILTER=y
CONFIG_NETFILTER_NETLINK=m
CONFIG_NETFILTER_NETLINK_QUEUE=m
CONFIG_NETFILTER_NETLINK_LOG=m
CONFIG_NF_CONNTRACK=m
CONFIG_NF_CONNTRACK_MARK=y
CONFIG_NF_CONNTRACK_ZONES=y
CONFIG_NF_CT_PROTO_DCCP=m
CONFIG_NF_CT_PROTO_GRE=m
CONFIG_NF_CT_PROTO_SCTP=m
CONFIG_NF_CT_PROTO_UDPLITE=m
CONFIG_NF_CONNTRACK_AMANDA=m
CONFIG_NF_CONNTRACK_FTP=m
CONFIG_NF_CONNTRACK_H323=m
CONFIG_NF_CONNTRACK_IRC=m
CONFIG_NF_CONNTRACK_BROADCAST=m
CONFIG_NF_CONNTRACK_NETBIOS_NS=m
CONFIG_NF_CONNTRACK_SNMP=m
CONFIG_NF_CONNTRACK_PPTP=m
CONFIG_NF_CONNTRACK_SANE=m
CONFIG_NF_CONNTRACK_SIP=m
CONFIG_NF_CONNTRACK_TFTP=m
CONFIG_NF_CT_NETLINK=m
CONFIG_NETFILTER_TPROXY=m
CONFIG_NETFILTER_XTABLES=m
CONFIG_NETFILTER_XT_MARK=m
CONFIG_NETFILTER_XT_CONNMARK=m
CONFIG_NETFILTER_XT_TARGET_CHECKSUM=m
CONFIG_NETFILTER_XT_TARGET_CLASSIFY=m
CONFIG_NETFILTER_XT_TARGET_CONNMARK=m
CONFIG_NETFILTER_XT_TARGET_CT=m
CONFIG_NETFILTER_XT_TARGET_DSCP=m
CONFIG_NETFILTER_XT_TARGET_HL=m
CONFIG_NETFILTER_XT_TARGET_IDLETIMER=m
CONFIG_NETFILTER_XT_TARGET_MARK=m
CONFIG_NETFILTER_XT_TARGET_NFLOG=m
CONFIG_NETFILTER_XT_TARGET_NFQUEUE=m
CONFIG_NETFILTER_XT_TARGET_NOTRACK=m
CONFIG_NETFILTER_XT_TARGET_RATEEST=m
CONFIG_NETFILTER_XT_TARGET_TEE=m
CONFIG_NETFILTER_XT_TARGET_TPROXY=m
CONFIG_NETFILTER_XT_TARGET_TRACE=m
CONFIG_NETFILTER_XT_TARGET_TCPMSS=m
CONFIG_NETFILTER_XT_TARGET_TCPOPTSTRIP=m
CONFIG_NETFILTER_XT_MATCH_ADDRTYPE=m
CONFIG_NETFILTER_XT_MATCH_CLUSTER=m
CONFIG_NETFILTER_XT_MATCH_COMMENT=m
CONFIG_NETFILTER_XT_MATCH_CONNBYTES=m
CONFIG_NETFILTER_XT_MATCH_CONNLIMIT=m
CONFIG_NETFILTER_XT_MATCH_CONNMARK=m
CONFIG_NETFILTER_XT_MATCH_CONNTRACK=m
CONFIG_NETFILTER_XT_MATCH_CPU=m
CONFIG_NETFILTER_XT_MATCH_DCCP=m
CONFIG_NETFILTER_XT_MATCH_DEVGROUP=m
CONFIG_NETFILTER_XT_MATCH_DSCP=m
CONFIG_NETFILTER_XT_MATCH_ESP=m
CONFIG_NETFILTER_XT_MATCH_HASHLIMIT=m
CONFIG_NETFILTER_XT_MATCH_HELPER=m
CONFIG_NETFILTER_XT_MATCH_HL=m
CONFIG_NETFILTER_XT_MATCH_IPRANGE=m
CONFIG_NETFILTER_XT_MATCH_LENGTH=m
CONFIG_NETFILTER_XT_MATCH_LIMIT=m
CONFIG_NETFILTER_XT_MATCH_MAC=m
CONFIG_NETFILTER_XT_MATCH_MARK=m
CONFIG_NETFILTER_XT_MATCH_MULTIPORT=m
CONFIG_NETFILTER_XT_MATCH_OSF=m
CONFIG_NETFILTER_XT_MATCH_OWNER=m
CONFIG_NETFILTER_XT_MATCH_POLICY=m
CONFIG_NETFILTER_XT_MATCH_PHYSDEV=m
CONFIG_NETFILTER_XT_MATCH_PKTTYPE=m
CONFIG_NETFILTER_XT_MATCH_QUOTA=m
CONFIG_NETFILTER_XT_MATCH_RATEEST=m
CONFIG_NETFILTER_XT_MATCH_REALM=m
CONFIG_NETFILTER_XT_MATCH_RECENT=m
CONFIG_NETFILTER_XT_MATCH_SCTP=m
CONFIG_NETFILTER_XT_MATCH_SOCKET=m
CONFIG_NETFILTER_XT_MATCH_STATE=m
CONFIG_NETFILTER_XT_MATCH_STATISTIC=m
CONFIG_NETFILTER_XT_MATCH_STRING=m
CONFIG_NETFILTER_XT_MATCH_TCPMSS=m
CONFIG_NETFILTER_XT_MATCH_TIME=m
CONFIG_NETFILTER_XT_MATCH_U32=m
CONFIG_NF_DEFRAG_IPV4=m
CONFIG_NF_CONNTRACK_IPV4=m
CONFIG_NF_CONNTRACK_PROC_COMPAT=y
CONFIG_IP_NF_QUEUE=m
CONFIG_IP_NF_IPTABLES=m
CONFIG_IP_NF_MATCH_AH=m
CONFIG_IP_NF_MATCH_ECN=m
CONFIG_IP_NF_MATCH_TTL=m
CONFIG_IP_NF_FILTER=m
CONFIG_IP_NF_TARGET_REJECT=m
CONFIG_IP_NF_TARGET_LOG=m
CONFIG_IP_NF_TARGET_ULOG=m
CONFIG_NF_NAT=m
CONFIG_NF_NAT_NEEDED=y
CONFIG_IP_NF_TARGET_MASQUERADE=m
CONFIG_IP_NF_TARGET_NETMAP=m
CONFIG_IP_NF_TARGET_REDIRECT=m
CONFIG_NF_NAT_SNMP_BASIC=m
CONFIG_NF_NAT_PROTO_DCCP=m
CONFIG_NF_NAT_PROTO_GRE=m
CONFIG_NF_NAT_PROTO_UDPLITE=m
CONFIG_NF_NAT_PROTO_SCTP=m
CONFIG_NF_NAT_FTP=m
CONFIG_NF_NAT_IRC=m
CONFIG_NF_NAT_TFTP=m
CONFIG_NF_NAT_AMANDA=m
CONFIG_NF_NAT_PPTP=m
CONFIG_NF_NAT_H323=m
CONFIG_NF_NAT_SIP=m
CONFIG_IP_NF_MANGLE=m
CONFIG_IP_NF_TARGET_CLUSTERIP=m
CONFIG_IP_NF_TARGET_ECN=m
CONFIG_IP_NF_TARGET_TTL=m
CONFIG_IP_NF_RAW=m
CONFIG_IP_NF_ARPTABLES=m
CONFIG_IP_NF_ARPFILTER=m
CONFIG_IP_NF_ARP_MANGLE=m
CONFIG_IP_SCTP=m
CONFIG_SCTP_HMAC_MD5=y
CONFIG_STP=y
CONFIG_BRIDGE=y
CONFIG_BRIDGE_IGMP_SNOOPING=y
CONFIG_VLAN_8021Q=y
CONFIG_LLC=y
CONFIG_NET_SCHED=y
CONFIG_NET_SCH_CBQ=m
CONFIG_NET_SCH_HTB=m
CONFIG_NET_SCH_HFSC=m
CONFIG_NET_SCH_PRIO=m
CONFIG_NET_SCH_RED=m
CONFIG_NET_SCH_SFB=m
CONFIG_NET_SCH_SFQ=m
CONFIG_NET_SCH_TEQL=m
CONFIG_NET_SCH_TBF=m
CONFIG_NET_SCH_GRED=m
CONFIG_NET_SCH_DSMARK=m
CONFIG_NET_SCH_NETEM=m
CONFIG_NET_SCH_MQPRIO=m
CONFIG_NET_SCH_CHOKE=m
CONFIG_NET_SCH_INGRESS=m
CONFIG_NET_CLS=y
CONFIG_NET_CLS_BASIC=m
CONFIG_NET_CLS_TCINDEX=m
CONFIG_NET_CLS_ROUTE4=m
CONFIG_NET_CLS_FW=m
CONFIG_NET_CLS_U32=m
CONFIG_NET_CLS_FLOW=m
CONFIG_NET_EMATCH=y
CONFIG_NET_EMATCH_STACK=32
CONFIG_NET_EMATCH_CMP=m
CONFIG_NET_EMATCH_NBYTE=m
CONFIG_NET_EMATCH_U32=m
CONFIG_NET_EMATCH_META=m
CONFIG_NET_EMATCH_TEXT=m
CONFIG_NET_CLS_ACT=y
CONFIG_NET_ACT_POLICE=m
CONFIG_NET_ACT_GACT=m
CONFIG_GACT_PROB=y
CONFIG_NET_ACT_MIRRED=m
CONFIG_NET_ACT_IPT=m
CONFIG_NET_ACT_NAT=m
CONFIG_NET_ACT_PEDIT=m
CONFIG_NET_ACT_SIMP=m
CONFIG_NET_ACT_SKBEDIT=m
CONFIG_NET_ACT_CSUM=m
CONFIG_NET_SCH_FIFO=y
CONFIG_RPS=y
CONFIG_RFS_ACCEL=y
CONFIG_XPS=y
CONFIG_UEVENT_HELPER_PATH="/sbin/hotplug"
CONFIG_DEVTMPFS=y
CONFIG_STANDALONE=y
CONFIG_PREVENT_FIRMWARE_BUILD=y
CONFIG_FW_LOADER=y
CONFIG_FIRMWARE_IN_KERNEL=y
CONFIG_EXTRA_FIRMWARE=""
CONFIG_ARCH_NO_SYSDEV_OPS=y
CONFIG_PNP=y
CONFIG_PNP_DEBUG_MESSAGES=y
CONFIG_PNPACPI=y
CONFIG_BLK_DEV=y
CONFIG_BLK_DEV_LOOP=y
CONFIG_HAVE_IDE=y
CONFIG_SCSI_MOD=y
CONFIG_RAID_ATTRS=m
CONFIG_SCSI=y
CONFIG_SCSI_DMA=y
CONFIG_SCSI_PROC_FS=y
CONFIG_BLK_DEV_SD=y
CONFIG_BLK_DEV_SR=y
CONFIG_CHR_DEV_SG=y
CONFIG_SCSI_WAIT_SCAN=m
CONFIG_SCSI_SAS_ATTRS=m
CONFIG_SCSI_LOWLEVEL=y
CONFIG_MEGARAID_SAS=m
CONFIG_ATA=y
CONFIG_ATA_VERBOSE_ERROR=y
CONFIG_ATA_ACPI=y
CONFIG_SATA_PMP=y
CONFIG_SATA_AHCI=m
CONFIG_SATA_SIL24=m
CONFIG_ATA_SFF=y
CONFIG_ATA_BMDMA=y
CONFIG_SATA_MV=m
CONFIG_SATA_NV=m
CONFIG_SATA_PROMISE=m
CONFIG_SATA_SIL=m
CONFIG_MD=y
CONFIG_BLK_DEV_MD=y
CONFIG_MD_LINEAR=y
CONFIG_MD_RAID0=y
CONFIG_MD_RAID1=y
CONFIG_MD_RAID10=y
CONFIG_MD_RAID456=y
CONFIG_BLK_DEV_DM=y
CONFIG_NETDEVICES=y
CONFIG_DUMMY=m
CONFIG_BONDING=m
CONFIG_TUN=y
CONFIG_MII=m
CONFIG_NET_ETHERNET=y
CONFIG_NET_PCI=y
CONFIG_FORCEDETH=m
CONFIG_NETDEV_1000=y
CONFIG_R8169=m
CONFIG_PPP=m
CONFIG_PPP_MULTILINK=y
CONFIG_PPP_FILTER=y
CONFIG_PPP_ASYNC=m
CONFIG_PPP_SYNC_TTY=m
CONFIG_PPP_DEFLATE=m
CONFIG_PPP_BSDCOMP=m
CONFIG_PPP_MPPE=m
CONFIG_PPPOE=m
CONFIG_SLHC=m
CONFIG_NETCONSOLE=m
CONFIG_NETCONSOLE_DYNAMIC=y
CONFIG_NETPOLL=y
CONFIG_NET_POLL_CONTROLLER=y
CONFIG_INPUT=y
CONFIG_INPUT_MOUSEDEV=y
CONFIG_INPUT_MOUSEDEV_SCREEN_X=1024
CONFIG_INPUT_MOUSEDEV_SCREEN_Y=768
CONFIG_INPUT_KEYBOARD=y
CONFIG_KEYBOARD_ATKBD=y
CONFIG_SERIO=y
CONFIG_SERIO_I8042=y
CONFIG_SERIO_LIBPS2=y
CONFIG_VT=y
CONFIG_CONSOLE_TRANSLATIONS=y
CONFIG_VT_CONSOLE=y
CONFIG_HW_CONSOLE=y
CONFIG_UNIX98_PTYS=y
CONFIG_LEGACY_PTYS=y
CONFIG_LEGACY_PTY_COUNT=256
CONFIG_SERIAL_8250=y
CONFIG_SERIAL_8250_CONSOLE=y
CONFIG_FIX_EARLYCON_MEM=y
CONFIG_SERIAL_8250_PCI=y
CONFIG_SERIAL_8250_PNP=y
CONFIG_SERIAL_8250_NR_UARTS=4
CONFIG_SERIAL_8250_RUNTIME_UARTS=4
CONFIG_SERIAL_CORE=y
CONFIG_SERIAL_CORE_CONSOLE=y
CONFIG_RTC=y
CONFIG_HPET=y
CONFIG_HPET_MMAP=y
CONFIG_DEVPORT=y
CONFIG_I2C=y
CONFIG_I2C_BOARDINFO=y
CONFIG_I2C_COMPAT=y
CONFIG_I2C_CHARDEV=y
CONFIG_I2C_HELPER_AUTO=y
CONFIG_I2C_ALGOBIT=y
CONFIG_I2C_AMD756=m
CONFIG_I2C_PIIX4=m
CONFIG_I2C_NFORCE2=m
CONFIG_I2C_VIA=m
CONFIG_I2C_VIAPRO=m
CONFIG_I2C_SCMI=m
CONFIG_ARCH_WANT_OPTIONAL_GPIOLIB=y
CONFIG_HWMON=y
CONFIG_HWMON_VID=m
CONFIG_SENSORS_K8TEMP=m
CONFIG_SENSORS_K10TEMP=m
CONFIG_SENSORS_W83781D=m
CONFIG_SENSORS_W83791D=m
CONFIG_SENSORS_W83792D=m
CONFIG_SENSORS_W83793=m
CONFIG_SENSORS_W83795=m
CONFIG_SENSORS_W83795_FANCTRL=y
CONFIG_SENSORS_W83L785TS=m
CONFIG_SENSORS_W83L786NG=m
CONFIG_SENSORS_W83627HF=m
CONFIG_SENSORS_W83627EHF=m
CONFIG_SENSORS_ATK0110=m
CONFIG_THERMAL=y
CONFIG_THERMAL_HWMON=y
CONFIG_SSB_POSSIBLE=y
CONFIG_VGA_ARB=y
CONFIG_VGA_ARB_MAX_GPUS=16
CONFIG_VGASTATE=y
CONFIG_FB=y
CONFIG_FB_DDC=y
CONFIG_FB_CFB_FILLRECT=y
CONFIG_FB_CFB_COPYAREA=y
CONFIG_FB_CFB_IMAGEBLIT=y
CONFIG_FB_MODE_HELPERS=y
CONFIG_FB_NVIDIA=y
CONFIG_FB_NVIDIA_I2C=y
CONFIG_FB_RADEON=y
CONFIG_FB_RADEON_I2C=y
CONFIG_VGA_CONSOLE=y
CONFIG_DUMMY_CONSOLE=y
CONFIG_FRAMEBUFFER_CONSOLE=y
CONFIG_FONT_8x8=y
CONFIG_FONT_8x16=y
CONFIG_LOGO=y
CONFIG_LOGO_LINUX_MONO=y
CONFIG_LOGO_LINUX_VGA16=y
CONFIG_LOGO_LINUX_CLUT224=y
CONFIG_HID_SUPPORT=y
CONFIG_HID=y
CONFIG_HIDRAW=y
CONFIG_USB_HID=m
CONFIG_USB_HIDDEV=y
CONFIG_HID_A4TECH=m
CONFIG_HID_APPLE=m
CONFIG_HID_BELKIN=m
CONFIG_HID_CHERRY=m
CONFIG_HID_CHICONY=m
CONFIG_HID_CYPRESS=m
CONFIG_HID_EZKEY=m
CONFIG_HID_KYE=m
CONFIG_HID_KENSINGTON=m
CONFIG_HID_LOGITECH=m
CONFIG_HID_MICROSOFT=m
CONFIG_HID_MONTEREY=m
CONFIG_USB_SUPPORT=y
CONFIG_USB_ARCH_HAS_HCD=y
CONFIG_USB_ARCH_HAS_OHCI=y
CONFIG_USB_ARCH_HAS_EHCI=y
CONFIG_USB=m
CONFIG_USB_XHCI_HCD=m
CONFIG_USB_EHCI_HCD=m
CONFIG_USB_OHCI_HCD=m
CONFIG_USB_OHCI_LITTLE_ENDIAN=y
CONFIG_USB_UHCI_HCD=m
CONFIG_USB_ACM=m
CONFIG_USB_STORAGE=m
CONFIG_USB_LIBUSUAL=y
CONFIG_USB_SERIAL=m
CONFIG_USB_SERIAL_GENERIC=y
CONFIG_USB_SERIAL_FTDI_SIO=m
CONFIG_USB_SERIAL_IPAQ=m
CONFIG_USB_SERIAL_PL2303=m
CONFIG_FIRMWARE_MEMMAP=y
CONFIG_DMIID=y
CONFIG_EXT2_FS=y
CONFIG_EXT2_FS_XATTR=y
CONFIG_EXT3_FS=y
CONFIG_EXT3_FS_XATTR=y
CONFIG_EXT4_FS=y
CONFIG_EXT4_FS_XATTR=y
CONFIG_JBD=y
CONFIG_JBD2=y
CONFIG_FS_MBCACHE=y
CONFIG_EXPORTFS=y
CONFIG_FILE_LOCKING=y
CONFIG_FSNOTIFY=y
CONFIG_DNOTIFY=y
CONFIG_INOTIFY_USER=y
CONFIG_FANOTIFY=y
CONFIG_FUSE_FS=m
CONFIG_CUSE=m
CONFIG_FSCACHE=m
CONFIG_FSCACHE_DEBUG=y
CONFIG_CACHEFILES=m
CONFIG_ISO9660_FS=m
CONFIG_JOLIET=y
CONFIG_ZISOFS=y
CONFIG_UDF_FS=m
CONFIG_UDF_NLS=y
CONFIG_FAT_FS=m
CONFIG_MSDOS_FS=m
CONFIG_VFAT_FS=m
CONFIG_FAT_DEFAULT_CODEPAGE=437
CONFIG_FAT_DEFAULT_IOCHARSET="iso8859-1"
CONFIG_NTFS_FS=m
CONFIG_PROC_FS=y
CONFIG_PROC_SYSCTL=y
CONFIG_PROC_PAGE_MONITOR=y
CONFIG_SYSFS=y
CONFIG_TMPFS=y
CONFIG_HUGETLBFS=y
CONFIG_HUGETLB_PAGE=y
CONFIG_CONFIGFS_FS=m
CONFIG_NETWORK_FILESYSTEMS=y
CONFIG_NFS_FS=y
CONFIG_NFS_V3=y
CONFIG_NFSD=y
CONFIG_NFSD_DEPRECATED=y
CONFIG_NFSD_V3=y
CONFIG_LOCKD=y
CONFIG_LOCKD_V4=y
CONFIG_NFS_COMMON=y
CONFIG_SUNRPC=y
CONFIG_PARTITION_ADVANCED=y
CONFIG_MSDOS_PARTITION=y
CONFIG_EFI_PARTITION=y
CONFIG_NLS=y
CONFIG_NLS_DEFAULT="iso8859-1"
CONFIG_NLS_CODEPAGE_437=y
CONFIG_NLS_ISO8859_1=y
CONFIG_DLM=m
CONFIG_TRACE_IRQFLAGS_SUPPORT=y
CONFIG_PRINTK_TIME=y
CONFIG_DEFAULT_MESSAGE_LOGLEVEL=4
CONFIG_ENABLE_WARN_DEPRECATED=y
CONFIG_ENABLE_MUST_CHECK=y
CONFIG_FRAME_WARN=2048
CONFIG_MAGIC_SYSRQ=y
CONFIG_DEBUG_FS=y
CONFIG_DEBUG_KERNEL=y
CONFIG_DETECT_HUNG_TASK=y
CONFIG_BOOTPARAM_HUNG_TASK_PANIC_VALUE=0
CONFIG_TIMER_STATS=y
CONFIG_DEBUG_BUGVERBOSE=y
CONFIG_DEBUG_MEMORY_INIT=y
CONFIG_ARCH_WANT_FRAME_POINTERS=y
CONFIG_SYSCTL_SYSCALL_CHECK=y
CONFIG_USER_STACKTRACE_SUPPORT=y
CONFIG_HAVE_FUNCTION_TRACER=y
CONFIG_HAVE_FUNCTION_GRAPH_TRACER=y
CONFIG_HAVE_FUNCTION_GRAPH_FP_TEST=y
CONFIG_HAVE_FUNCTION_TRACE_MCOUNT_TEST=y
CONFIG_HAVE_DYNAMIC_FTRACE=y
CONFIG_HAVE_FTRACE_MCOUNT_RECORD=y
CONFIG_HAVE_SYSCALL_TRACEPOINTS=y
CONFIG_HAVE_C_RECORDMCOUNT=y
CONFIG_TRACING_SUPPORT=y
CONFIG_HAVE_ARCH_KGDB=y
CONFIG_HAVE_ARCH_KMEMCHECK=y
CONFIG_X86_VERBOSE_BOOTUP=y
CONFIG_EARLY_PRINTK=y
CONFIG_HAVE_MMIOTRACE_SUPPORT=y
CONFIG_IO_DELAY_TYPE_0X80=0
CONFIG_IO_DELAY_TYPE_0XED=1
CONFIG_IO_DELAY_TYPE_UDELAY=2
CONFIG_IO_DELAY_TYPE_NONE=3
CONFIG_IO_DELAY_0X80=y
CONFIG_DEFAULT_IO_DELAY_TYPE=0
CONFIG_DEFAULT_SECURITY_DAC=y
CONFIG_DEFAULT_SECURITY=""
CONFIG_XOR_BLOCKS=y
CONFIG_ASYNC_CORE=y
CONFIG_ASYNC_MEMCPY=y
CONFIG_ASYNC_XOR=y
CONFIG_ASYNC_PQ=y
CONFIG_ASYNC_RAID6_RECOV=y
CONFIG_CRYPTO=y
CONFIG_CRYPTO_ALGAPI=m
CONFIG_CRYPTO_ALGAPI2=m
CONFIG_CRYPTO_AEAD=m
CONFIG_CRYPTO_AEAD2=m
CONFIG_CRYPTO_BLKCIPHER=m
CONFIG_CRYPTO_BLKCIPHER2=m
CONFIG_CRYPTO_HASH=m
CONFIG_CRYPTO_HASH2=m
CONFIG_CRYPTO_RNG2=m
CONFIG_CRYPTO_PCOMP2=m
CONFIG_CRYPTO_MANAGER=m
CONFIG_CRYPTO_MANAGER2=m
CONFIG_CRYPTO_MANAGER_DISABLE_TESTS=y
CONFIG_CRYPTO_WORKQUEUE=m
CONFIG_CRYPTO_AUTHENC=m
CONFIG_CRYPTO_CBC=m
CONFIG_CRYPTO_ECB=m
CONFIG_CRYPTO_HMAC=m
CONFIG_CRYPTO_CRC32C=m
CONFIG_CRYPTO_MD5=m
CONFIG_CRYPTO_SHA1=m
CONFIG_CRYPTO_ARC4=m
CONFIG_CRYPTO_DES=m
CONFIG_CRYPTO_DEFLATE=m
CONFIG_HAVE_KVM=y
CONFIG_HAVE_KVM_IRQCHIP=y
CONFIG_HAVE_KVM_EVENTFD=y
CONFIG_KVM_APIC_ARCHITECTURE=y
CONFIG_KVM_MMIO=y
CONFIG_KVM_ASYNC_PF=y
CONFIG_VIRTUALIZATION=y
CONFIG_KVM=m
CONFIG_KVM_INTEL=m
CONFIG_KVM_AMD=m
CONFIG_VHOST_NET=m
CONFIG_RAID6_PQ=y
CONFIG_BITREVERSE=y
CONFIG_GENERIC_FIND_FIRST_BIT=y
CONFIG_GENERIC_FIND_NEXT_BIT=y
CONFIG_GENERIC_FIND_LAST_BIT=y
CONFIG_CRC_CCITT=m
CONFIG_CRC16=y
CONFIG_CRC_T10DIF=y
CONFIG_CRC_ITU_T=y
CONFIG_CRC32=y
CONFIG_LIBCRC32C=m
CONFIG_ZLIB_INFLATE=y
CONFIG_ZLIB_DEFLATE=m
CONFIG_LZO_DECOMPRESS=y
CONFIG_XZ_DEC=y
CONFIG_XZ_DEC_X86=y
CONFIG_XZ_DEC_POWERPC=y
CONFIG_XZ_DEC_IA64=y
CONFIG_XZ_DEC_ARM=y
CONFIG_XZ_DEC_ARMTHUMB=y
CONFIG_XZ_DEC_SPARC=y
CONFIG_XZ_DEC_BCJ=y
CONFIG_DECOMPRESS_GZIP=y
CONFIG_DECOMPRESS_BZIP2=y
CONFIG_DECOMPRESS_LZMA=y
CONFIG_DECOMPRESS_XZ=y
CONFIG_DECOMPRESS_LZO=y
CONFIG_TEXTSEARCH=y
CONFIG_TEXTSEARCH_KMP=m
CONFIG_TEXTSEARCH_BM=m
CONFIG_TEXTSEARCH_FSM=m
CONFIG_HAS_IOMEM=y
CONFIG_HAS_IOPORT=y
CONFIG_HAS_DMA=y
CONFIG_CPU_RMAP=y
CONFIG_NLATTR=y
00:00.0 Host bridge: Advanced Micro Devices [AMD] RS880 Host Bridge
00:01.0 PCI bridge: ASRock Incorporation Device 9602
00:02.0 PCI bridge: Advanced Micro Devices [AMD] RS780 PCI to PCI bridge
(ext gfx port 0)
00:03.0 PCI bridge: Advanced Micro Devices [AMD] RS780 PCI to PCI bridge
(ext gfx port 1)
00:04.0 PCI bridge: Advanced Micro Devices [AMD] RS780 PCI to PCI bridge
(PCIE port 0)
00:09.0 PCI bridge: Advanced Micro Devices [AMD] RS780 PCI to PCI bridge
(PCIE port 4)
00:0a.0 PCI bridge: Advanced Micro Devices [AMD] RS780 PCI to PCI bridge
(PCIE port 5)
00:11.0 SATA controller: ATI Technologies Inc SB700/SB800 SATA
Controller [AHCI mode] (rev 40)
00:12.0 USB Controller: ATI Technologies Inc SB700/SB800 USB OHCI0
Controller
00:12.2 USB Controller: ATI Technologies Inc SB700/SB800 USB EHCI Controller
00:13.0 USB Controller: ATI Technologies Inc SB700/SB800 USB OHCI0
Controller
00:13.2 USB Controller: ATI Technologies Inc SB700/SB800 USB EHCI Controller
00:14.0 SMBus: ATI Technologies Inc SBx00 SMBus Controller (rev 41)
00:14.2 Audio device: ATI Technologies Inc SBx00 Azalia (Intel HDA) (rev 40)
00:14.3 ISA bridge: ATI Technologies Inc SB700/SB800 LPC host controller
(rev 40)
00:14.4 PCI bridge: ATI Technologies Inc SBx00 PCI to PCI Bridge (rev 40)
00:14.5 USB Controller: ATI Technologies Inc SB700/SB800 USB OHCI2
Controller
00:16.0 USB Controller: ATI Technologies Inc SB700/SB800 USB OHCI0
Controller
00:16.2 USB Controller: ATI Technologies Inc SB700/SB800 USB EHCI Controller
00:18.0 Host bridge: Advanced Micro Devices [AMD] Family 10h Processor
HyperTransport Configuration
00:18.1 Host bridge: Advanced Micro Devices [AMD] Family 10h Processor
Address Map
00:18.2 Host bridge: Advanced Micro Devices [AMD] Family 10h Processor
DRAM Controller
00:18.3 Host bridge: Advanced Micro Devices [AMD] Family 10h Processor
Miscellaneous Control
00:18.4 Host bridge: Advanced Micro Devices [AMD] Family 10h Processor
Link Control
01:05.0 VGA compatible controller: ATI Technologies Inc RS880 [Radeon HD
4250]
02:00.0 USB Controller: NEC Corporation uPD720200 USB 3.0 Host
Controller (rev 03)
03:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd.
RTL8111/8168B PCI Express Gigabit Ethernet controller (rev 06)
04:00.0 RAID bus controller: LSI Logic / Symbios Logic MegaRAID SAS 9240
(rev 02)
05:00.0 RAID bus controller: LSI Logic / Symbios Logic MegaRAID SAS 9240
(rev 02)
07:00.0 SCSI storage controller: Marvell Technology Group Ltd. 88SX7042
PCI-e 4-port SATA-II (rev 02)
^ permalink raw reply [flat|nested] 111+ messages in thread
* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
2011-05-31 1:24 KVM induced panic on 2.6.38[2367] & 2.6.39 Brad Campbell
@ 2011-05-31 5:47 ` Borislav Petkov
0 siblings, 0 replies; 111+ messages in thread
From: Borislav Petkov @ 2011-05-31 5:47 UTC (permalink / raw)
To: Brad Campbell; +Cc: linux-kernel, kvm, linux-mm
Looks like a KSM issue. Disabling CONFIG_KSM should at least stop your
machine from oopsing.
Adding linux-mm.
On Tue, May 31, 2011 at 09:24:03AM +0800, Brad Campbell wrote:
> G'day all,
>
> I'm running a pretty standard home server
> x86_64 Phenom-II 6 Core
> 16GB DDR 3
>
> I run some virtual machines under that. 3 x Debian 64 Bit, 1 x XP 32
> Bit. These run at boot.
>
> When I fire up another XP 32 bit instance and play with it for more
> than about 2 minutes, I get the panics included in this mail.
>
> I've included three of them here. The first and third are as booted.
> The second was with ksmd disabled just as a data point.
>
> The machine passes every load test and memory test I can throw at
> it, but I still can't rule out this being a hardware issue.
>
> Provided I don't start this XP VM the machine is quite stable, but
> running this VM will kill it within minutes.
>
> This was tested with qemu-kvm.
> The last commit in the git tree was
> commit c007db193eb6b2557acb5caf2dc4d7023639e6f3
> Author: Avi Kivity <avi@redhat.com>
> Date: Sun May 29 09:00:42 2011 -0400
> (I pulled it yesterday)
>
> These panics were captured with netconsole to a remote syslog
> daemon, and the formatting was ruined, so I've reformatted them by
> hand prior to posting.
>
> I've tested and reproduced this on 2.6.38.[2,3,6 & 7] and 2.6.39 obviously.
>
> Can anyone help shed some light on this?
>
> Regards,
> Brad
>
> [ 438.632061] general protection fault: 0000 [#1] SMP
> [ 438.632196] last sysfs file: /sys/module/x_tables/initstate
> [ 438.632242] CPU 4
> [ 438.632282] Modules linked in: xt_iprange xt_DSCP xt_length
> xt_CLASSIFY sch_sfq xt_CHECKSUM ipt_REJECT ipt_MASQUERADE
> ipt_REDIRECT xt_recent xt_state iptable_nat nf_nat nf_conntrack_ipv4
> nf_conntrack nf_defrag_ipv4 iptable_filter xt_TCPMSS xt_tcpmss
> xt_tcpudp iptable_mangle ip_tables x_tables pppoe pppox ppp_generic
> slhc cls_u32 sch_htb deflate zlib_deflate des_generic cbc ecb
> crypto_blkcipher sha1_generic md5 hmac crypto_hash cryptomgr aead
> crypto_algapi af_key fuse netconsole configfs= vhost_net powernow_k8
> mperf i2c_nforce2 kvm_amd kvm pl2303 usbserial xhci_hcd k10temp
> i2c_piix4 usb_storage usb_libusual ohci_hcd ehci_hcd usbcore ahci
> libahci r8169 mii sata_mv megaraid_sas [last unloaded:
> scsi_wait_scan]
> [ 438.634960]
> [ 438.635006] Pid: 551, comm: ksmd Not tainted 2.6.39 #3 To Be
> Filled By O.E.M. To Be Filled By O.E.M. /880G Extreme3
> [ 438.635170] RIP: 0010:[<ffffffff810b4596>] [<ffffffff810b4596>]
> remove_rmap_item_from_tree+0x96/0x150
> [ 438.635268] RSP: 0018:ffff88041c065e20 EFLAGS: 00010282
> [ 438.635314] RAX: ffff8804153bd8b0 RBX: ffff8804176c3fc0 RCX:
> 0000000000057754
> [ 438.635362] RDX: 0000880415418030 RSI: ffff880414b65003 RDI:
> ffffea000dde6030
> [ 438.635410] RBP: ffff880414b65000 R08: 0000000000057755 R09:
> 000000001bbde1bd
> [ 438.635458] R10: 000000001c28e0c3 R11: 0000000000000002 R12:
> ffffea000dde6030
> [ 438.635506] R13: ffff8804176c3f80 R14: ffff8804151ed7b0 R15:
> ffff88041bf23be0
> [ 438.635555] FS: 00007f617772e700(0000) GS:ffff88041fd00000(0000)
> knlGS:0000000000000000
> [ 438.635607] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [ 438.635654] CR2: 0000000000e70000 CR3: 0000000001583000 CR4:
> 00000000000006e0
> [ 438.635703] DR0: 0000000000000045 DR1: 0000000000000000 DR2:
> 0000000000000000
> [ 438.635750] DR3: 0000000000000005 DR6: 00000000ffff0ff0 DR7:
> 0000000000000400
> [ 438.635799] Process ksmd (pid: 551, threadinfo ffff88041c064000,
> task ffff88041d8caa70)
> [ 438.635851] Stack:
> [ 438.635893] ffffea000c43c408 ffff8804176c3fc0 000000000000036c
> ffffffff810b58f2
> [ 438.636088] ffff88041c782a00 00007fc2d81b5000 ffff88041c064000
> 00000000003808ff
> [ 438.636281] ffff88041c064000 ffff88041c065e98 ffff88041d8caa70
> ffff8804165d4480
> [ 438.636479] Call Trace:
> [ 438.636528] [<ffffffff810b58f2>] ? ksm_scan_thread+0x4e2/0xc20
> [ 438.636580] [<ffffffff81052a20>] ? wake_up_bit+0x40/0x40
> [ 438.636628] [<ffffffff810b5410>] ? try_to_merge_with_ksm_page+0x570/0x570
> [ 438.636679] [<ffffffff810b5410>] ? try_to_merge_with_ksm_page+0x570/0x570
> [ 438.636730] [<ffffffff810525b6>] ? kthread+0x96/0xa0
> [ 438.636781] [<ffffffff813d1794>] ? kernel_thread_helper+0x4/0x10
> [ 438.636832] [<ffffffff81052520>] ? kthread_worker_fn+0x120/0x120
> [ 438.636882] [<ffffffff813d1790>] ? gs_change+0xb/0xb
> [ 438.636926] Code: 28 48 89 ef e8 6c fe ff ff 48 85 c0 49 89 c4 74
> d2 f0 0f ba 28 00 19 c0 85 c0 0f 85 ae 00 00 00 00 48 8b 43 30 48 8b
> 53 38 48 85 c0
> [ 438.638504] 89 02 74 04 48 89 50 08 48 ba 00 01 10 00 00 00 00 ad de 48 b8
> [ 438.639329] RIP [<ffffffff810b4596>] remove_rmap_item_from_tree+0x96/0x150
> [ 438.639414] RSP <ffff88041c065e20>
> [ 438.639460] ---[ end trace c29fb871f6b874e3 ]---
> [ 438.639506] Kernel panic - not syncing: Fatal exception
> [ 438.639553] Pid: 551, comm: ksmd Tainted: G D 2.6.39 #3
> [ 438.639598] Call Trace:
> [ 438.639644] [<ffffffff813cd6f5>] ? panic+0x92/0x18a
> [ 438.639693] [<ffffffff81038b61>] ? kmsg_dump+0x41/0xf0
> [ 438.639745] [<ffffffff810050ad>] ? oops_end+0x8d/0xa0
> [ 438.639794] [<ffffffff813d05ef>] ? general_protection+0x1f/0x30
> [ 438.639842] [<ffffffff810b4596>] ? remove_rmap_item_from_tree+0x96/0x150
> [ 438.639891] [<ffffffff810b58f2>] ? ksm_scan_thread+0x4e2/0xc20
> [ 438.639941] [<ffffffff81052a20>] ? wake_up_bit+0x40/0x40
> [ 438.640045] [<ffffffff810b5410>] ? try_to_merge_with_ksm_page+0x570/0x570
> [ 438.640096] [<ffffffff810b5410>] ? try_to_merge_with_ksm_page+0x570/0x570
> [ 438.640147] [<ffffffff810525b6>] ? kthread+0x96/0xa0
> [ 438.640196] [<ffffffff813d1794>] ? kernel_thread_helper+0x4/0x10
> [ 438.640247] [<ffffffff81052520>] ? kthread_worker_fn+0x120/0x120
> [ 438.640297] [<ffffffff813d1790>] ? gs_change+0xb/0xb
> [ 438.640349] Rebooting in 60 seconds..
>
>
>
> [ 469.706013] general protection fault: 0000 [#1] SMP
> [ 469.706125] last sysfs file:
> /sys/devices/pci0000:00/0000:00:18.3/temp1_input
> [ 469.706162] CPU 1
> [ 469.706195] Modules linked in: xt_iprange xt_DSCP xt_length
> xt_CLASSIFY sch_sfq xt_CHECKSUM ipt_REJECT ipt_MASQUERADE
> ipt_REDIRECT xt_recent xt_state iptable_nat nf_nat nf_conntrack_ipv4
> nf_conntrack nf_defrag_ipv4 iptable_filter xt_TCPMSS xt_tcpmss
> xt_tcpudp iptable_mangle ip_tables x_tables pppoe pppox ppp_generic
> slhc cls_u32 sch_htb deflate zlib_deflate des_generi cbc ecb
> crypto_blkcipher sha1_generic md5 hmac crypto_hash cryptomgr aead
> crypto_algapi af_key fuse netconsole configfs vhost_net powernow_k8
> mperf i2c_nforce2 kvm_amd kvm pl2303 usbserial xhci_hcd i2c_piix4
> k10temp ahci usb_storage usb_libusual ohci_hcd ehci_hcd r8169
> libahci usbcore mii sata_mv megaraid_sas [last unloaded:
> scsi_wait_scan]
> [ 469.708573]
> [ 469.708608] Pid: 4942, comm: runbot Not tainted 2.6.39 #3 To Be
> Filled By O.E.M. To Be Filled By O.E.M. /880G Extreme3
> [ 469.708753] RIP: 0010:[<ffffffff810db878>] [<ffffffff810db878>]
> dup_fd+0x168/0x300
> [ 469.708829] RSP: 0018:ffff880417279dd0 EFLAGS: 00010202
> [ 469.708870] RAX: 00000000000007f8 RBX: ffff88041c1658c0 RCX:
> bfffffffffffffff
> [ 469.708907] RDX: 0000880413155540 RSI: 00000000000000ff RDI:
> 0000000000000800
> [ 469.708945] RBP: ffff88041bf22000 R08: ffff880412839dc0 R09:
> 0000000000000003
> [ 469.708981] R10: 0000000000000001 R11: 4000000000000000 R12:
> 0000000000000100
> [ 469.709018] R13: ffff88041d3cc000 R14: ffff880412839d40 R15:
> ffff88041c72f440
> [ 469.709054] FS: 00007f510c0b4700(0000) GS:ffff88041fc40000(0000)
> knlGS:0000000000000000
> [ 469.709094] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [ 469.709130] CR2: 00007f510ba56810 CR3: 0000000417c93000 CR4:
> 00000000000006e0
> [ 469.709167] DR0: 00000000000000a0 DR1: 0000000000000000 DR2:
> 0000000000000003
> [ 469.709204] DR3: 00000000000000b0 DR6: 00000000ffff0ff0 DR7:
> 0000000000000400
> [ 469.709240] Process runbot (pid: 4942, threadinfo
> ffff880417278000, task ffff880401351230)
> [ 469.709279] Stack:
> [ 469.709313] ffff880401351230 0000000000000020 ffff880412e9ca80
> ffff88041c1658d0 ffff88041c1658d0
> [ 469.709477] ffff88041723f330 0000000001200011 ffff88041723f330
> 0000000000000000
> [ 469.709641] 00007f510c0b49d0 ffff880401351230 0000000000000000
> ffffffff81037645
> [ 469.709812] Call Trace:
> [ 469.709849] [<ffffffff81037645>] ? copy_process+0xa75/0xfd0
> [ 469.709885] [<ffffffff81037c0d>] ? do_fork+0x6d/0x2b0
> [ 469.709921] [<ffffffff810dbc23>] ? alloc_fd+0x43/0x130
> [ 469.709958] [<ffffffff810c08c5>] ? fd_install+0x35/0x70
> [ 469.709998] [<ffffffff810457a9>] ? sigprocmask+0x69/0x100
> [ 469.710042] [<ffffffff813d0ca3>] ? stub_clone+0x13/0x20
> [ 469.710079] [<ffffffff813d0a3b>] ? system_call_fastpath+0x16/0x1b
> [ 469.710114] Code: 4c 89 c2 e8 6b e5 0f 00 45 85 e4 74 78 41 8d 44
> 24 ff 31 f6 41 ba 01 00 00 00 00 48 8d 3c c5 08 00 00 00 00 31 c0 eb
> 1a 0f 1f 44 00 00 f0> 48 ff 42 30 48 89 54 05 00 48 83 c0
> May 30 16:39:36 192.168.2.1 08 ff c6 48 39 f8 74 3b
> [ 469.712247] RIP [<ffffffff810db878>] dup_fd+0x168/0x300
> [ 469.712317] RSP <ffff880417279dd0>
> [ 469.712391] ---[ end trace b80596608cd6bb6b ]---
> [ 469.712432] Kernel panic - not syncing: Fatal exception
> [ 469.712470] Pid: 4942, comm: runbot Tainted: G D 2.6.39 #3
> [ 469.712508] Call Trace:
> [ 469.712544] [<ffffffff813cd6f5>] ? panic+0x92/0x18a
> [ 469.712581] [<ffffffff81038b61>] ? kmsg_dump+0x41/0xf0
> [ 469.712618] [<ffffffff810050ad>] ? oops_end+0x8d/0xa0
> [ 469.712654] [<ffffffff813d05ef>] ? general_protection+0x1f/0x30
> [ 469.712691] [<ffffffff810db878>] ? dup_fd+0x168/0x300
> [ 469.712727] [<ffffffff81037645>] ? copy_process+0xa75/0xfd0
> [ 469.712767] [<ffffffff81037c0d>] ? do_fork+0x6d/0x2b0
> [ 469.712803] [<ffffffff810dbc23>] ? alloc_fd+0x43/0x130
> [ 469.712839] [<ffffffff810c08c5>] ? fd_install+0x35/0x70
> [ 469.712875] [<ffffffff810457a9>] ? sigprocmask+0x69/0x100
> [ 469.712912] [<ffffffff813d0ca3>] ? stub_clone+0x13/0x20
> [ 469.712948] [<ffffffff813d0a3b>] ? system_call_fastpath+0x16/0x1b
> [ 469.712989] Rebooting in 60 seconds..
>
>
> [ 1100.473612] general protection fault: 0000 [#1] SMP
> [ 1100.473718] last sysfs file:
> /sys/devices/pci0000:00/0000:00:18.3/temp1_input
> [ 1100.473755] CPU 0
> [ 1100.473788] Modules linked in: xt_iprange xt_DSCP xt_length
> xt_CLASSIFY sch_sfq xt_CHECKSUM ipt_REJECT ipt_MASQUERADE
> ipt_REDIRECT xt_recent xt_state iptable_nat nf_nat nf_conntrack_ipv4
> nf_conntrack nf_defrag_ipv4 xt_TCPMSS xt_tcpmss xt_tcpudp
> iptable_mangle pppoe pppox iptable_filter ip_tables x_tables
> ppp_generic slhc cls_u32 sch_htb deflate zlib_deflate des_generic
> cbc ecb crypto_blkcipher sha1_generic md5 hmac crypto_hash cryptomgr
> aead crypto_algapi af_key fuse netconsole configfs vhost_net
> powernow_k8 mperf i2c_nforce2 kvm_amd kvm pl2303 xhci_hcd usbserial
> k10temp i2c_piix4 usb_storage usb_libusual ohci_hcd ehci_hcd ahci
> usbcore sata_mv r8169 mii libahci megaraid_sas [last unloaded:
> scsi_wait_scan]
> [ 1100.476102]
> [ 1100.476137] Pid: 551, comm: ksmd Not tainted 2.6.39 #3 To Be
> Filled By O.E.M. To Be Filled By O.E.M. /880G Extreme3
> [ 1100.476277] RIP: 0010:[<ffffffff810b5c71>] [<ffffffff810b5c71>]
> ksm_scan_thread+0x861/0xc20
> [ 1100.476351] RSP: 0018:ffff88041c065e40 EFLAGS: 00010282
> [ 1100.476387] RAX: 0000000000000000 RBX: ffffea000a5c8468 RCX:
> 0000000000000f01
> [ 1100.476424] RDX: 0000880403456828 RSI: ffff8802f5dcb002 RDI:
> ffffea000ce7eac8
> [ 1100.476460] RBP: ffff880418377fc0 R08: 6db6db6db6db6db7 R09:
> 0000160000000000
> [ 1100.476498] R10: ffffea000ce7eac8 R11: 0000000080100073 R12:
> 00000000000000ab
> [ 1100.476535] R13: ffff880418377ff8 R14: ffffea000ce7eac8 R15:
> ffff880418377fe8
> [ 1100.476572] FS: 00007f08c6498700(0000) GS:ffff88041fc00000(0000)
> knlGS:0000000000000000
> [ 1100.476611] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [ 1100.476646] CR2: 00000000e1dd3000 CR3: 00000004025f7000 CR4:
> 00000000000006f0
> [ 1100.477007] DR0: 0000000000000045 DR1: 0000000000000000 DR2:
> 0000000000000000
> [ 1100.477042] DR3: 0000000000000005 DR6: 00000000ffff0ff0 DR7:
> 0000000000000400
> [ 1100.477078] Process ksmd (pid: 551, threadinfo ffff88041c064000,
> task ffff88041d0454e0)
> [ 1100.477115] Stack:
> [ 1100.477148] ffff880403a8b100 00007fc1863cb000 ffff88041c064000
> 00000000002f5dcb
> [ 1100.477317] ffff88041c064000 ffff88041c065e98 ffff88041d0454e0
> ffff88041267a580
> [ 1100.477480] 0000000000000000 ffff88041d0454e0 ffffffff81052a20
> ffff88041c065e98
> [ 1100.477644] Call Trace:
> [ 1100.477681] [<ffffffff81052a20>] ? wake_up_bit+0x40/0x40
> [ 1100.477717] [<ffffffff810b5410>] ?
> try_to_merge_with_ksm_page+0x570/0x570
> [ 1100.477754] [<ffffffff810b5410>] ?
> try_to_merge_with_ksm_page+0x570/0x570
> [ 1100.477791] [<ffffffff810525b6>] ? kthread+0x96/0xa0
> [ 1100.477829] [<ffffffff813d1794>] ? kernel_thread_helper+0x4/0x10
> [ 1100.477865] [<ffffffff81052520>] ? kthread_worker_fn+0x120/0x120
> [ 1100.477902] [<ffffffff813d1790>] ? gs_change+0xb/0xb
> [ 1100.477937] Code: 05 99 6e 6e 00 4c 89 ff e8 ed 12 fa ff e9 6c f9
> ff ff 48 89 6c 24 38 49 83 7d 00 00 0f 84 63 01 00 00 e8 b3 8b 31 00
> 49 8b 55 00
> [ 1100.479306] 8b 42 e8 4c 8b 72 f0 48 8d 6a d8 4c 8d 78 60 48 89 04 24 4c
> [ 1100.480012] RIP [<ffffffff810b5c71>] ksm_scan_thread+0x861/0xc20
> [ 1100.480015] RSP <ffff88041c065e40>
> [ 1100.480141] ---[ end trace 7382777d9e74ea23 ]---
> [ 1100.480178] Kernel panic - not syncing: Fatal exception
> [ 1100.480216] Pid: 551, comm: ksmd Tainted: G D 2.6.39 #3
> [ 1100.480257] Call Trace:
> [ 1100.480297] [<ffffffff813cd6f5>] ? panic+0x92/0x18a
> [ 1100.480340] [<ffffffff81038b61>] ? kmsg_dump+0x41/0xf0
> [ 1100.480384] [<ffffffff810050ad>] ? oops_end+0x8d/0xa0
> [ 1100.480425] [<ffffffff813d05ef>] ? general_protection+0x1f/0x30
> [ 1100.480465] [<ffffffff810b5c71>] ? ksm_scan_thread+0x861/0xc20
> [ 1100.480507] [<ffffffff81052a20>] ? wake_up_bit+0x40/0x40
> [ 1100.480545] [<ffffffff810b5410>] ?
> try_to_merge_with_ksm_page+0x570/0x570
> [ 1100.480584] [<ffffffff810b5410>] ?
> try_to_merge_with_ksm_page+0x570/0x570
> [ 1100.480624] [<ffffffff810525b6>] ? kthread+0x96/0xa0
> [ 1100.480669] [<ffffffff813d1794>] ? kernel_thread_helper+0x4/0x10
> [ 1100.480708] [<ffffffff81052520>] ? kthread_worker_fn+0x120/0x120
> [ 1100.480748] [<ffffffff813d1790>] ? gs_change+0xb/0xb
> [ 1100.480794] Rebooting in 60 seconds..
>
> brad@srv:~$ zcat /proc/config.gz | grep -v "#"
> CONFIG_64BIT=y
> CONFIG_X86_64=y
> CONFIG_X86=y
> CONFIG_INSTRUCTION_DECODER=y
> CONFIG_OUTPUT_FORMAT="elf64-x86-64"
> CONFIG_ARCH_DEFCONFIG="arch/x86/configs/x86_64_defconfig"
> CONFIG_GENERIC_CMOS_UPDATE=y
> CONFIG_CLOCKSOURCE_WATCHDOG=y
> CONFIG_GENERIC_CLOCKEVENTS=y
> CONFIG_GENERIC_CLOCKEVENTS_BROADCAST=y
> CONFIG_LOCKDEP_SUPPORT=y
> CONFIG_STACKTRACE_SUPPORT=y
> CONFIG_HAVE_LATENCYTOP_SUPPORT=y
> CONFIG_MMU=y
> CONFIG_ZONE_DMA=y
> CONFIG_NEED_DMA_MAP_STATE=y
> CONFIG_NEED_SG_DMA_LENGTH=y
> CONFIG_GENERIC_ISA_DMA=y
> CONFIG_GENERIC_IOMAP=y
> CONFIG_GENERIC_BUG=y
> CONFIG_GENERIC_BUG_RELATIVE_POINTERS=y
> CONFIG_GENERIC_HWEIGHT=y
> CONFIG_ARCH_MAY_HAVE_PC_FDC=y
> CONFIG_RWSEM_XCHGADD_ALGORITHM=y
> CONFIG_ARCH_HAS_CPU_IDLE_WAIT=y
> CONFIG_GENERIC_CALIBRATE_DELAY=y
> CONFIG_GENERIC_TIME_VSYSCALL=y
> CONFIG_ARCH_HAS_CPU_RELAX=y
> CONFIG_ARCH_HAS_DEFAULT_IDLE=y
> CONFIG_ARCH_HAS_CACHE_LINE_SIZE=y
> CONFIG_HAVE_SETUP_PER_CPU_AREA=y
> CONFIG_NEED_PER_CPU_EMBED_FIRST_CHUNK=y
> CONFIG_NEED_PER_CPU_PAGE_FIRST_CHUNK=y
> CONFIG_HAVE_CPUMASK_OF_CPU_MAP=y
> CONFIG_ARCH_HIBERNATION_POSSIBLE=y
> CONFIG_ARCH_SUSPEND_POSSIBLE=y
> CONFIG_ZONE_DMA32=y
> CONFIG_ARCH_POPULATES_NODE_MAP=y
> CONFIG_AUDIT_ARCH=y
> CONFIG_ARCH_SUPPORTS_OPTIMIZED_INLINING=y
> CONFIG_ARCH_SUPPORTS_DEBUG_PAGEALLOC=y
> CONFIG_X86_64_SMP=y
> CONFIG_X86_HT=y
> CONFIG_ARCH_HWEIGHT_CFLAGS="-fcall-saved-rdi -fcall-saved-rsi
> -fcall-saved-rdx -fcall-saved-rcx -fcall-saved-r8 -fcall-saved-r9
> -fcall-saved-r10 -fcall-saved-r11"
> CONFIG_DEFCONFIG_LIST="/lib/modules/$UNAME_RELEASE/.config"
> CONFIG_CONSTRUCTORS=y
> CONFIG_HAVE_IRQ_WORK=y
> CONFIG_IRQ_WORK=y
>
> CONFIG_EXPERIMENTAL=y
> CONFIG_INIT_ENV_ARG_LIMIT=32
> CONFIG_CROSS_COMPILE=""
> CONFIG_LOCALVERSION=""
> CONFIG_LOCALVERSION_AUTO=y
> CONFIG_HAVE_KERNEL_GZIP=y
> CONFIG_HAVE_KERNEL_BZIP2=y
> CONFIG_HAVE_KERNEL_LZMA=y
> CONFIG_HAVE_KERNEL_XZ=y
> CONFIG_HAVE_KERNEL_LZO=y
> CONFIG_KERNEL_GZIP=y
> CONFIG_SWAP=y
> CONFIG_SYSVIPC=y
> CONFIG_SYSVIPC_SYSCTL=y
> CONFIG_BSD_PROCESS_ACCT=y
> CONFIG_TASKSTATS=y
> CONFIG_TASK_DELAY_ACCT=y
> CONFIG_TASK_XACCT=y
> CONFIG_TASK_IO_ACCOUNTING=y
> CONFIG_HAVE_GENERIC_HARDIRQS=y
>
> CONFIG_GENERIC_HARDIRQS=y
> CONFIG_HAVE_SPARSE_IRQ=y
> CONFIG_GENERIC_IRQ_PROBE=y
> CONFIG_GENERIC_IRQ_SHOW=y
> CONFIG_GENERIC_PENDING_IRQ=y
> CONFIG_IRQ_FORCED_THREADING=y
>
> CONFIG_TREE_RCU=y
> CONFIG_RCU_FANOUT=64
> CONFIG_IKCONFIG=y
> CONFIG_IKCONFIG_PROC=y
> CONFIG_LOG_BUF_SHIFT=20
> CONFIG_HAVE_UNSTABLE_SCHED_CLOCK=y
> CONFIG_NAMESPACES=y
> CONFIG_BLK_DEV_INITRD=y
> CONFIG_INITRAMFS_SOURCE=""
> CONFIG_RD_GZIP=y
> CONFIG_RD_BZIP2=y
> CONFIG_RD_LZMA=y
> CONFIG_RD_XZ=y
> CONFIG_RD_LZO=y
> CONFIG_SYSCTL=y
> CONFIG_ANON_INODES=y
> CONFIG_UID16=y
> CONFIG_SYSCTL_SYSCALL=y
> CONFIG_KALLSYMS=y
> CONFIG_KALLSYMS_ALL=y
> CONFIG_KALLSYMS_EXTRA_PASS=y
> CONFIG_HOTPLUG=y
> CONFIG_PRINTK=y
> CONFIG_BUG=y
> CONFIG_ELF_CORE=y
> CONFIG_PCSPKR_PLATFORM=y
> CONFIG_BASE_FULL=y
> CONFIG_FUTEX=y
> CONFIG_EPOLL=y
> CONFIG_SIGNALFD=y
> CONFIG_TIMERFD=y
> CONFIG_EVENTFD=y
> CONFIG_SHMEM=y
> CONFIG_AIO=y
> CONFIG_HAVE_PERF_EVENTS=y
>
> CONFIG_PERF_EVENTS=y
> CONFIG_VM_EVENT_COUNTERS=y
> CONFIG_PCI_QUIRKS=y
> CONFIG_SLUB_DEBUG=y
> CONFIG_SLUB=y
> CONFIG_HAVE_OPROFILE=y
> CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS=y
> CONFIG_USER_RETURN_NOTIFIER=y
> CONFIG_HAVE_IOREMAP_PROT=y
> CONFIG_HAVE_KPROBES=y
> CONFIG_HAVE_KRETPROBES=y
> CONFIG_HAVE_OPTPROBES=y
> CONFIG_HAVE_ARCH_TRACEHOOK=y
> CONFIG_HAVE_DMA_ATTRS=y
> CONFIG_USE_GENERIC_SMP_HELPERS=y
> CONFIG_HAVE_REGS_AND_STACK_ACCESS_API=y
> CONFIG_HAVE_DMA_API_DEBUG=y
> CONFIG_HAVE_HW_BREAKPOINT=y
> CONFIG_HAVE_MIXED_BREAKPOINTS_REGS=y
> CONFIG_HAVE_USER_RETURN_NOTIFIER=y
> CONFIG_HAVE_PERF_EVENTS_NMI=y
> CONFIG_HAVE_ARCH_JUMP_LABEL=y
>
> CONFIG_SLABINFO=y
> CONFIG_RT_MUTEXES=y
> CONFIG_BASE_SMALL=0
> CONFIG_MODULES=y
> CONFIG_MODULE_UNLOAD=y
> CONFIG_STOP_MACHINE=y
> CONFIG_BLOCK=y
> CONFIG_BLK_DEV_BSG=y
> CONFIG_BLOCK_COMPAT=y
>
> CONFIG_IOSCHED_NOOP=y
> CONFIG_IOSCHED_DEADLINE=y
> CONFIG_IOSCHED_CFQ=y
> CONFIG_DEFAULT_CFQ=y
> CONFIG_DEFAULT_IOSCHED="cfq"
> CONFIG_PREEMPT_NOTIFIERS=y
> CONFIG_INLINE_SPIN_UNLOCK=y
> CONFIG_INLINE_SPIN_UNLOCK_IRQ=y
> CONFIG_INLINE_READ_UNLOCK=y
> CONFIG_INLINE_READ_UNLOCK_IRQ=y
> CONFIG_INLINE_WRITE_UNLOCK=y
> CONFIG_INLINE_WRITE_UNLOCK_IRQ=y
> CONFIG_MUTEX_SPIN_ON_OWNER=y
>
> CONFIG_TICK_ONESHOT=y
> CONFIG_NO_HZ=y
> CONFIG_HIGH_RES_TIMERS=y
> CONFIG_GENERIC_CLOCKEVENTS_BUILD=y
> CONFIG_SMP=y
> CONFIG_X86_SUPPORTS_MEMORY_FAILURE=y
> CONFIG_SCHED_OMIT_FRAME_POINTER=y
> CONFIG_NO_BOOTMEM=y
> CONFIG_MK8=y
> CONFIG_X86_INTERNODE_CACHE_SHIFT=6
> CONFIG_X86_CMPXCHG=y
> CONFIG_CMPXCHG_LOCAL=y
> CONFIG_X86_L1_CACHE_SHIFT=6
> CONFIG_X86_XADD=y
> CONFIG_X86_WP_WORKS_OK=y
> CONFIG_X86_INTEL_USERCOPY=y
> CONFIG_X86_USE_PPRO_CHECKSUM=y
> CONFIG_X86_TSC=y
> CONFIG_X86_CMPXCHG64=y
> CONFIG_X86_CMOV=y
> CONFIG_X86_MINIMUM_CPU_FAMILY=64
> CONFIG_X86_DEBUGCTLMSR=y
> CONFIG_CPU_SUP_INTEL=y
> CONFIG_CPU_SUP_AMD=y
> CONFIG_CPU_SUP_CENTAUR=y
> CONFIG_HPET_TIMER=y
> CONFIG_HPET_EMULATE_RTC=y
> CONFIG_DMI=y
> CONFIG_GART_IOMMU=y
> CONFIG_AMD_IOMMU=y
> CONFIG_SWIOTLB=y
> CONFIG_IOMMU_HELPER=y
> CONFIG_IOMMU_API=y
> CONFIG_NR_CPUS=8
> CONFIG_SCHED_MC=y
> CONFIG_PREEMPT_NONE=y
> CONFIG_X86_LOCAL_APIC=y
> CONFIG_X86_IO_APIC=y
> CONFIG_X86_MCE=y
> CONFIG_X86_MCE_AMD=y
> CONFIG_X86_MCE_THRESHOLD=y
> CONFIG_ARCH_PHYS_ADDR_T_64BIT=y
> CONFIG_ARCH_DMA_ADDR_T_64BIT=y
> CONFIG_DIRECT_GBPAGES=y
> CONFIG_ARCH_SPARSEMEM_DEFAULT=y
> CONFIG_ARCH_SPARSEMEM_ENABLE=y
> CONFIG_ARCH_SELECT_MEMORY_MODEL=y
> CONFIG_ARCH_MEMORY_PROBE=y
> CONFIG_ILLEGAL_POINTER_VALUE=0xdead000000000000
> CONFIG_SELECT_MEMORY_MODEL=y
> CONFIG_SPARSEMEM_MANUAL=y
> CONFIG_SPARSEMEM=y
> CONFIG_HAVE_MEMORY_PRESENT=y
> CONFIG_SPARSEMEM_EXTREME=y
> CONFIG_SPARSEMEM_VMEMMAP_ENABLE=y
> CONFIG_SPARSEMEM_ALLOC_MEM_MAP_TOGETHER=y
> CONFIG_SPARSEMEM_VMEMMAP=y
> CONFIG_HAVE_MEMBLOCK=y
> CONFIG_MEMORY_HOTPLUG=y
> CONFIG_MEMORY_HOTPLUG_SPARSE=y
> CONFIG_PAGEFLAGS_EXTENDED=y
> CONFIG_SPLIT_PTLOCK_CPUS=4
> CONFIG_COMPACTION=y
> CONFIG_MIGRATION=y
> CONFIG_PHYS_ADDR_T_64BIT=y
> CONFIG_ZONE_DMA_FLAG=1
> CONFIG_BOUNCE=y
> CONFIG_VIRT_TO_BUS=y
> CONFIG_MMU_NOTIFIER=y
> CONFIG_KSM=y
> CONFIG_DEFAULT_MMAP_MIN_ADDR=4096
> CONFIG_ARCH_SUPPORTS_MEMORY_FAILURE=y
> CONFIG_TRANSPARENT_HUGEPAGE=y
> CONFIG_TRANSPARENT_HUGEPAGE_ALWAYS=y
> CONFIG_X86_RESERVE_LOW=64
> CONFIG_MTRR=y
> CONFIG_MTRR_SANITIZER=y
> CONFIG_MTRR_SANITIZER_ENABLE_DEFAULT=0
> CONFIG_MTRR_SANITIZER_SPARE_REG_NR_DEFAULT=1
> CONFIG_X86_PAT=y
> CONFIG_ARCH_USES_PG_UNCACHED=y
> CONFIG_SECCOMP=y
> CONFIG_HZ_100=y
> CONFIG_HZ=100
> CONFIG_SCHED_HRTICK=y
> CONFIG_PHYSICAL_START=0x1000000
> CONFIG_PHYSICAL_ALIGN=0x1000000
> CONFIG_ARCH_ENABLE_MEMORY_HOTPLUG=y
> CONFIG_ARCH_ENABLE_MEMORY_HOTREMOVE=y
>
> CONFIG_ACPI=y
> CONFIG_ACPI_BUTTON=y
> CONFIG_ACPI_DOCK=y
> CONFIG_ACPI_PROCESSOR=y
> CONFIG_ACPI_THERMAL=y
> CONFIG_ACPI_BLACKLIST_YEAR=0
> CONFIG_X86_PM_TIMER=y
>
> CONFIG_CPU_FREQ=y
> CONFIG_CPU_FREQ_TABLE=y
> CONFIG_CPU_FREQ_STAT=y
> CONFIG_CPU_FREQ_STAT_DETAILS=y
> CONFIG_CPU_FREQ_DEFAULT_GOV_ONDEMAND=y
> CONFIG_CPU_FREQ_GOV_PERFORMANCE=y
> CONFIG_CPU_FREQ_GOV_POWERSAVE=y
> CONFIG_CPU_FREQ_GOV_USERSPACE=y
> CONFIG_CPU_FREQ_GOV_ONDEMAND=y
> CONFIG_CPU_FREQ_GOV_CONSERVATIVE=y
>
> CONFIG_X86_ACPI_CPUFREQ=m
> CONFIG_X86_POWERNOW_K8=m
>
> CONFIG_CPU_IDLE=y
> CONFIG_CPU_IDLE_GOV_LADDER=y
> CONFIG_CPU_IDLE_GOV_MENU=y
>
>
> CONFIG_PCI=y
> CONFIG_PCI_DIRECT=y
> CONFIG_PCI_MMCONFIG=y
> CONFIG_PCI_DOMAINS=y
> CONFIG_PCIEPORTBUS=y
> CONFIG_PCIEAER=y
> CONFIG_PCIEASPM=y
> CONFIG_ARCH_SUPPORTS_MSI=y
> CONFIG_PCI_MSI=y
> CONFIG_HT_IRQ=y
> CONFIG_PCI_IOAPIC=y
> CONFIG_PCI_LABEL=y
> CONFIG_ISA_DMA_API=y
> CONFIG_AMD_NB=y
>
> CONFIG_BINFMT_ELF=y
> CONFIG_COMPAT_BINFMT_ELF=y
> CONFIG_BINFMT_MISC=m
> CONFIG_IA32_EMULATION=y
> CONFIG_IA32_AOUT=m
> CONFIG_COMPAT=y
> CONFIG_COMPAT_FOR_U64_ALIGNMENT=y
> CONFIG_SYSVIPC_COMPAT=y
> CONFIG_HAVE_TEXT_POKE_SMP=y
> CONFIG_NET=y
>
> CONFIG_PACKET=y
> CONFIG_UNIX=y
> CONFIG_XFRM=y
> CONFIG_XFRM_MIGRATE=y
> CONFIG_XFRM_IPCOMP=m
> CONFIG_NET_KEY=m
> CONFIG_NET_KEY_MIGRATE=y
> CONFIG_INET=y
> CONFIG_IP_MULTICAST=y
> CONFIG_IP_ROUTE_CLASSID=y
> CONFIG_NET_IPGRE_DEMUX=m
> CONFIG_INET_AH=m
> CONFIG_INET_ESP=m
> CONFIG_INET_IPCOMP=m
> CONFIG_INET_XFRM_TUNNEL=m
> CONFIG_INET_TUNNEL=m
> CONFIG_INET_XFRM_MODE_TRANSPORT=m
> CONFIG_INET_XFRM_MODE_TUNNEL=m
> CONFIG_INET_XFRM_MODE_BEET=m
> CONFIG_INET_DIAG=y
> CONFIG_INET_TCP_DIAG=y
> CONFIG_TCP_CONG_CUBIC=y
> CONFIG_DEFAULT_TCP_CONG="cubic"
> CONFIG_NETFILTER=y
> CONFIG_NETFILTER_ADVANCED=y
> CONFIG_BRIDGE_NETFILTER=y
>
> CONFIG_NETFILTER_NETLINK=m
> CONFIG_NETFILTER_NETLINK_QUEUE=m
> CONFIG_NETFILTER_NETLINK_LOG=m
> CONFIG_NF_CONNTRACK=m
> CONFIG_NF_CONNTRACK_MARK=y
> CONFIG_NF_CONNTRACK_ZONES=y
> CONFIG_NF_CT_PROTO_DCCP=m
> CONFIG_NF_CT_PROTO_GRE=m
> CONFIG_NF_CT_PROTO_SCTP=m
> CONFIG_NF_CT_PROTO_UDPLITE=m
> CONFIG_NF_CONNTRACK_AMANDA=m
> CONFIG_NF_CONNTRACK_FTP=m
> CONFIG_NF_CONNTRACK_H323=m
> CONFIG_NF_CONNTRACK_IRC=m
> CONFIG_NF_CONNTRACK_BROADCAST=m
> CONFIG_NF_CONNTRACK_NETBIOS_NS=m
> CONFIG_NF_CONNTRACK_SNMP=m
> CONFIG_NF_CONNTRACK_PPTP=m
> CONFIG_NF_CONNTRACK_SANE=m
> CONFIG_NF_CONNTRACK_SIP=m
> CONFIG_NF_CONNTRACK_TFTP=m
> CONFIG_NF_CT_NETLINK=m
> CONFIG_NETFILTER_TPROXY=m
> CONFIG_NETFILTER_XTABLES=m
>
> CONFIG_NETFILTER_XT_MARK=m
> CONFIG_NETFILTER_XT_CONNMARK=m
>
> CONFIG_NETFILTER_XT_TARGET_CHECKSUM=m
> CONFIG_NETFILTER_XT_TARGET_CLASSIFY=m
> CONFIG_NETFILTER_XT_TARGET_CONNMARK=m
> CONFIG_NETFILTER_XT_TARGET_CT=m
> CONFIG_NETFILTER_XT_TARGET_DSCP=m
> CONFIG_NETFILTER_XT_TARGET_HL=m
> CONFIG_NETFILTER_XT_TARGET_IDLETIMER=m
> CONFIG_NETFILTER_XT_TARGET_MARK=m
> CONFIG_NETFILTER_XT_TARGET_NFLOG=m
> CONFIG_NETFILTER_XT_TARGET_NFQUEUE=m
> CONFIG_NETFILTER_XT_TARGET_NOTRACK=m
> CONFIG_NETFILTER_XT_TARGET_RATEEST=m
> CONFIG_NETFILTER_XT_TARGET_TEE=m
> CONFIG_NETFILTER_XT_TARGET_TPROXY=m
> CONFIG_NETFILTER_XT_TARGET_TRACE=m
> CONFIG_NETFILTER_XT_TARGET_TCPMSS=m
> CONFIG_NETFILTER_XT_TARGET_TCPOPTSTRIP=m
>
> CONFIG_NETFILTER_XT_MATCH_ADDRTYPE=m
> CONFIG_NETFILTER_XT_MATCH_CLUSTER=m
> CONFIG_NETFILTER_XT_MATCH_COMMENT=m
> CONFIG_NETFILTER_XT_MATCH_CONNBYTES=m
> CONFIG_NETFILTER_XT_MATCH_CONNLIMIT=m
> CONFIG_NETFILTER_XT_MATCH_CONNMARK=m
> CONFIG_NETFILTER_XT_MATCH_CONNTRACK=m
> CONFIG_NETFILTER_XT_MATCH_CPU=m
> CONFIG_NETFILTER_XT_MATCH_DCCP=m
> CONFIG_NETFILTER_XT_MATCH_DEVGROUP=m
> CONFIG_NETFILTER_XT_MATCH_DSCP=m
> CONFIG_NETFILTER_XT_MATCH_ESP=m
> CONFIG_NETFILTER_XT_MATCH_HASHLIMIT=m
> CONFIG_NETFILTER_XT_MATCH_HELPER=m
> CONFIG_NETFILTER_XT_MATCH_HL=m
> CONFIG_NETFILTER_XT_MATCH_IPRANGE=m
> CONFIG_NETFILTER_XT_MATCH_LENGTH=m
> CONFIG_NETFILTER_XT_MATCH_LIMIT=m
> CONFIG_NETFILTER_XT_MATCH_MAC=m
> CONFIG_NETFILTER_XT_MATCH_MARK=m
> CONFIG_NETFILTER_XT_MATCH_MULTIPORT=m
> CONFIG_NETFILTER_XT_MATCH_OSF=m
> CONFIG_NETFILTER_XT_MATCH_OWNER=m
> CONFIG_NETFILTER_XT_MATCH_POLICY=m
> CONFIG_NETFILTER_XT_MATCH_PHYSDEV=m
> CONFIG_NETFILTER_XT_MATCH_PKTTYPE=m
> CONFIG_NETFILTER_XT_MATCH_QUOTA=m
> CONFIG_NETFILTER_XT_MATCH_RATEEST=m
> CONFIG_NETFILTER_XT_MATCH_REALM=m
> CONFIG_NETFILTER_XT_MATCH_RECENT=m
> CONFIG_NETFILTER_XT_MATCH_SCTP=m
> CONFIG_NETFILTER_XT_MATCH_SOCKET=m
> CONFIG_NETFILTER_XT_MATCH_STATE=m
> CONFIG_NETFILTER_XT_MATCH_STATISTIC=m
> CONFIG_NETFILTER_XT_MATCH_STRING=m
> CONFIG_NETFILTER_XT_MATCH_TCPMSS=m
> CONFIG_NETFILTER_XT_MATCH_TIME=m
> CONFIG_NETFILTER_XT_MATCH_U32=m
>
> CONFIG_NF_DEFRAG_IPV4=m
> CONFIG_NF_CONNTRACK_IPV4=m
> CONFIG_NF_CONNTRACK_PROC_COMPAT=y
> CONFIG_IP_NF_QUEUE=m
> CONFIG_IP_NF_IPTABLES=m
> CONFIG_IP_NF_MATCH_AH=m
> CONFIG_IP_NF_MATCH_ECN=m
> CONFIG_IP_NF_MATCH_TTL=m
> CONFIG_IP_NF_FILTER=m
> CONFIG_IP_NF_TARGET_REJECT=m
> CONFIG_IP_NF_TARGET_LOG=m
> CONFIG_IP_NF_TARGET_ULOG=m
> CONFIG_NF_NAT=m
> CONFIG_NF_NAT_NEEDED=y
> CONFIG_IP_NF_TARGET_MASQUERADE=m
> CONFIG_IP_NF_TARGET_NETMAP=m
> CONFIG_IP_NF_TARGET_REDIRECT=m
> CONFIG_NF_NAT_SNMP_BASIC=m
> CONFIG_NF_NAT_PROTO_DCCP=m
> CONFIG_NF_NAT_PROTO_GRE=m
> CONFIG_NF_NAT_PROTO_UDPLITE=m
> CONFIG_NF_NAT_PROTO_SCTP=m
> CONFIG_NF_NAT_FTP=m
> CONFIG_NF_NAT_IRC=m
> CONFIG_NF_NAT_TFTP=m
> CONFIG_NF_NAT_AMANDA=m
> CONFIG_NF_NAT_PPTP=m
> CONFIG_NF_NAT_H323=m
> CONFIG_NF_NAT_SIP=m
> CONFIG_IP_NF_MANGLE=m
> CONFIG_IP_NF_TARGET_CLUSTERIP=m
> CONFIG_IP_NF_TARGET_ECN=m
> CONFIG_IP_NF_TARGET_TTL=m
> CONFIG_IP_NF_RAW=m
> CONFIG_IP_NF_ARPTABLES=m
> CONFIG_IP_NF_ARPFILTER=m
> CONFIG_IP_NF_ARP_MANGLE=m
> CONFIG_IP_SCTP=m
> CONFIG_SCTP_HMAC_MD5=y
> CONFIG_STP=y
> CONFIG_BRIDGE=y
> CONFIG_BRIDGE_IGMP_SNOOPING=y
> CONFIG_VLAN_8021Q=y
> CONFIG_LLC=y
> CONFIG_NET_SCHED=y
>
> CONFIG_NET_SCH_CBQ=m
> CONFIG_NET_SCH_HTB=m
> CONFIG_NET_SCH_HFSC=m
> CONFIG_NET_SCH_PRIO=m
> CONFIG_NET_SCH_RED=m
> CONFIG_NET_SCH_SFB=m
> CONFIG_NET_SCH_SFQ=m
> CONFIG_NET_SCH_TEQL=m
> CONFIG_NET_SCH_TBF=m
> CONFIG_NET_SCH_GRED=m
> CONFIG_NET_SCH_DSMARK=m
> CONFIG_NET_SCH_NETEM=m
> CONFIG_NET_SCH_MQPRIO=m
> CONFIG_NET_SCH_CHOKE=m
> CONFIG_NET_SCH_INGRESS=m
>
> CONFIG_NET_CLS=y
> CONFIG_NET_CLS_BASIC=m
> CONFIG_NET_CLS_TCINDEX=m
> CONFIG_NET_CLS_ROUTE4=m
> CONFIG_NET_CLS_FW=m
> CONFIG_NET_CLS_U32=m
> CONFIG_NET_CLS_FLOW=m
> CONFIG_NET_EMATCH=y
> CONFIG_NET_EMATCH_STACK=32
> CONFIG_NET_EMATCH_CMP=m
> CONFIG_NET_EMATCH_NBYTE=m
> CONFIG_NET_EMATCH_U32=m
> CONFIG_NET_EMATCH_META=m
> CONFIG_NET_EMATCH_TEXT=m
> CONFIG_NET_CLS_ACT=y
> CONFIG_NET_ACT_POLICE=m
> CONFIG_NET_ACT_GACT=m
> CONFIG_GACT_PROB=y
> CONFIG_NET_ACT_MIRRED=m
> CONFIG_NET_ACT_IPT=m
> CONFIG_NET_ACT_NAT=m
> CONFIG_NET_ACT_PEDIT=m
> CONFIG_NET_ACT_SIMP=m
> CONFIG_NET_ACT_SKBEDIT=m
> CONFIG_NET_ACT_CSUM=m
> CONFIG_NET_SCH_FIFO=y
> CONFIG_RPS=y
> CONFIG_RFS_ACCEL=y
> CONFIG_XPS=y
>
>
>
> CONFIG_UEVENT_HELPER_PATH="/sbin/hotplug"
> CONFIG_DEVTMPFS=y
> CONFIG_STANDALONE=y
> CONFIG_PREVENT_FIRMWARE_BUILD=y
> CONFIG_FW_LOADER=y
> CONFIG_FIRMWARE_IN_KERNEL=y
> CONFIG_EXTRA_FIRMWARE=""
> CONFIG_ARCH_NO_SYSDEV_OPS=y
> CONFIG_PNP=y
> CONFIG_PNP_DEBUG_MESSAGES=y
>
> CONFIG_PNPACPI=y
> CONFIG_BLK_DEV=y
> CONFIG_BLK_DEV_LOOP=y
>
> CONFIG_HAVE_IDE=y
>
> CONFIG_SCSI_MOD=y
> CONFIG_RAID_ATTRS=m
> CONFIG_SCSI=y
> CONFIG_SCSI_DMA=y
> CONFIG_SCSI_PROC_FS=y
>
> CONFIG_BLK_DEV_SD=y
> CONFIG_BLK_DEV_SR=y
> CONFIG_CHR_DEV_SG=y
> CONFIG_SCSI_WAIT_SCAN=m
>
> CONFIG_SCSI_SAS_ATTRS=m
> CONFIG_SCSI_LOWLEVEL=y
> CONFIG_MEGARAID_SAS=m
> CONFIG_ATA=y
> CONFIG_ATA_VERBOSE_ERROR=y
> CONFIG_ATA_ACPI=y
> CONFIG_SATA_PMP=y
>
> CONFIG_SATA_AHCI=m
> CONFIG_SATA_SIL24=m
> CONFIG_ATA_SFF=y
>
> CONFIG_ATA_BMDMA=y
>
> CONFIG_SATA_MV=m
> CONFIG_SATA_NV=m
> CONFIG_SATA_PROMISE=m
> CONFIG_SATA_SIL=m
>
>
>
> CONFIG_MD=y
> CONFIG_BLK_DEV_MD=y
> CONFIG_MD_LINEAR=y
> CONFIG_MD_RAID0=y
> CONFIG_MD_RAID1=y
> CONFIG_MD_RAID10=y
> CONFIG_MD_RAID456=y
> CONFIG_BLK_DEV_DM=y
>
> CONFIG_NETDEVICES=y
> CONFIG_DUMMY=m
> CONFIG_BONDING=m
> CONFIG_TUN=y
> CONFIG_MII=m
> CONFIG_NET_ETHERNET=y
> CONFIG_NET_PCI=y
> CONFIG_FORCEDETH=m
> CONFIG_NETDEV_1000=y
> CONFIG_R8169=m
>
>
>
> CONFIG_PPP=m
> CONFIG_PPP_MULTILINK=y
> CONFIG_PPP_FILTER=y
> CONFIG_PPP_ASYNC=m
> CONFIG_PPP_SYNC_TTY=m
> CONFIG_PPP_DEFLATE=m
> CONFIG_PPP_BSDCOMP=m
> CONFIG_PPP_MPPE=m
> CONFIG_PPPOE=m
> CONFIG_SLHC=m
> CONFIG_NETCONSOLE=m
> CONFIG_NETCONSOLE_DYNAMIC=y
> CONFIG_NETPOLL=y
> CONFIG_NET_POLL_CONTROLLER=y
>
> CONFIG_INPUT=y
>
> CONFIG_INPUT_MOUSEDEV=y
> CONFIG_INPUT_MOUSEDEV_SCREEN_X=1024
> CONFIG_INPUT_MOUSEDEV_SCREEN_Y=768
>
> CONFIG_INPUT_KEYBOARD=y
> CONFIG_KEYBOARD_ATKBD=y
>
> CONFIG_SERIO=y
> CONFIG_SERIO_I8042=y
> CONFIG_SERIO_LIBPS2=y
>
> CONFIG_VT=y
> CONFIG_CONSOLE_TRANSLATIONS=y
> CONFIG_VT_CONSOLE=y
> CONFIG_HW_CONSOLE=y
> CONFIG_UNIX98_PTYS=y
> CONFIG_LEGACY_PTYS=y
> CONFIG_LEGACY_PTY_COUNT=256
>
> CONFIG_SERIAL_8250=y
> CONFIG_SERIAL_8250_CONSOLE=y
> CONFIG_FIX_EARLYCON_MEM=y
> CONFIG_SERIAL_8250_PCI=y
> CONFIG_SERIAL_8250_PNP=y
> CONFIG_SERIAL_8250_NR_UARTS=4
> CONFIG_SERIAL_8250_RUNTIME_UARTS=4
>
> CONFIG_SERIAL_CORE=y
> CONFIG_SERIAL_CORE_CONSOLE=y
> CONFIG_RTC=y
> CONFIG_HPET=y
> CONFIG_HPET_MMAP=y
> CONFIG_DEVPORT=y
> CONFIG_I2C=y
> CONFIG_I2C_BOARDINFO=y
> CONFIG_I2C_COMPAT=y
> CONFIG_I2C_CHARDEV=y
> CONFIG_I2C_HELPER_AUTO=y
> CONFIG_I2C_ALGOBIT=y
>
>
> CONFIG_I2C_AMD756=m
> CONFIG_I2C_PIIX4=m
> CONFIG_I2C_NFORCE2=m
> CONFIG_I2C_VIA=m
> CONFIG_I2C_VIAPRO=m
>
> CONFIG_I2C_SCMI=m
>
>
>
>
>
> CONFIG_ARCH_WANT_OPTIONAL_GPIOLIB=y
> CONFIG_HWMON=y
> CONFIG_HWMON_VID=m
>
> CONFIG_SENSORS_K8TEMP=m
> CONFIG_SENSORS_K10TEMP=m
> CONFIG_SENSORS_W83781D=m
> CONFIG_SENSORS_W83791D=m
> CONFIG_SENSORS_W83792D=m
> CONFIG_SENSORS_W83793=m
> CONFIG_SENSORS_W83795=m
> CONFIG_SENSORS_W83795_FANCTRL=y
> CONFIG_SENSORS_W83L785TS=m
> CONFIG_SENSORS_W83L786NG=m
> CONFIG_SENSORS_W83627HF=m
> CONFIG_SENSORS_W83627EHF=m
>
> CONFIG_SENSORS_ATK0110=m
> CONFIG_THERMAL=y
> CONFIG_THERMAL_HWMON=y
> CONFIG_SSB_POSSIBLE=y
>
>
> CONFIG_VGA_ARB=y
> CONFIG_VGA_ARB_MAX_GPUS=16
> CONFIG_VGASTATE=y
> CONFIG_FB=y
> CONFIG_FB_DDC=y
> CONFIG_FB_CFB_FILLRECT=y
> CONFIG_FB_CFB_COPYAREA=y
> CONFIG_FB_CFB_IMAGEBLIT=y
> CONFIG_FB_MODE_HELPERS=y
>
> CONFIG_FB_NVIDIA=y
> CONFIG_FB_NVIDIA_I2C=y
> CONFIG_FB_RADEON=y
> CONFIG_FB_RADEON_I2C=y
>
>
> CONFIG_VGA_CONSOLE=y
> CONFIG_DUMMY_CONSOLE=y
> CONFIG_FRAMEBUFFER_CONSOLE=y
> CONFIG_FONT_8x8=y
> CONFIG_FONT_8x16=y
> CONFIG_LOGO=y
> CONFIG_LOGO_LINUX_MONO=y
> CONFIG_LOGO_LINUX_VGA16=y
> CONFIG_LOGO_LINUX_CLUT224=y
> CONFIG_HID_SUPPORT=y
> CONFIG_HID=y
> CONFIG_HIDRAW=y
>
> CONFIG_USB_HID=m
> CONFIG_USB_HIDDEV=y
>
> CONFIG_HID_A4TECH=m
> CONFIG_HID_APPLE=m
> CONFIG_HID_BELKIN=m
> CONFIG_HID_CHERRY=m
> CONFIG_HID_CHICONY=m
> CONFIG_HID_CYPRESS=m
> CONFIG_HID_EZKEY=m
> CONFIG_HID_KYE=m
> CONFIG_HID_KENSINGTON=m
> CONFIG_HID_LOGITECH=m
> CONFIG_HID_MICROSOFT=m
> CONFIG_HID_MONTEREY=m
> CONFIG_USB_SUPPORT=y
> CONFIG_USB_ARCH_HAS_HCD=y
> CONFIG_USB_ARCH_HAS_OHCI=y
> CONFIG_USB_ARCH_HAS_EHCI=y
> CONFIG_USB=m
>
>
> CONFIG_USB_XHCI_HCD=m
> CONFIG_USB_EHCI_HCD=m
> CONFIG_USB_OHCI_HCD=m
> CONFIG_USB_OHCI_LITTLE_ENDIAN=y
> CONFIG_USB_UHCI_HCD=m
>
>
> CONFIG_USB_ACM=m
>
>
> CONFIG_USB_STORAGE=m
> CONFIG_USB_LIBUSUAL=y
>
>
> CONFIG_USB_SERIAL=m
> CONFIG_USB_SERIAL_GENERIC=y
> CONFIG_USB_SERIAL_FTDI_SIO=m
> CONFIG_USB_SERIAL_IPAQ=m
> CONFIG_USB_SERIAL_PL2303=m
>
>
>
> CONFIG_FIRMWARE_MEMMAP=y
> CONFIG_DMIID=y
>
> CONFIG_EXT2_FS=y
> CONFIG_EXT2_FS_XATTR=y
> CONFIG_EXT3_FS=y
> CONFIG_EXT3_FS_XATTR=y
> CONFIG_EXT4_FS=y
> CONFIG_EXT4_FS_XATTR=y
> CONFIG_JBD=y
> CONFIG_JBD2=y
> CONFIG_FS_MBCACHE=y
> CONFIG_EXPORTFS=y
> CONFIG_FILE_LOCKING=y
> CONFIG_FSNOTIFY=y
> CONFIG_DNOTIFY=y
> CONFIG_INOTIFY_USER=y
> CONFIG_FANOTIFY=y
> CONFIG_FUSE_FS=m
> CONFIG_CUSE=m
>
> CONFIG_FSCACHE=m
> CONFIG_FSCACHE_DEBUG=y
> CONFIG_CACHEFILES=m
>
> CONFIG_ISO9660_FS=m
> CONFIG_JOLIET=y
> CONFIG_ZISOFS=y
> CONFIG_UDF_FS=m
> CONFIG_UDF_NLS=y
>
> CONFIG_FAT_FS=m
> CONFIG_MSDOS_FS=m
> CONFIG_VFAT_FS=m
> CONFIG_FAT_DEFAULT_CODEPAGE=437
> CONFIG_FAT_DEFAULT_IOCHARSET="iso8859-1"
> CONFIG_NTFS_FS=m
>
> CONFIG_PROC_FS=y
> CONFIG_PROC_SYSCTL=y
> CONFIG_PROC_PAGE_MONITOR=y
> CONFIG_SYSFS=y
> CONFIG_TMPFS=y
> CONFIG_HUGETLBFS=y
> CONFIG_HUGETLB_PAGE=y
> CONFIG_CONFIGFS_FS=m
> CONFIG_NETWORK_FILESYSTEMS=y
> CONFIG_NFS_FS=y
> CONFIG_NFS_V3=y
> CONFIG_NFSD=y
> CONFIG_NFSD_DEPRECATED=y
> CONFIG_NFSD_V3=y
> CONFIG_LOCKD=y
> CONFIG_LOCKD_V4=y
> CONFIG_NFS_COMMON=y
> CONFIG_SUNRPC=y
>
> CONFIG_PARTITION_ADVANCED=y
> CONFIG_MSDOS_PARTITION=y
> CONFIG_EFI_PARTITION=y
> CONFIG_NLS=y
> CONFIG_NLS_DEFAULT="iso8859-1"
> CONFIG_NLS_CODEPAGE_437=y
> CONFIG_NLS_ISO8859_1=y
> CONFIG_DLM=m
>
> CONFIG_TRACE_IRQFLAGS_SUPPORT=y
> CONFIG_PRINTK_TIME=y
> CONFIG_DEFAULT_MESSAGE_LOGLEVEL=4
> CONFIG_ENABLE_WARN_DEPRECATED=y
> CONFIG_ENABLE_MUST_CHECK=y
> CONFIG_FRAME_WARN=2048
> CONFIG_MAGIC_SYSRQ=y
> CONFIG_DEBUG_FS=y
> CONFIG_DEBUG_KERNEL=y
> CONFIG_DETECT_HUNG_TASK=y
> CONFIG_BOOTPARAM_HUNG_TASK_PANIC_VALUE=0
> CONFIG_TIMER_STATS=y
> CONFIG_DEBUG_BUGVERBOSE=y
> CONFIG_DEBUG_MEMORY_INIT=y
> CONFIG_ARCH_WANT_FRAME_POINTERS=y
> CONFIG_SYSCTL_SYSCALL_CHECK=y
> CONFIG_USER_STACKTRACE_SUPPORT=y
> CONFIG_HAVE_FUNCTION_TRACER=y
> CONFIG_HAVE_FUNCTION_GRAPH_TRACER=y
> CONFIG_HAVE_FUNCTION_GRAPH_FP_TEST=y
> CONFIG_HAVE_FUNCTION_TRACE_MCOUNT_TEST=y
> CONFIG_HAVE_DYNAMIC_FTRACE=y
> CONFIG_HAVE_FTRACE_MCOUNT_RECORD=y
> CONFIG_HAVE_SYSCALL_TRACEPOINTS=y
> CONFIG_HAVE_C_RECORDMCOUNT=y
> CONFIG_TRACING_SUPPORT=y
> CONFIG_HAVE_ARCH_KGDB=y
> CONFIG_HAVE_ARCH_KMEMCHECK=y
> CONFIG_X86_VERBOSE_BOOTUP=y
> CONFIG_EARLY_PRINTK=y
> CONFIG_HAVE_MMIOTRACE_SUPPORT=y
> CONFIG_IO_DELAY_TYPE_0X80=0
> CONFIG_IO_DELAY_TYPE_0XED=1
> CONFIG_IO_DELAY_TYPE_UDELAY=2
> CONFIG_IO_DELAY_TYPE_NONE=3
> CONFIG_IO_DELAY_0X80=y
> CONFIG_DEFAULT_IO_DELAY_TYPE=0
>
> CONFIG_DEFAULT_SECURITY_DAC=y
> CONFIG_DEFAULT_SECURITY=""
> CONFIG_XOR_BLOCKS=y
> CONFIG_ASYNC_CORE=y
> CONFIG_ASYNC_MEMCPY=y
> CONFIG_ASYNC_XOR=y
> CONFIG_ASYNC_PQ=y
> CONFIG_ASYNC_RAID6_RECOV=y
> CONFIG_CRYPTO=y
>
> CONFIG_CRYPTO_ALGAPI=m
> CONFIG_CRYPTO_ALGAPI2=m
> CONFIG_CRYPTO_AEAD=m
> CONFIG_CRYPTO_AEAD2=m
> CONFIG_CRYPTO_BLKCIPHER=m
> CONFIG_CRYPTO_BLKCIPHER2=m
> CONFIG_CRYPTO_HASH=m
> CONFIG_CRYPTO_HASH2=m
> CONFIG_CRYPTO_RNG2=m
> CONFIG_CRYPTO_PCOMP2=m
> CONFIG_CRYPTO_MANAGER=m
> CONFIG_CRYPTO_MANAGER2=m
> CONFIG_CRYPTO_MANAGER_DISABLE_TESTS=y
> CONFIG_CRYPTO_WORKQUEUE=m
> CONFIG_CRYPTO_AUTHENC=m
>
>
> CONFIG_CRYPTO_CBC=m
> CONFIG_CRYPTO_ECB=m
>
> CONFIG_CRYPTO_HMAC=m
>
> CONFIG_CRYPTO_CRC32C=m
> CONFIG_CRYPTO_MD5=m
> CONFIG_CRYPTO_SHA1=m
>
> CONFIG_CRYPTO_ARC4=m
> CONFIG_CRYPTO_DES=m
>
> CONFIG_CRYPTO_DEFLATE=m
>
> CONFIG_HAVE_KVM=y
> CONFIG_HAVE_KVM_IRQCHIP=y
> CONFIG_HAVE_KVM_EVENTFD=y
> CONFIG_KVM_APIC_ARCHITECTURE=y
> CONFIG_KVM_MMIO=y
> CONFIG_KVM_ASYNC_PF=y
> CONFIG_VIRTUALIZATION=y
> CONFIG_KVM=m
> CONFIG_KVM_INTEL=m
> CONFIG_KVM_AMD=m
> CONFIG_VHOST_NET=m
>
> CONFIG_RAID6_PQ=y
> CONFIG_BITREVERSE=y
> CONFIG_GENERIC_FIND_FIRST_BIT=y
> CONFIG_GENERIC_FIND_NEXT_BIT=y
> CONFIG_GENERIC_FIND_LAST_BIT=y
> CONFIG_CRC_CCITT=m
> CONFIG_CRC16=y
> CONFIG_CRC_T10DIF=y
> CONFIG_CRC_ITU_T=y
> CONFIG_CRC32=y
> CONFIG_LIBCRC32C=m
> CONFIG_ZLIB_INFLATE=y
> CONFIG_ZLIB_DEFLATE=m
> CONFIG_LZO_DECOMPRESS=y
> CONFIG_XZ_DEC=y
> CONFIG_XZ_DEC_X86=y
> CONFIG_XZ_DEC_POWERPC=y
> CONFIG_XZ_DEC_IA64=y
> CONFIG_XZ_DEC_ARM=y
> CONFIG_XZ_DEC_ARMTHUMB=y
> CONFIG_XZ_DEC_SPARC=y
> CONFIG_XZ_DEC_BCJ=y
> CONFIG_DECOMPRESS_GZIP=y
> CONFIG_DECOMPRESS_BZIP2=y
> CONFIG_DECOMPRESS_LZMA=y
> CONFIG_DECOMPRESS_XZ=y
> CONFIG_DECOMPRESS_LZO=y
> CONFIG_TEXTSEARCH=y
> CONFIG_TEXTSEARCH_KMP=m
> CONFIG_TEXTSEARCH_BM=m
> CONFIG_TEXTSEARCH_FSM=m
> CONFIG_HAS_IOMEM=y
> CONFIG_HAS_IOPORT=y
> CONFIG_HAS_DMA=y
> CONFIG_CPU_RMAP=y
> CONFIG_NLATTR=y
>
> 00:00.0 Host bridge: Advanced Micro Devices [AMD] RS880 Host Bridge
> 00:01.0 PCI bridge: ASRock Incorporation Device 9602
> 00:02.0 PCI bridge: Advanced Micro Devices [AMD] RS780 PCI to PCI
> bridge (ext gfx port 0)
> 00:03.0 PCI bridge: Advanced Micro Devices [AMD] RS780 PCI to PCI
> bridge (ext gfx port 1)
> 00:04.0 PCI bridge: Advanced Micro Devices [AMD] RS780 PCI to PCI
> bridge (PCIE port 0)
> 00:09.0 PCI bridge: Advanced Micro Devices [AMD] RS780 PCI to PCI
> bridge (PCIE port 4)
> 00:0a.0 PCI bridge: Advanced Micro Devices [AMD] RS780 PCI to PCI
> bridge (PCIE port 5)
> 00:11.0 SATA controller: ATI Technologies Inc SB700/SB800 SATA
> Controller [AHCI mode] (rev 40)
> 00:12.0 USB Controller: ATI Technologies Inc SB700/SB800 USB OHCI0
> Controller
> 00:12.2 USB Controller: ATI Technologies Inc SB700/SB800 USB EHCI Controller
> 00:13.0 USB Controller: ATI Technologies Inc SB700/SB800 USB OHCI0
> Controller
> 00:13.2 USB Controller: ATI Technologies Inc SB700/SB800 USB EHCI Controller
> 00:14.0 SMBus: ATI Technologies Inc SBx00 SMBus Controller (rev 41)
> 00:14.2 Audio device: ATI Technologies Inc SBx00 Azalia (Intel HDA) (rev 40)
> 00:14.3 ISA bridge: ATI Technologies Inc SB700/SB800 LPC host
> controller (rev 40)
> 00:14.4 PCI bridge: ATI Technologies Inc SBx00 PCI to PCI Bridge (rev 40)
> 00:14.5 USB Controller: ATI Technologies Inc SB700/SB800 USB OHCI2
> Controller
> 00:16.0 USB Controller: ATI Technologies Inc SB700/SB800 USB OHCI0
> Controller
> 00:16.2 USB Controller: ATI Technologies Inc SB700/SB800 USB EHCI Controller
> 00:18.0 Host bridge: Advanced Micro Devices [AMD] Family 10h
> Processor HyperTransport Configuration
> 00:18.1 Host bridge: Advanced Micro Devices [AMD] Family 10h
> Processor Address Map
> 00:18.2 Host bridge: Advanced Micro Devices [AMD] Family 10h
> Processor DRAM Controller
> 00:18.3 Host bridge: Advanced Micro Devices [AMD] Family 10h
> Processor Miscellaneous Control
> 00:18.4 Host bridge: Advanced Micro Devices [AMD] Family 10h
> Processor Link Control
> 01:05.0 VGA compatible controller: ATI Technologies Inc RS880
> [Radeon HD 4250]
> 02:00.0 USB Controller: NEC Corporation uPD720200 USB 3.0 Host
> Controller (rev 03)
> 03:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd.
> RTL8111/8168B PCI Express Gigabit Ethernet controller (rev 06)
> 04:00.0 RAID bus controller: LSI Logic / Symbios Logic MegaRAID SAS
> 9240 (rev 02)
> 05:00.0 RAID bus controller: LSI Logic / Symbios Logic MegaRAID SAS
> 9240 (rev 02)
> 07:00.0 SCSI storage controller: Marvell Technology Group Ltd.
> 88SX7042 PCI-e 4-port SATA-II (rev 02)
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/
--
Regards/Gruss,
Boris.
^ permalink raw reply [flat|nested] 111+ messages in thread
* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
@ 2011-05-31 5:47 ` Borislav Petkov
0 siblings, 0 replies; 111+ messages in thread
From: Borislav Petkov @ 2011-05-31 5:47 UTC (permalink / raw)
To: Brad Campbell; +Cc: linux-kernel, kvm, linux-mm
Looks like a KSM issue. Disabling CONFIG_KSM should at least stop your
machine from oopsing.
Adding linux-mm.
On Tue, May 31, 2011 at 09:24:03AM +0800, Brad Campbell wrote:
> G'day all,
>
> I'm running a pretty standard home server
> x86_64 Phenom-II 6 Core
> 16GB DDR 3
>
> I run some virtual machines under that. 3 x Debian 64 Bit, 1 x XP 32
> Bit. These run at boot.
>
> When I fire up another XP 32 bit instance and play with it for more
> than about 2 minutes, I get the panics included in this mail.
>
> I've included three of them here. The first and third are as booted.
> The second was with ksmd disabled just as a data point.
>
> The machine passes every load test and memory test I can throw at
> it, but I still can't rule out this being a hardware issue.
>
> Provided I don't start this XP VM the machine is quite stable, but
> running this VM will kill it within minutes.
>
> This was tested with qemu-kvm.
> The last commit in the git tree was
> commit c007db193eb6b2557acb5caf2dc4d7023639e6f3
> Author: Avi Kivity <avi@redhat.com>
> Date: Sun May 29 09:00:42 2011 -0400
> (I pulled it yesterday)
>
> These panics were captured with netconsole to a remote syslog
> daemon, and the formatting was ruined, so I've reformatted them by
> hand prior to posting.
>
> I've tested and reproduced this on 2.6.38.[2,3,6 & 7] and 2.6.39 obviously.
>
> Can anyone help shed some light on this?
>
> Regards,
> Brad
>
> [ 438.632061] general protection fault: 0000 [#1] SMP
> [ 438.632196] last sysfs file: /sys/module/x_tables/initstate
> [ 438.632242] CPU 4
> [ 438.632282] Modules linked in: xt_iprange xt_DSCP xt_length
> xt_CLASSIFY sch_sfq xt_CHECKSUM ipt_REJECT ipt_MASQUERADE
> ipt_REDIRECT xt_recent xt_state iptable_nat nf_nat nf_conntrack_ipv4
> nf_conntrack nf_defrag_ipv4 iptable_filter xt_TCPMSS xt_tcpmss
> xt_tcpudp iptable_mangle ip_tables x_tables pppoe pppox ppp_generic
> slhc cls_u32 sch_htb deflate zlib_deflate des_generic cbc ecb
> crypto_blkcipher sha1_generic md5 hmac crypto_hash cryptomgr aead
> crypto_algapi af_key fuse netconsole configfs= vhost_net powernow_k8
> mperf i2c_nforce2 kvm_amd kvm pl2303 usbserial xhci_hcd k10temp
> i2c_piix4 usb_storage usb_libusual ohci_hcd ehci_hcd usbcore ahci
> libahci r8169 mii sata_mv megaraid_sas [last unloaded:
> scsi_wait_scan]
> [ 438.634960]
> [ 438.635006] Pid: 551, comm: ksmd Not tainted 2.6.39 #3 To Be
> Filled By O.E.M. To Be Filled By O.E.M. /880G Extreme3
> [ 438.635170] RIP: 0010:[<ffffffff810b4596>] [<ffffffff810b4596>]
> remove_rmap_item_from_tree+0x96/0x150
> [ 438.635268] RSP: 0018:ffff88041c065e20 EFLAGS: 00010282
> [ 438.635314] RAX: ffff8804153bd8b0 RBX: ffff8804176c3fc0 RCX:
> 0000000000057754
> [ 438.635362] RDX: 0000880415418030 RSI: ffff880414b65003 RDI:
> ffffea000dde6030
> [ 438.635410] RBP: ffff880414b65000 R08: 0000000000057755 R09:
> 000000001bbde1bd
> [ 438.635458] R10: 000000001c28e0c3 R11: 0000000000000002 R12:
> ffffea000dde6030
> [ 438.635506] R13: ffff8804176c3f80 R14: ffff8804151ed7b0 R15:
> ffff88041bf23be0
> [ 438.635555] FS: 00007f617772e700(0000) GS:ffff88041fd00000(0000)
> knlGS:0000000000000000
> [ 438.635607] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [ 438.635654] CR2: 0000000000e70000 CR3: 0000000001583000 CR4:
> 00000000000006e0
> [ 438.635703] DR0: 0000000000000045 DR1: 0000000000000000 DR2:
> 0000000000000000
> [ 438.635750] DR3: 0000000000000005 DR6: 00000000ffff0ff0 DR7:
> 0000000000000400
> [ 438.635799] Process ksmd (pid: 551, threadinfo ffff88041c064000,
> task ffff88041d8caa70)
> [ 438.635851] Stack:
> [ 438.635893] ffffea000c43c408 ffff8804176c3fc0 000000000000036c
> ffffffff810b58f2
> [ 438.636088] ffff88041c782a00 00007fc2d81b5000 ffff88041c064000
> 00000000003808ff
> [ 438.636281] ffff88041c064000 ffff88041c065e98 ffff88041d8caa70
> ffff8804165d4480
> [ 438.636479] Call Trace:
> [ 438.636528] [<ffffffff810b58f2>] ? ksm_scan_thread+0x4e2/0xc20
> [ 438.636580] [<ffffffff81052a20>] ? wake_up_bit+0x40/0x40
> [ 438.636628] [<ffffffff810b5410>] ? try_to_merge_with_ksm_page+0x570/0x570
> [ 438.636679] [<ffffffff810b5410>] ? try_to_merge_with_ksm_page+0x570/0x570
> [ 438.636730] [<ffffffff810525b6>] ? kthread+0x96/0xa0
> [ 438.636781] [<ffffffff813d1794>] ? kernel_thread_helper+0x4/0x10
> [ 438.636832] [<ffffffff81052520>] ? kthread_worker_fn+0x120/0x120
> [ 438.636882] [<ffffffff813d1790>] ? gs_change+0xb/0xb
> [ 438.636926] Code: 28 48 89 ef e8 6c fe ff ff 48 85 c0 49 89 c4 74
> d2 f0 0f ba 28 00 19 c0 85 c0 0f 85 ae 00 00 00 00 48 8b 43 30 48 8b
> 53 38 48 85 c0
> [ 438.638504] 89 02 74 04 48 89 50 08 48 ba 00 01 10 00 00 00 00 ad de 48 b8
> [ 438.639329] RIP [<ffffffff810b4596>] remove_rmap_item_from_tree+0x96/0x150
> [ 438.639414] RSP <ffff88041c065e20>
> [ 438.639460] ---[ end trace c29fb871f6b874e3 ]---
> [ 438.639506] Kernel panic - not syncing: Fatal exception
> [ 438.639553] Pid: 551, comm: ksmd Tainted: G D 2.6.39 #3
> [ 438.639598] Call Trace:
> [ 438.639644] [<ffffffff813cd6f5>] ? panic+0x92/0x18a
> [ 438.639693] [<ffffffff81038b61>] ? kmsg_dump+0x41/0xf0
> [ 438.639745] [<ffffffff810050ad>] ? oops_end+0x8d/0xa0
> [ 438.639794] [<ffffffff813d05ef>] ? general_protection+0x1f/0x30
> [ 438.639842] [<ffffffff810b4596>] ? remove_rmap_item_from_tree+0x96/0x150
> [ 438.639891] [<ffffffff810b58f2>] ? ksm_scan_thread+0x4e2/0xc20
> [ 438.639941] [<ffffffff81052a20>] ? wake_up_bit+0x40/0x40
> [ 438.640045] [<ffffffff810b5410>] ? try_to_merge_with_ksm_page+0x570/0x570
> [ 438.640096] [<ffffffff810b5410>] ? try_to_merge_with_ksm_page+0x570/0x570
> [ 438.640147] [<ffffffff810525b6>] ? kthread+0x96/0xa0
> [ 438.640196] [<ffffffff813d1794>] ? kernel_thread_helper+0x4/0x10
> [ 438.640247] [<ffffffff81052520>] ? kthread_worker_fn+0x120/0x120
> [ 438.640297] [<ffffffff813d1790>] ? gs_change+0xb/0xb
> [ 438.640349] Rebooting in 60 seconds..
>
>
>
> [ 469.706013] general protection fault: 0000 [#1] SMP
> [ 469.706125] last sysfs file:
> /sys/devices/pci0000:00/0000:00:18.3/temp1_input
> [ 469.706162] CPU 1
> [ 469.706195] Modules linked in: xt_iprange xt_DSCP xt_length
> xt_CLASSIFY sch_sfq xt_CHECKSUM ipt_REJECT ipt_MASQUERADE
> ipt_REDIRECT xt_recent xt_state iptable_nat nf_nat nf_conntrack_ipv4
> nf_conntrack nf_defrag_ipv4 iptable_filter xt_TCPMSS xt_tcpmss
> xt_tcpudp iptable_mangle ip_tables x_tables pppoe pppox ppp_generic
> slhc cls_u32 sch_htb deflate zlib_deflate des_generi cbc ecb
> crypto_blkcipher sha1_generic md5 hmac crypto_hash cryptomgr aead
> crypto_algapi af_key fuse netconsole configfs vhost_net powernow_k8
> mperf i2c_nforce2 kvm_amd kvm pl2303 usbserial xhci_hcd i2c_piix4
> k10temp ahci usb_storage usb_libusual ohci_hcd ehci_hcd r8169
> libahci usbcore mii sata_mv megaraid_sas [last unloaded:
> scsi_wait_scan]
> [ 469.708573]
> [ 469.708608] Pid: 4942, comm: runbot Not tainted 2.6.39 #3 To Be
> Filled By O.E.M. To Be Filled By O.E.M. /880G Extreme3
> [ 469.708753] RIP: 0010:[<ffffffff810db878>] [<ffffffff810db878>]
> dup_fd+0x168/0x300
> [ 469.708829] RSP: 0018:ffff880417279dd0 EFLAGS: 00010202
> [ 469.708870] RAX: 00000000000007f8 RBX: ffff88041c1658c0 RCX:
> bfffffffffffffff
> [ 469.708907] RDX: 0000880413155540 RSI: 00000000000000ff RDI:
> 0000000000000800
> [ 469.708945] RBP: ffff88041bf22000 R08: ffff880412839dc0 R09:
> 0000000000000003
> [ 469.708981] R10: 0000000000000001 R11: 4000000000000000 R12:
> 0000000000000100
> [ 469.709018] R13: ffff88041d3cc000 R14: ffff880412839d40 R15:
> ffff88041c72f440
> [ 469.709054] FS: 00007f510c0b4700(0000) GS:ffff88041fc40000(0000)
> knlGS:0000000000000000
> [ 469.709094] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [ 469.709130] CR2: 00007f510ba56810 CR3: 0000000417c93000 CR4:
> 00000000000006e0
> [ 469.709167] DR0: 00000000000000a0 DR1: 0000000000000000 DR2:
> 0000000000000003
> [ 469.709204] DR3: 00000000000000b0 DR6: 00000000ffff0ff0 DR7:
> 0000000000000400
> [ 469.709240] Process runbot (pid: 4942, threadinfo
> ffff880417278000, task ffff880401351230)
> [ 469.709279] Stack:
> [ 469.709313] ffff880401351230 0000000000000020 ffff880412e9ca80
> ffff88041c1658d0 ffff88041c1658d0
> [ 469.709477] ffff88041723f330 0000000001200011 ffff88041723f330
> 0000000000000000
> [ 469.709641] 00007f510c0b49d0 ffff880401351230 0000000000000000
> ffffffff81037645
> [ 469.709812] Call Trace:
> [ 469.709849] [<ffffffff81037645>] ? copy_process+0xa75/0xfd0
> [ 469.709885] [<ffffffff81037c0d>] ? do_fork+0x6d/0x2b0
> [ 469.709921] [<ffffffff810dbc23>] ? alloc_fd+0x43/0x130
> [ 469.709958] [<ffffffff810c08c5>] ? fd_install+0x35/0x70
> [ 469.709998] [<ffffffff810457a9>] ? sigprocmask+0x69/0x100
> [ 469.710042] [<ffffffff813d0ca3>] ? stub_clone+0x13/0x20
> [ 469.710079] [<ffffffff813d0a3b>] ? system_call_fastpath+0x16/0x1b
> [ 469.710114] Code: 4c 89 c2 e8 6b e5 0f 00 45 85 e4 74 78 41 8d 44
> 24 ff 31 f6 41 ba 01 00 00 00 00 48 8d 3c c5 08 00 00 00 00 31 c0 eb
> 1a 0f 1f 44 00 00 f0> 48 ff 42 30 48 89 54 05 00 48 83 c0
> May 30 16:39:36 192.168.2.1 08 ff c6 48 39 f8 74 3b
> [ 469.712247] RIP [<ffffffff810db878>] dup_fd+0x168/0x300
> [ 469.712317] RSP <ffff880417279dd0>
> [ 469.712391] ---[ end trace b80596608cd6bb6b ]---
> [ 469.712432] Kernel panic - not syncing: Fatal exception
> [ 469.712470] Pid: 4942, comm: runbot Tainted: G D 2.6.39 #3
> [ 469.712508] Call Trace:
> [ 469.712544] [<ffffffff813cd6f5>] ? panic+0x92/0x18a
> [ 469.712581] [<ffffffff81038b61>] ? kmsg_dump+0x41/0xf0
> [ 469.712618] [<ffffffff810050ad>] ? oops_end+0x8d/0xa0
> [ 469.712654] [<ffffffff813d05ef>] ? general_protection+0x1f/0x30
> [ 469.712691] [<ffffffff810db878>] ? dup_fd+0x168/0x300
> [ 469.712727] [<ffffffff81037645>] ? copy_process+0xa75/0xfd0
> [ 469.712767] [<ffffffff81037c0d>] ? do_fork+0x6d/0x2b0
> [ 469.712803] [<ffffffff810dbc23>] ? alloc_fd+0x43/0x130
> [ 469.712839] [<ffffffff810c08c5>] ? fd_install+0x35/0x70
> [ 469.712875] [<ffffffff810457a9>] ? sigprocmask+0x69/0x100
> [ 469.712912] [<ffffffff813d0ca3>] ? stub_clone+0x13/0x20
> [ 469.712948] [<ffffffff813d0a3b>] ? system_call_fastpath+0x16/0x1b
> [ 469.712989] Rebooting in 60 seconds..
>
>
> [ 1100.473612] general protection fault: 0000 [#1] SMP
> [ 1100.473718] last sysfs file:
> /sys/devices/pci0000:00/0000:00:18.3/temp1_input
> [ 1100.473755] CPU 0
> [ 1100.473788] Modules linked in: xt_iprange xt_DSCP xt_length
> xt_CLASSIFY sch_sfq xt_CHECKSUM ipt_REJECT ipt_MASQUERADE
> ipt_REDIRECT xt_recent xt_state iptable_nat nf_nat nf_conntrack_ipv4
> nf_conntrack nf_defrag_ipv4 xt_TCPMSS xt_tcpmss xt_tcpudp
> iptable_mangle pppoe pppox iptable_filter ip_tables x_tables
> ppp_generic slhc cls_u32 sch_htb deflate zlib_deflate des_generic
> cbc ecb crypto_blkcipher sha1_generic md5 hmac crypto_hash cryptomgr
> aead crypto_algapi af_key fuse netconsole configfs vhost_net
> powernow_k8 mperf i2c_nforce2 kvm_amd kvm pl2303 xhci_hcd usbserial
> k10temp i2c_piix4 usb_storage usb_libusual ohci_hcd ehci_hcd ahci
> usbcore sata_mv r8169 mii libahci megaraid_sas [last unloaded:
> scsi_wait_scan]
> [ 1100.476102]
> [ 1100.476137] Pid: 551, comm: ksmd Not tainted 2.6.39 #3 To Be
> Filled By O.E.M. To Be Filled By O.E.M. /880G Extreme3
> [ 1100.476277] RIP: 0010:[<ffffffff810b5c71>] [<ffffffff810b5c71>]
> ksm_scan_thread+0x861/0xc20
> [ 1100.476351] RSP: 0018:ffff88041c065e40 EFLAGS: 00010282
> [ 1100.476387] RAX: 0000000000000000 RBX: ffffea000a5c8468 RCX:
> 0000000000000f01
> [ 1100.476424] RDX: 0000880403456828 RSI: ffff8802f5dcb002 RDI:
> ffffea000ce7eac8
> [ 1100.476460] RBP: ffff880418377fc0 R08: 6db6db6db6db6db7 R09:
> 0000160000000000
> [ 1100.476498] R10: ffffea000ce7eac8 R11: 0000000080100073 R12:
> 00000000000000ab
> [ 1100.476535] R13: ffff880418377ff8 R14: ffffea000ce7eac8 R15:
> ffff880418377fe8
> [ 1100.476572] FS: 00007f08c6498700(0000) GS:ffff88041fc00000(0000)
> knlGS:0000000000000000
> [ 1100.476611] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [ 1100.476646] CR2: 00000000e1dd3000 CR3: 00000004025f7000 CR4:
> 00000000000006f0
> [ 1100.477007] DR0: 0000000000000045 DR1: 0000000000000000 DR2:
> 0000000000000000
> [ 1100.477042] DR3: 0000000000000005 DR6: 00000000ffff0ff0 DR7:
> 0000000000000400
> [ 1100.477078] Process ksmd (pid: 551, threadinfo ffff88041c064000,
> task ffff88041d0454e0)
> [ 1100.477115] Stack:
> [ 1100.477148] ffff880403a8b100 00007fc1863cb000 ffff88041c064000
> 00000000002f5dcb
> [ 1100.477317] ffff88041c064000 ffff88041c065e98 ffff88041d0454e0
> ffff88041267a580
> [ 1100.477480] 0000000000000000 ffff88041d0454e0 ffffffff81052a20
> ffff88041c065e98
> [ 1100.477644] Call Trace:
> [ 1100.477681] [<ffffffff81052a20>] ? wake_up_bit+0x40/0x40
> [ 1100.477717] [<ffffffff810b5410>] ?
> try_to_merge_with_ksm_page+0x570/0x570
> [ 1100.477754] [<ffffffff810b5410>] ?
> try_to_merge_with_ksm_page+0x570/0x570
> [ 1100.477791] [<ffffffff810525b6>] ? kthread+0x96/0xa0
> [ 1100.477829] [<ffffffff813d1794>] ? kernel_thread_helper+0x4/0x10
> [ 1100.477865] [<ffffffff81052520>] ? kthread_worker_fn+0x120/0x120
> [ 1100.477902] [<ffffffff813d1790>] ? gs_change+0xb/0xb
> [ 1100.477937] Code: 05 99 6e 6e 00 4c 89 ff e8 ed 12 fa ff e9 6c f9
> ff ff 48 89 6c 24 38 49 83 7d 00 00 0f 84 63 01 00 00 e8 b3 8b 31 00
> 49 8b 55 00
> [ 1100.479306] 8b 42 e8 4c 8b 72 f0 48 8d 6a d8 4c 8d 78 60 48 89 04 24 4c
> [ 1100.480012] RIP [<ffffffff810b5c71>] ksm_scan_thread+0x861/0xc20
> [ 1100.480015] RSP <ffff88041c065e40>
> [ 1100.480141] ---[ end trace 7382777d9e74ea23 ]---
> [ 1100.480178] Kernel panic - not syncing: Fatal exception
> [ 1100.480216] Pid: 551, comm: ksmd Tainted: G D 2.6.39 #3
> [ 1100.480257] Call Trace:
> [ 1100.480297] [<ffffffff813cd6f5>] ? panic+0x92/0x18a
> [ 1100.480340] [<ffffffff81038b61>] ? kmsg_dump+0x41/0xf0
> [ 1100.480384] [<ffffffff810050ad>] ? oops_end+0x8d/0xa0
> [ 1100.480425] [<ffffffff813d05ef>] ? general_protection+0x1f/0x30
> [ 1100.480465] [<ffffffff810b5c71>] ? ksm_scan_thread+0x861/0xc20
> [ 1100.480507] [<ffffffff81052a20>] ? wake_up_bit+0x40/0x40
> [ 1100.480545] [<ffffffff810b5410>] ?
> try_to_merge_with_ksm_page+0x570/0x570
> [ 1100.480584] [<ffffffff810b5410>] ?
> try_to_merge_with_ksm_page+0x570/0x570
> [ 1100.480624] [<ffffffff810525b6>] ? kthread+0x96/0xa0
> [ 1100.480669] [<ffffffff813d1794>] ? kernel_thread_helper+0x4/0x10
> [ 1100.480708] [<ffffffff81052520>] ? kthread_worker_fn+0x120/0x120
> [ 1100.480748] [<ffffffff813d1790>] ? gs_change+0xb/0xb
> [ 1100.480794] Rebooting in 60 seconds..
>
> brad@srv:~$ zcat /proc/config.gz | grep -v "#"
> CONFIG_64BIT=y
> CONFIG_X86_64=y
> CONFIG_X86=y
> CONFIG_INSTRUCTION_DECODER=y
> CONFIG_OUTPUT_FORMAT="elf64-x86-64"
> CONFIG_ARCH_DEFCONFIG="arch/x86/configs/x86_64_defconfig"
> CONFIG_GENERIC_CMOS_UPDATE=y
> CONFIG_CLOCKSOURCE_WATCHDOG=y
> CONFIG_GENERIC_CLOCKEVENTS=y
> CONFIG_GENERIC_CLOCKEVENTS_BROADCAST=y
> CONFIG_LOCKDEP_SUPPORT=y
> CONFIG_STACKTRACE_SUPPORT=y
> CONFIG_HAVE_LATENCYTOP_SUPPORT=y
> CONFIG_MMU=y
> CONFIG_ZONE_DMA=y
> CONFIG_NEED_DMA_MAP_STATE=y
> CONFIG_NEED_SG_DMA_LENGTH=y
> CONFIG_GENERIC_ISA_DMA=y
> CONFIG_GENERIC_IOMAP=y
> CONFIG_GENERIC_BUG=y
> CONFIG_GENERIC_BUG_RELATIVE_POINTERS=y
> CONFIG_GENERIC_HWEIGHT=y
> CONFIG_ARCH_MAY_HAVE_PC_FDC=y
> CONFIG_RWSEM_XCHGADD_ALGORITHM=y
> CONFIG_ARCH_HAS_CPU_IDLE_WAIT=y
> CONFIG_GENERIC_CALIBRATE_DELAY=y
> CONFIG_GENERIC_TIME_VSYSCALL=y
> CONFIG_ARCH_HAS_CPU_RELAX=y
> CONFIG_ARCH_HAS_DEFAULT_IDLE=y
> CONFIG_ARCH_HAS_CACHE_LINE_SIZE=y
> CONFIG_HAVE_SETUP_PER_CPU_AREA=y
> CONFIG_NEED_PER_CPU_EMBED_FIRST_CHUNK=y
> CONFIG_NEED_PER_CPU_PAGE_FIRST_CHUNK=y
> CONFIG_HAVE_CPUMASK_OF_CPU_MAP=y
> CONFIG_ARCH_HIBERNATION_POSSIBLE=y
> CONFIG_ARCH_SUSPEND_POSSIBLE=y
> CONFIG_ZONE_DMA32=y
> CONFIG_ARCH_POPULATES_NODE_MAP=y
> CONFIG_AUDIT_ARCH=y
> CONFIG_ARCH_SUPPORTS_OPTIMIZED_INLINING=y
> CONFIG_ARCH_SUPPORTS_DEBUG_PAGEALLOC=y
> CONFIG_X86_64_SMP=y
> CONFIG_X86_HT=y
> CONFIG_ARCH_HWEIGHT_CFLAGS="-fcall-saved-rdi -fcall-saved-rsi
> -fcall-saved-rdx -fcall-saved-rcx -fcall-saved-r8 -fcall-saved-r9
> -fcall-saved-r10 -fcall-saved-r11"
> CONFIG_DEFCONFIG_LIST="/lib/modules/$UNAME_RELEASE/.config"
> CONFIG_CONSTRUCTORS=y
> CONFIG_HAVE_IRQ_WORK=y
> CONFIG_IRQ_WORK=y
>
> CONFIG_EXPERIMENTAL=y
> CONFIG_INIT_ENV_ARG_LIMIT=32
> CONFIG_CROSS_COMPILE=""
> CONFIG_LOCALVERSION=""
> CONFIG_LOCALVERSION_AUTO=y
> CONFIG_HAVE_KERNEL_GZIP=y
> CONFIG_HAVE_KERNEL_BZIP2=y
> CONFIG_HAVE_KERNEL_LZMA=y
> CONFIG_HAVE_KERNEL_XZ=y
> CONFIG_HAVE_KERNEL_LZO=y
> CONFIG_KERNEL_GZIP=y
> CONFIG_SWAP=y
> CONFIG_SYSVIPC=y
> CONFIG_SYSVIPC_SYSCTL=y
> CONFIG_BSD_PROCESS_ACCT=y
> CONFIG_TASKSTATS=y
> CONFIG_TASK_DELAY_ACCT=y
> CONFIG_TASK_XACCT=y
> CONFIG_TASK_IO_ACCOUNTING=y
> CONFIG_HAVE_GENERIC_HARDIRQS=y
>
> CONFIG_GENERIC_HARDIRQS=y
> CONFIG_HAVE_SPARSE_IRQ=y
> CONFIG_GENERIC_IRQ_PROBE=y
> CONFIG_GENERIC_IRQ_SHOW=y
> CONFIG_GENERIC_PENDING_IRQ=y
> CONFIG_IRQ_FORCED_THREADING=y
>
> CONFIG_TREE_RCU=y
> CONFIG_RCU_FANOUT=64
> CONFIG_IKCONFIG=y
> CONFIG_IKCONFIG_PROC=y
> CONFIG_LOG_BUF_SHIFT=20
> CONFIG_HAVE_UNSTABLE_SCHED_CLOCK=y
> CONFIG_NAMESPACES=y
> CONFIG_BLK_DEV_INITRD=y
> CONFIG_INITRAMFS_SOURCE=""
> CONFIG_RD_GZIP=y
> CONFIG_RD_BZIP2=y
> CONFIG_RD_LZMA=y
> CONFIG_RD_XZ=y
> CONFIG_RD_LZO=y
> CONFIG_SYSCTL=y
> CONFIG_ANON_INODES=y
> CONFIG_UID16=y
> CONFIG_SYSCTL_SYSCALL=y
> CONFIG_KALLSYMS=y
> CONFIG_KALLSYMS_ALL=y
> CONFIG_KALLSYMS_EXTRA_PASS=y
> CONFIG_HOTPLUG=y
> CONFIG_PRINTK=y
> CONFIG_BUG=y
> CONFIG_ELF_CORE=y
> CONFIG_PCSPKR_PLATFORM=y
> CONFIG_BASE_FULL=y
> CONFIG_FUTEX=y
> CONFIG_EPOLL=y
> CONFIG_SIGNALFD=y
> CONFIG_TIMERFD=y
> CONFIG_EVENTFD=y
> CONFIG_SHMEM=y
> CONFIG_AIO=y
> CONFIG_HAVE_PERF_EVENTS=y
>
> CONFIG_PERF_EVENTS=y
> CONFIG_VM_EVENT_COUNTERS=y
> CONFIG_PCI_QUIRKS=y
> CONFIG_SLUB_DEBUG=y
> CONFIG_SLUB=y
> CONFIG_HAVE_OPROFILE=y
> CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS=y
> CONFIG_USER_RETURN_NOTIFIER=y
> CONFIG_HAVE_IOREMAP_PROT=y
> CONFIG_HAVE_KPROBES=y
> CONFIG_HAVE_KRETPROBES=y
> CONFIG_HAVE_OPTPROBES=y
> CONFIG_HAVE_ARCH_TRACEHOOK=y
> CONFIG_HAVE_DMA_ATTRS=y
> CONFIG_USE_GENERIC_SMP_HELPERS=y
> CONFIG_HAVE_REGS_AND_STACK_ACCESS_API=y
> CONFIG_HAVE_DMA_API_DEBUG=y
> CONFIG_HAVE_HW_BREAKPOINT=y
> CONFIG_HAVE_MIXED_BREAKPOINTS_REGS=y
> CONFIG_HAVE_USER_RETURN_NOTIFIER=y
> CONFIG_HAVE_PERF_EVENTS_NMI=y
> CONFIG_HAVE_ARCH_JUMP_LABEL=y
>
> CONFIG_SLABINFO=y
> CONFIG_RT_MUTEXES=y
> CONFIG_BASE_SMALL=0
> CONFIG_MODULES=y
> CONFIG_MODULE_UNLOAD=y
> CONFIG_STOP_MACHINE=y
> CONFIG_BLOCK=y
> CONFIG_BLK_DEV_BSG=y
> CONFIG_BLOCK_COMPAT=y
>
> CONFIG_IOSCHED_NOOP=y
> CONFIG_IOSCHED_DEADLINE=y
> CONFIG_IOSCHED_CFQ=y
> CONFIG_DEFAULT_CFQ=y
> CONFIG_DEFAULT_IOSCHED="cfq"
> CONFIG_PREEMPT_NOTIFIERS=y
> CONFIG_INLINE_SPIN_UNLOCK=y
> CONFIG_INLINE_SPIN_UNLOCK_IRQ=y
> CONFIG_INLINE_READ_UNLOCK=y
> CONFIG_INLINE_READ_UNLOCK_IRQ=y
> CONFIG_INLINE_WRITE_UNLOCK=y
> CONFIG_INLINE_WRITE_UNLOCK_IRQ=y
> CONFIG_MUTEX_SPIN_ON_OWNER=y
>
> CONFIG_TICK_ONESHOT=y
> CONFIG_NO_HZ=y
> CONFIG_HIGH_RES_TIMERS=y
> CONFIG_GENERIC_CLOCKEVENTS_BUILD=y
> CONFIG_SMP=y
> CONFIG_X86_SUPPORTS_MEMORY_FAILURE=y
> CONFIG_SCHED_OMIT_FRAME_POINTER=y
> CONFIG_NO_BOOTMEM=y
> CONFIG_MK8=y
> CONFIG_X86_INTERNODE_CACHE_SHIFT=6
> CONFIG_X86_CMPXCHG=y
> CONFIG_CMPXCHG_LOCAL=y
> CONFIG_X86_L1_CACHE_SHIFT=6
> CONFIG_X86_XADD=y
> CONFIG_X86_WP_WORKS_OK=y
> CONFIG_X86_INTEL_USERCOPY=y
> CONFIG_X86_USE_PPRO_CHECKSUM=y
> CONFIG_X86_TSC=y
> CONFIG_X86_CMPXCHG64=y
> CONFIG_X86_CMOV=y
> CONFIG_X86_MINIMUM_CPU_FAMILY=64
> CONFIG_X86_DEBUGCTLMSR=y
> CONFIG_CPU_SUP_INTEL=y
> CONFIG_CPU_SUP_AMD=y
> CONFIG_CPU_SUP_CENTAUR=y
> CONFIG_HPET_TIMER=y
> CONFIG_HPET_EMULATE_RTC=y
> CONFIG_DMI=y
> CONFIG_GART_IOMMU=y
> CONFIG_AMD_IOMMU=y
> CONFIG_SWIOTLB=y
> CONFIG_IOMMU_HELPER=y
> CONFIG_IOMMU_API=y
> CONFIG_NR_CPUS=8
> CONFIG_SCHED_MC=y
> CONFIG_PREEMPT_NONE=y
> CONFIG_X86_LOCAL_APIC=y
> CONFIG_X86_IO_APIC=y
> CONFIG_X86_MCE=y
> CONFIG_X86_MCE_AMD=y
> CONFIG_X86_MCE_THRESHOLD=y
> CONFIG_ARCH_PHYS_ADDR_T_64BIT=y
> CONFIG_ARCH_DMA_ADDR_T_64BIT=y
> CONFIG_DIRECT_GBPAGES=y
> CONFIG_ARCH_SPARSEMEM_DEFAULT=y
> CONFIG_ARCH_SPARSEMEM_ENABLE=y
> CONFIG_ARCH_SELECT_MEMORY_MODEL=y
> CONFIG_ARCH_MEMORY_PROBE=y
> CONFIG_ILLEGAL_POINTER_VALUE=0xdead000000000000
> CONFIG_SELECT_MEMORY_MODEL=y
> CONFIG_SPARSEMEM_MANUAL=y
> CONFIG_SPARSEMEM=y
> CONFIG_HAVE_MEMORY_PRESENT=y
> CONFIG_SPARSEMEM_EXTREME=y
> CONFIG_SPARSEMEM_VMEMMAP_ENABLE=y
> CONFIG_SPARSEMEM_ALLOC_MEM_MAP_TOGETHER=y
> CONFIG_SPARSEMEM_VMEMMAP=y
> CONFIG_HAVE_MEMBLOCK=y
> CONFIG_MEMORY_HOTPLUG=y
> CONFIG_MEMORY_HOTPLUG_SPARSE=y
> CONFIG_PAGEFLAGS_EXTENDED=y
> CONFIG_SPLIT_PTLOCK_CPUS=4
> CONFIG_COMPACTION=y
> CONFIG_MIGRATION=y
> CONFIG_PHYS_ADDR_T_64BIT=y
> CONFIG_ZONE_DMA_FLAG=1
> CONFIG_BOUNCE=y
> CONFIG_VIRT_TO_BUS=y
> CONFIG_MMU_NOTIFIER=y
> CONFIG_KSM=y
> CONFIG_DEFAULT_MMAP_MIN_ADDR=4096
> CONFIG_ARCH_SUPPORTS_MEMORY_FAILURE=y
> CONFIG_TRANSPARENT_HUGEPAGE=y
> CONFIG_TRANSPARENT_HUGEPAGE_ALWAYS=y
> CONFIG_X86_RESERVE_LOW=64
> CONFIG_MTRR=y
> CONFIG_MTRR_SANITIZER=y
> CONFIG_MTRR_SANITIZER_ENABLE_DEFAULT=0
> CONFIG_MTRR_SANITIZER_SPARE_REG_NR_DEFAULT=1
> CONFIG_X86_PAT=y
> CONFIG_ARCH_USES_PG_UNCACHED=y
> CONFIG_SECCOMP=y
> CONFIG_HZ_100=y
> CONFIG_HZ=100
> CONFIG_SCHED_HRTICK=y
> CONFIG_PHYSICAL_START=0x1000000
> CONFIG_PHYSICAL_ALIGN=0x1000000
> CONFIG_ARCH_ENABLE_MEMORY_HOTPLUG=y
> CONFIG_ARCH_ENABLE_MEMORY_HOTREMOVE=y
>
> CONFIG_ACPI=y
> CONFIG_ACPI_BUTTON=y
> CONFIG_ACPI_DOCK=y
> CONFIG_ACPI_PROCESSOR=y
> CONFIG_ACPI_THERMAL=y
> CONFIG_ACPI_BLACKLIST_YEAR=0
> CONFIG_X86_PM_TIMER=y
>
> CONFIG_CPU_FREQ=y
> CONFIG_CPU_FREQ_TABLE=y
> CONFIG_CPU_FREQ_STAT=y
> CONFIG_CPU_FREQ_STAT_DETAILS=y
> CONFIG_CPU_FREQ_DEFAULT_GOV_ONDEMAND=y
> CONFIG_CPU_FREQ_GOV_PERFORMANCE=y
> CONFIG_CPU_FREQ_GOV_POWERSAVE=y
> CONFIG_CPU_FREQ_GOV_USERSPACE=y
> CONFIG_CPU_FREQ_GOV_ONDEMAND=y
> CONFIG_CPU_FREQ_GOV_CONSERVATIVE=y
>
> CONFIG_X86_ACPI_CPUFREQ=m
> CONFIG_X86_POWERNOW_K8=m
>
> CONFIG_CPU_IDLE=y
> CONFIG_CPU_IDLE_GOV_LADDER=y
> CONFIG_CPU_IDLE_GOV_MENU=y
>
>
> CONFIG_PCI=y
> CONFIG_PCI_DIRECT=y
> CONFIG_PCI_MMCONFIG=y
> CONFIG_PCI_DOMAINS=y
> CONFIG_PCIEPORTBUS=y
> CONFIG_PCIEAER=y
> CONFIG_PCIEASPM=y
> CONFIG_ARCH_SUPPORTS_MSI=y
> CONFIG_PCI_MSI=y
> CONFIG_HT_IRQ=y
> CONFIG_PCI_IOAPIC=y
> CONFIG_PCI_LABEL=y
> CONFIG_ISA_DMA_API=y
> CONFIG_AMD_NB=y
>
> CONFIG_BINFMT_ELF=y
> CONFIG_COMPAT_BINFMT_ELF=y
> CONFIG_BINFMT_MISC=m
> CONFIG_IA32_EMULATION=y
> CONFIG_IA32_AOUT=m
> CONFIG_COMPAT=y
> CONFIG_COMPAT_FOR_U64_ALIGNMENT=y
> CONFIG_SYSVIPC_COMPAT=y
> CONFIG_HAVE_TEXT_POKE_SMP=y
> CONFIG_NET=y
>
> CONFIG_PACKET=y
> CONFIG_UNIX=y
> CONFIG_XFRM=y
> CONFIG_XFRM_MIGRATE=y
> CONFIG_XFRM_IPCOMP=m
> CONFIG_NET_KEY=m
> CONFIG_NET_KEY_MIGRATE=y
> CONFIG_INET=y
> CONFIG_IP_MULTICAST=y
> CONFIG_IP_ROUTE_CLASSID=y
> CONFIG_NET_IPGRE_DEMUX=m
> CONFIG_INET_AH=m
> CONFIG_INET_ESP=m
> CONFIG_INET_IPCOMP=m
> CONFIG_INET_XFRM_TUNNEL=m
> CONFIG_INET_TUNNEL=m
> CONFIG_INET_XFRM_MODE_TRANSPORT=m
> CONFIG_INET_XFRM_MODE_TUNNEL=m
> CONFIG_INET_XFRM_MODE_BEET=m
> CONFIG_INET_DIAG=y
> CONFIG_INET_TCP_DIAG=y
> CONFIG_TCP_CONG_CUBIC=y
> CONFIG_DEFAULT_TCP_CONG="cubic"
> CONFIG_NETFILTER=y
> CONFIG_NETFILTER_ADVANCED=y
> CONFIG_BRIDGE_NETFILTER=y
>
> CONFIG_NETFILTER_NETLINK=m
> CONFIG_NETFILTER_NETLINK_QUEUE=m
> CONFIG_NETFILTER_NETLINK_LOG=m
> CONFIG_NF_CONNTRACK=m
> CONFIG_NF_CONNTRACK_MARK=y
> CONFIG_NF_CONNTRACK_ZONES=y
> CONFIG_NF_CT_PROTO_DCCP=m
> CONFIG_NF_CT_PROTO_GRE=m
> CONFIG_NF_CT_PROTO_SCTP=m
> CONFIG_NF_CT_PROTO_UDPLITE=m
> CONFIG_NF_CONNTRACK_AMANDA=m
> CONFIG_NF_CONNTRACK_FTP=m
> CONFIG_NF_CONNTRACK_H323=m
> CONFIG_NF_CONNTRACK_IRC=m
> CONFIG_NF_CONNTRACK_BROADCAST=m
> CONFIG_NF_CONNTRACK_NETBIOS_NS=m
> CONFIG_NF_CONNTRACK_SNMP=m
> CONFIG_NF_CONNTRACK_PPTP=m
> CONFIG_NF_CONNTRACK_SANE=m
> CONFIG_NF_CONNTRACK_SIP=m
> CONFIG_NF_CONNTRACK_TFTP=m
> CONFIG_NF_CT_NETLINK=m
> CONFIG_NETFILTER_TPROXY=m
> CONFIG_NETFILTER_XTABLES=m
>
> CONFIG_NETFILTER_XT_MARK=m
> CONFIG_NETFILTER_XT_CONNMARK=m
>
> CONFIG_NETFILTER_XT_TARGET_CHECKSUM=m
> CONFIG_NETFILTER_XT_TARGET_CLASSIFY=m
> CONFIG_NETFILTER_XT_TARGET_CONNMARK=m
> CONFIG_NETFILTER_XT_TARGET_CT=m
> CONFIG_NETFILTER_XT_TARGET_DSCP=m
> CONFIG_NETFILTER_XT_TARGET_HL=m
> CONFIG_NETFILTER_XT_TARGET_IDLETIMER=m
> CONFIG_NETFILTER_XT_TARGET_MARK=m
> CONFIG_NETFILTER_XT_TARGET_NFLOG=m
> CONFIG_NETFILTER_XT_TARGET_NFQUEUE=m
> CONFIG_NETFILTER_XT_TARGET_NOTRACK=m
> CONFIG_NETFILTER_XT_TARGET_RATEEST=m
> CONFIG_NETFILTER_XT_TARGET_TEE=m
> CONFIG_NETFILTER_XT_TARGET_TPROXY=m
> CONFIG_NETFILTER_XT_TARGET_TRACE=m
> CONFIG_NETFILTER_XT_TARGET_TCPMSS=m
> CONFIG_NETFILTER_XT_TARGET_TCPOPTSTRIP=m
>
> CONFIG_NETFILTER_XT_MATCH_ADDRTYPE=m
> CONFIG_NETFILTER_XT_MATCH_CLUSTER=m
> CONFIG_NETFILTER_XT_MATCH_COMMENT=m
> CONFIG_NETFILTER_XT_MATCH_CONNBYTES=m
> CONFIG_NETFILTER_XT_MATCH_CONNLIMIT=m
> CONFIG_NETFILTER_XT_MATCH_CONNMARK=m
> CONFIG_NETFILTER_XT_MATCH_CONNTRACK=m
> CONFIG_NETFILTER_XT_MATCH_CPU=m
> CONFIG_NETFILTER_XT_MATCH_DCCP=m
> CONFIG_NETFILTER_XT_MATCH_DEVGROUP=m
> CONFIG_NETFILTER_XT_MATCH_DSCP=m
> CONFIG_NETFILTER_XT_MATCH_ESP=m
> CONFIG_NETFILTER_XT_MATCH_HASHLIMIT=m
> CONFIG_NETFILTER_XT_MATCH_HELPER=m
> CONFIG_NETFILTER_XT_MATCH_HL=m
> CONFIG_NETFILTER_XT_MATCH_IPRANGE=m
> CONFIG_NETFILTER_XT_MATCH_LENGTH=m
> CONFIG_NETFILTER_XT_MATCH_LIMIT=m
> CONFIG_NETFILTER_XT_MATCH_MAC=m
> CONFIG_NETFILTER_XT_MATCH_MARK=m
> CONFIG_NETFILTER_XT_MATCH_MULTIPORT=m
> CONFIG_NETFILTER_XT_MATCH_OSF=m
> CONFIG_NETFILTER_XT_MATCH_OWNER=m
> CONFIG_NETFILTER_XT_MATCH_POLICY=m
> CONFIG_NETFILTER_XT_MATCH_PHYSDEV=m
> CONFIG_NETFILTER_XT_MATCH_PKTTYPE=m
> CONFIG_NETFILTER_XT_MATCH_QUOTA=m
> CONFIG_NETFILTER_XT_MATCH_RATEEST=m
> CONFIG_NETFILTER_XT_MATCH_REALM=m
> CONFIG_NETFILTER_XT_MATCH_RECENT=m
> CONFIG_NETFILTER_XT_MATCH_SCTP=m
> CONFIG_NETFILTER_XT_MATCH_SOCKET=m
> CONFIG_NETFILTER_XT_MATCH_STATE=m
> CONFIG_NETFILTER_XT_MATCH_STATISTIC=m
> CONFIG_NETFILTER_XT_MATCH_STRING=m
> CONFIG_NETFILTER_XT_MATCH_TCPMSS=m
> CONFIG_NETFILTER_XT_MATCH_TIME=m
> CONFIG_NETFILTER_XT_MATCH_U32=m
>
> CONFIG_NF_DEFRAG_IPV4=m
> CONFIG_NF_CONNTRACK_IPV4=m
> CONFIG_NF_CONNTRACK_PROC_COMPAT=y
> CONFIG_IP_NF_QUEUE=m
> CONFIG_IP_NF_IPTABLES=m
> CONFIG_IP_NF_MATCH_AH=m
> CONFIG_IP_NF_MATCH_ECN=m
> CONFIG_IP_NF_MATCH_TTL=m
> CONFIG_IP_NF_FILTER=m
> CONFIG_IP_NF_TARGET_REJECT=m
> CONFIG_IP_NF_TARGET_LOG=m
> CONFIG_IP_NF_TARGET_ULOG=m
> CONFIG_NF_NAT=m
> CONFIG_NF_NAT_NEEDED=y
> CONFIG_IP_NF_TARGET_MASQUERADE=m
> CONFIG_IP_NF_TARGET_NETMAP=m
> CONFIG_IP_NF_TARGET_REDIRECT=m
> CONFIG_NF_NAT_SNMP_BASIC=m
> CONFIG_NF_NAT_PROTO_DCCP=m
> CONFIG_NF_NAT_PROTO_GRE=m
> CONFIG_NF_NAT_PROTO_UDPLITE=m
> CONFIG_NF_NAT_PROTO_SCTP=m
> CONFIG_NF_NAT_FTP=m
> CONFIG_NF_NAT_IRC=m
> CONFIG_NF_NAT_TFTP=m
> CONFIG_NF_NAT_AMANDA=m
> CONFIG_NF_NAT_PPTP=m
> CONFIG_NF_NAT_H323=m
> CONFIG_NF_NAT_SIP=m
> CONFIG_IP_NF_MANGLE=m
> CONFIG_IP_NF_TARGET_CLUSTERIP=m
> CONFIG_IP_NF_TARGET_ECN=m
> CONFIG_IP_NF_TARGET_TTL=m
> CONFIG_IP_NF_RAW=m
> CONFIG_IP_NF_ARPTABLES=m
> CONFIG_IP_NF_ARPFILTER=m
> CONFIG_IP_NF_ARP_MANGLE=m
> CONFIG_IP_SCTP=m
> CONFIG_SCTP_HMAC_MD5=y
> CONFIG_STP=y
> CONFIG_BRIDGE=y
> CONFIG_BRIDGE_IGMP_SNOOPING=y
> CONFIG_VLAN_8021Q=y
> CONFIG_LLC=y
> CONFIG_NET_SCHED=y
>
> CONFIG_NET_SCH_CBQ=m
> CONFIG_NET_SCH_HTB=m
> CONFIG_NET_SCH_HFSC=m
> CONFIG_NET_SCH_PRIO=m
> CONFIG_NET_SCH_RED=m
> CONFIG_NET_SCH_SFB=m
> CONFIG_NET_SCH_SFQ=m
> CONFIG_NET_SCH_TEQL=m
> CONFIG_NET_SCH_TBF=m
> CONFIG_NET_SCH_GRED=m
> CONFIG_NET_SCH_DSMARK=m
> CONFIG_NET_SCH_NETEM=m
> CONFIG_NET_SCH_MQPRIO=m
> CONFIG_NET_SCH_CHOKE=m
> CONFIG_NET_SCH_INGRESS=m
>
> CONFIG_NET_CLS=y
> CONFIG_NET_CLS_BASIC=m
> CONFIG_NET_CLS_TCINDEX=m
> CONFIG_NET_CLS_ROUTE4=m
> CONFIG_NET_CLS_FW=m
> CONFIG_NET_CLS_U32=m
> CONFIG_NET_CLS_FLOW=m
> CONFIG_NET_EMATCH=y
> CONFIG_NET_EMATCH_STACK=32
> CONFIG_NET_EMATCH_CMP=m
> CONFIG_NET_EMATCH_NBYTE=m
> CONFIG_NET_EMATCH_U32=m
> CONFIG_NET_EMATCH_META=m
> CONFIG_NET_EMATCH_TEXT=m
> CONFIG_NET_CLS_ACT=y
> CONFIG_NET_ACT_POLICE=m
> CONFIG_NET_ACT_GACT=m
> CONFIG_GACT_PROB=y
> CONFIG_NET_ACT_MIRRED=m
> CONFIG_NET_ACT_IPT=m
> CONFIG_NET_ACT_NAT=m
> CONFIG_NET_ACT_PEDIT=m
> CONFIG_NET_ACT_SIMP=m
> CONFIG_NET_ACT_SKBEDIT=m
> CONFIG_NET_ACT_CSUM=m
> CONFIG_NET_SCH_FIFO=y
> CONFIG_RPS=y
> CONFIG_RFS_ACCEL=y
> CONFIG_XPS=y
>
>
>
> CONFIG_UEVENT_HELPER_PATH="/sbin/hotplug"
> CONFIG_DEVTMPFS=y
> CONFIG_STANDALONE=y
> CONFIG_PREVENT_FIRMWARE_BUILD=y
> CONFIG_FW_LOADER=y
> CONFIG_FIRMWARE_IN_KERNEL=y
> CONFIG_EXTRA_FIRMWARE=""
> CONFIG_ARCH_NO_SYSDEV_OPS=y
> CONFIG_PNP=y
> CONFIG_PNP_DEBUG_MESSAGES=y
>
> CONFIG_PNPACPI=y
> CONFIG_BLK_DEV=y
> CONFIG_BLK_DEV_LOOP=y
>
> CONFIG_HAVE_IDE=y
>
> CONFIG_SCSI_MOD=y
> CONFIG_RAID_ATTRS=m
> CONFIG_SCSI=y
> CONFIG_SCSI_DMA=y
> CONFIG_SCSI_PROC_FS=y
>
> CONFIG_BLK_DEV_SD=y
> CONFIG_BLK_DEV_SR=y
> CONFIG_CHR_DEV_SG=y
> CONFIG_SCSI_WAIT_SCAN=m
>
> CONFIG_SCSI_SAS_ATTRS=m
> CONFIG_SCSI_LOWLEVEL=y
> CONFIG_MEGARAID_SAS=m
> CONFIG_ATA=y
> CONFIG_ATA_VERBOSE_ERROR=y
> CONFIG_ATA_ACPI=y
> CONFIG_SATA_PMP=y
>
> CONFIG_SATA_AHCI=m
> CONFIG_SATA_SIL24=m
> CONFIG_ATA_SFF=y
>
> CONFIG_ATA_BMDMA=y
>
> CONFIG_SATA_MV=m
> CONFIG_SATA_NV=m
> CONFIG_SATA_PROMISE=m
> CONFIG_SATA_SIL=m
>
>
>
> CONFIG_MD=y
> CONFIG_BLK_DEV_MD=y
> CONFIG_MD_LINEAR=y
> CONFIG_MD_RAID0=y
> CONFIG_MD_RAID1=y
> CONFIG_MD_RAID10=y
> CONFIG_MD_RAID456=y
> CONFIG_BLK_DEV_DM=y
>
> CONFIG_NETDEVICES=y
> CONFIG_DUMMY=m
> CONFIG_BONDING=m
> CONFIG_TUN=y
> CONFIG_MII=m
> CONFIG_NET_ETHERNET=y
> CONFIG_NET_PCI=y
> CONFIG_FORCEDETH=m
> CONFIG_NETDEV_1000=y
> CONFIG_R8169=m
>
>
>
> CONFIG_PPP=m
> CONFIG_PPP_MULTILINK=y
> CONFIG_PPP_FILTER=y
> CONFIG_PPP_ASYNC=m
> CONFIG_PPP_SYNC_TTY=m
> CONFIG_PPP_DEFLATE=m
> CONFIG_PPP_BSDCOMP=m
> CONFIG_PPP_MPPE=m
> CONFIG_PPPOE=m
> CONFIG_SLHC=m
> CONFIG_NETCONSOLE=m
> CONFIG_NETCONSOLE_DYNAMIC=y
> CONFIG_NETPOLL=y
> CONFIG_NET_POLL_CONTROLLER=y
>
> CONFIG_INPUT=y
>
> CONFIG_INPUT_MOUSEDEV=y
> CONFIG_INPUT_MOUSEDEV_SCREEN_X=1024
> CONFIG_INPUT_MOUSEDEV_SCREEN_Y=768
>
> CONFIG_INPUT_KEYBOARD=y
> CONFIG_KEYBOARD_ATKBD=y
>
> CONFIG_SERIO=y
> CONFIG_SERIO_I8042=y
> CONFIG_SERIO_LIBPS2=y
>
> CONFIG_VT=y
> CONFIG_CONSOLE_TRANSLATIONS=y
> CONFIG_VT_CONSOLE=y
> CONFIG_HW_CONSOLE=y
> CONFIG_UNIX98_PTYS=y
> CONFIG_LEGACY_PTYS=y
> CONFIG_LEGACY_PTY_COUNT=256
>
> CONFIG_SERIAL_8250=y
> CONFIG_SERIAL_8250_CONSOLE=y
> CONFIG_FIX_EARLYCON_MEM=y
> CONFIG_SERIAL_8250_PCI=y
> CONFIG_SERIAL_8250_PNP=y
> CONFIG_SERIAL_8250_NR_UARTS=4
> CONFIG_SERIAL_8250_RUNTIME_UARTS=4
>
> CONFIG_SERIAL_CORE=y
> CONFIG_SERIAL_CORE_CONSOLE=y
> CONFIG_RTC=y
> CONFIG_HPET=y
> CONFIG_HPET_MMAP=y
> CONFIG_DEVPORT=y
> CONFIG_I2C=y
> CONFIG_I2C_BOARDINFO=y
> CONFIG_I2C_COMPAT=y
> CONFIG_I2C_CHARDEV=y
> CONFIG_I2C_HELPER_AUTO=y
> CONFIG_I2C_ALGOBIT=y
>
>
> CONFIG_I2C_AMD756=m
> CONFIG_I2C_PIIX4=m
> CONFIG_I2C_NFORCE2=m
> CONFIG_I2C_VIA=m
> CONFIG_I2C_VIAPRO=m
>
> CONFIG_I2C_SCMI=m
>
>
>
>
>
> CONFIG_ARCH_WANT_OPTIONAL_GPIOLIB=y
> CONFIG_HWMON=y
> CONFIG_HWMON_VID=m
>
> CONFIG_SENSORS_K8TEMP=m
> CONFIG_SENSORS_K10TEMP=m
> CONFIG_SENSORS_W83781D=m
> CONFIG_SENSORS_W83791D=m
> CONFIG_SENSORS_W83792D=m
> CONFIG_SENSORS_W83793=m
> CONFIG_SENSORS_W83795=m
> CONFIG_SENSORS_W83795_FANCTRL=y
> CONFIG_SENSORS_W83L785TS=m
> CONFIG_SENSORS_W83L786NG=m
> CONFIG_SENSORS_W83627HF=m
> CONFIG_SENSORS_W83627EHF=m
>
> CONFIG_SENSORS_ATK0110=m
> CONFIG_THERMAL=y
> CONFIG_THERMAL_HWMON=y
> CONFIG_SSB_POSSIBLE=y
>
>
> CONFIG_VGA_ARB=y
> CONFIG_VGA_ARB_MAX_GPUS=16
> CONFIG_VGASTATE=y
> CONFIG_FB=y
> CONFIG_FB_DDC=y
> CONFIG_FB_CFB_FILLRECT=y
> CONFIG_FB_CFB_COPYAREA=y
> CONFIG_FB_CFB_IMAGEBLIT=y
> CONFIG_FB_MODE_HELPERS=y
>
> CONFIG_FB_NVIDIA=y
> CONFIG_FB_NVIDIA_I2C=y
> CONFIG_FB_RADEON=y
> CONFIG_FB_RADEON_I2C=y
>
>
> CONFIG_VGA_CONSOLE=y
> CONFIG_DUMMY_CONSOLE=y
> CONFIG_FRAMEBUFFER_CONSOLE=y
> CONFIG_FONT_8x8=y
> CONFIG_FONT_8x16=y
> CONFIG_LOGO=y
> CONFIG_LOGO_LINUX_MONO=y
> CONFIG_LOGO_LINUX_VGA16=y
> CONFIG_LOGO_LINUX_CLUT224=y
> CONFIG_HID_SUPPORT=y
> CONFIG_HID=y
> CONFIG_HIDRAW=y
>
> CONFIG_USB_HID=m
> CONFIG_USB_HIDDEV=y
>
> CONFIG_HID_A4TECH=m
> CONFIG_HID_APPLE=m
> CONFIG_HID_BELKIN=m
> CONFIG_HID_CHERRY=m
> CONFIG_HID_CHICONY=m
> CONFIG_HID_CYPRESS=m
> CONFIG_HID_EZKEY=m
> CONFIG_HID_KYE=m
> CONFIG_HID_KENSINGTON=m
> CONFIG_HID_LOGITECH=m
> CONFIG_HID_MICROSOFT=m
> CONFIG_HID_MONTEREY=m
> CONFIG_USB_SUPPORT=y
> CONFIG_USB_ARCH_HAS_HCD=y
> CONFIG_USB_ARCH_HAS_OHCI=y
> CONFIG_USB_ARCH_HAS_EHCI=y
> CONFIG_USB=m
>
>
> CONFIG_USB_XHCI_HCD=m
> CONFIG_USB_EHCI_HCD=m
> CONFIG_USB_OHCI_HCD=m
> CONFIG_USB_OHCI_LITTLE_ENDIAN=y
> CONFIG_USB_UHCI_HCD=m
>
>
> CONFIG_USB_ACM=m
>
>
> CONFIG_USB_STORAGE=m
> CONFIG_USB_LIBUSUAL=y
>
>
> CONFIG_USB_SERIAL=m
> CONFIG_USB_SERIAL_GENERIC=y
> CONFIG_USB_SERIAL_FTDI_SIO=m
> CONFIG_USB_SERIAL_IPAQ=m
> CONFIG_USB_SERIAL_PL2303=m
>
>
>
> CONFIG_FIRMWARE_MEMMAP=y
> CONFIG_DMIID=y
>
> CONFIG_EXT2_FS=y
> CONFIG_EXT2_FS_XATTR=y
> CONFIG_EXT3_FS=y
> CONFIG_EXT3_FS_XATTR=y
> CONFIG_EXT4_FS=y
> CONFIG_EXT4_FS_XATTR=y
> CONFIG_JBD=y
> CONFIG_JBD2=y
> CONFIG_FS_MBCACHE=y
> CONFIG_EXPORTFS=y
> CONFIG_FILE_LOCKING=y
> CONFIG_FSNOTIFY=y
> CONFIG_DNOTIFY=y
> CONFIG_INOTIFY_USER=y
> CONFIG_FANOTIFY=y
> CONFIG_FUSE_FS=m
> CONFIG_CUSE=m
>
> CONFIG_FSCACHE=m
> CONFIG_FSCACHE_DEBUG=y
> CONFIG_CACHEFILES=m
>
> CONFIG_ISO9660_FS=m
> CONFIG_JOLIET=y
> CONFIG_ZISOFS=y
> CONFIG_UDF_FS=m
> CONFIG_UDF_NLS=y
>
> CONFIG_FAT_FS=m
> CONFIG_MSDOS_FS=m
> CONFIG_VFAT_FS=m
> CONFIG_FAT_DEFAULT_CODEPAGE=437
> CONFIG_FAT_DEFAULT_IOCHARSET="iso8859-1"
> CONFIG_NTFS_FS=m
>
> CONFIG_PROC_FS=y
> CONFIG_PROC_SYSCTL=y
> CONFIG_PROC_PAGE_MONITOR=y
> CONFIG_SYSFS=y
> CONFIG_TMPFS=y
> CONFIG_HUGETLBFS=y
> CONFIG_HUGETLB_PAGE=y
> CONFIG_CONFIGFS_FS=m
> CONFIG_NETWORK_FILESYSTEMS=y
> CONFIG_NFS_FS=y
> CONFIG_NFS_V3=y
> CONFIG_NFSD=y
> CONFIG_NFSD_DEPRECATED=y
> CONFIG_NFSD_V3=y
> CONFIG_LOCKD=y
> CONFIG_LOCKD_V4=y
> CONFIG_NFS_COMMON=y
> CONFIG_SUNRPC=y
>
> CONFIG_PARTITION_ADVANCED=y
> CONFIG_MSDOS_PARTITION=y
> CONFIG_EFI_PARTITION=y
> CONFIG_NLS=y
> CONFIG_NLS_DEFAULT="iso8859-1"
> CONFIG_NLS_CODEPAGE_437=y
> CONFIG_NLS_ISO8859_1=y
> CONFIG_DLM=m
>
> CONFIG_TRACE_IRQFLAGS_SUPPORT=y
> CONFIG_PRINTK_TIME=y
> CONFIG_DEFAULT_MESSAGE_LOGLEVEL=4
> CONFIG_ENABLE_WARN_DEPRECATED=y
> CONFIG_ENABLE_MUST_CHECK=y
> CONFIG_FRAME_WARN=2048
> CONFIG_MAGIC_SYSRQ=y
> CONFIG_DEBUG_FS=y
> CONFIG_DEBUG_KERNEL=y
> CONFIG_DETECT_HUNG_TASK=y
> CONFIG_BOOTPARAM_HUNG_TASK_PANIC_VALUE=0
> CONFIG_TIMER_STATS=y
> CONFIG_DEBUG_BUGVERBOSE=y
> CONFIG_DEBUG_MEMORY_INIT=y
> CONFIG_ARCH_WANT_FRAME_POINTERS=y
> CONFIG_SYSCTL_SYSCALL_CHECK=y
> CONFIG_USER_STACKTRACE_SUPPORT=y
> CONFIG_HAVE_FUNCTION_TRACER=y
> CONFIG_HAVE_FUNCTION_GRAPH_TRACER=y
> CONFIG_HAVE_FUNCTION_GRAPH_FP_TEST=y
> CONFIG_HAVE_FUNCTION_TRACE_MCOUNT_TEST=y
> CONFIG_HAVE_DYNAMIC_FTRACE=y
> CONFIG_HAVE_FTRACE_MCOUNT_RECORD=y
> CONFIG_HAVE_SYSCALL_TRACEPOINTS=y
> CONFIG_HAVE_C_RECORDMCOUNT=y
> CONFIG_TRACING_SUPPORT=y
> CONFIG_HAVE_ARCH_KGDB=y
> CONFIG_HAVE_ARCH_KMEMCHECK=y
> CONFIG_X86_VERBOSE_BOOTUP=y
> CONFIG_EARLY_PRINTK=y
> CONFIG_HAVE_MMIOTRACE_SUPPORT=y
> CONFIG_IO_DELAY_TYPE_0X80=0
> CONFIG_IO_DELAY_TYPE_0XED=1
> CONFIG_IO_DELAY_TYPE_UDELAY=2
> CONFIG_IO_DELAY_TYPE_NONE=3
> CONFIG_IO_DELAY_0X80=y
> CONFIG_DEFAULT_IO_DELAY_TYPE=0
>
> CONFIG_DEFAULT_SECURITY_DAC=y
> CONFIG_DEFAULT_SECURITY=""
> CONFIG_XOR_BLOCKS=y
> CONFIG_ASYNC_CORE=y
> CONFIG_ASYNC_MEMCPY=y
> CONFIG_ASYNC_XOR=y
> CONFIG_ASYNC_PQ=y
> CONFIG_ASYNC_RAID6_RECOV=y
> CONFIG_CRYPTO=y
>
> CONFIG_CRYPTO_ALGAPI=m
> CONFIG_CRYPTO_ALGAPI2=m
> CONFIG_CRYPTO_AEAD=m
> CONFIG_CRYPTO_AEAD2=m
> CONFIG_CRYPTO_BLKCIPHER=m
> CONFIG_CRYPTO_BLKCIPHER2=m
> CONFIG_CRYPTO_HASH=m
> CONFIG_CRYPTO_HASH2=m
> CONFIG_CRYPTO_RNG2=m
> CONFIG_CRYPTO_PCOMP2=m
> CONFIG_CRYPTO_MANAGER=m
> CONFIG_CRYPTO_MANAGER2=m
> CONFIG_CRYPTO_MANAGER_DISABLE_TESTS=y
> CONFIG_CRYPTO_WORKQUEUE=m
> CONFIG_CRYPTO_AUTHENC=m
>
>
> CONFIG_CRYPTO_CBC=m
> CONFIG_CRYPTO_ECB=m
>
> CONFIG_CRYPTO_HMAC=m
>
> CONFIG_CRYPTO_CRC32C=m
> CONFIG_CRYPTO_MD5=m
> CONFIG_CRYPTO_SHA1=m
>
> CONFIG_CRYPTO_ARC4=m
> CONFIG_CRYPTO_DES=m
>
> CONFIG_CRYPTO_DEFLATE=m
>
> CONFIG_HAVE_KVM=y
> CONFIG_HAVE_KVM_IRQCHIP=y
> CONFIG_HAVE_KVM_EVENTFD=y
> CONFIG_KVM_APIC_ARCHITECTURE=y
> CONFIG_KVM_MMIO=y
> CONFIG_KVM_ASYNC_PF=y
> CONFIG_VIRTUALIZATION=y
> CONFIG_KVM=m
> CONFIG_KVM_INTEL=m
> CONFIG_KVM_AMD=m
> CONFIG_VHOST_NET=m
>
> CONFIG_RAID6_PQ=y
> CONFIG_BITREVERSE=y
> CONFIG_GENERIC_FIND_FIRST_BIT=y
> CONFIG_GENERIC_FIND_NEXT_BIT=y
> CONFIG_GENERIC_FIND_LAST_BIT=y
> CONFIG_CRC_CCITT=m
> CONFIG_CRC16=y
> CONFIG_CRC_T10DIF=y
> CONFIG_CRC_ITU_T=y
> CONFIG_CRC32=y
> CONFIG_LIBCRC32C=m
> CONFIG_ZLIB_INFLATE=y
> CONFIG_ZLIB_DEFLATE=m
> CONFIG_LZO_DECOMPRESS=y
> CONFIG_XZ_DEC=y
> CONFIG_XZ_DEC_X86=y
> CONFIG_XZ_DEC_POWERPC=y
> CONFIG_XZ_DEC_IA64=y
> CONFIG_XZ_DEC_ARM=y
> CONFIG_XZ_DEC_ARMTHUMB=y
> CONFIG_XZ_DEC_SPARC=y
> CONFIG_XZ_DEC_BCJ=y
> CONFIG_DECOMPRESS_GZIP=y
> CONFIG_DECOMPRESS_BZIP2=y
> CONFIG_DECOMPRESS_LZMA=y
> CONFIG_DECOMPRESS_XZ=y
> CONFIG_DECOMPRESS_LZO=y
> CONFIG_TEXTSEARCH=y
> CONFIG_TEXTSEARCH_KMP=m
> CONFIG_TEXTSEARCH_BM=m
> CONFIG_TEXTSEARCH_FSM=m
> CONFIG_HAS_IOMEM=y
> CONFIG_HAS_IOPORT=y
> CONFIG_HAS_DMA=y
> CONFIG_CPU_RMAP=y
> CONFIG_NLATTR=y
>
> 00:00.0 Host bridge: Advanced Micro Devices [AMD] RS880 Host Bridge
> 00:01.0 PCI bridge: ASRock Incorporation Device 9602
> 00:02.0 PCI bridge: Advanced Micro Devices [AMD] RS780 PCI to PCI
> bridge (ext gfx port 0)
> 00:03.0 PCI bridge: Advanced Micro Devices [AMD] RS780 PCI to PCI
> bridge (ext gfx port 1)
> 00:04.0 PCI bridge: Advanced Micro Devices [AMD] RS780 PCI to PCI
> bridge (PCIE port 0)
> 00:09.0 PCI bridge: Advanced Micro Devices [AMD] RS780 PCI to PCI
> bridge (PCIE port 4)
> 00:0a.0 PCI bridge: Advanced Micro Devices [AMD] RS780 PCI to PCI
> bridge (PCIE port 5)
> 00:11.0 SATA controller: ATI Technologies Inc SB700/SB800 SATA
> Controller [AHCI mode] (rev 40)
> 00:12.0 USB Controller: ATI Technologies Inc SB700/SB800 USB OHCI0
> Controller
> 00:12.2 USB Controller: ATI Technologies Inc SB700/SB800 USB EHCI Controller
> 00:13.0 USB Controller: ATI Technologies Inc SB700/SB800 USB OHCI0
> Controller
> 00:13.2 USB Controller: ATI Technologies Inc SB700/SB800 USB EHCI Controller
> 00:14.0 SMBus: ATI Technologies Inc SBx00 SMBus Controller (rev 41)
> 00:14.2 Audio device: ATI Technologies Inc SBx00 Azalia (Intel HDA) (rev 40)
> 00:14.3 ISA bridge: ATI Technologies Inc SB700/SB800 LPC host
> controller (rev 40)
> 00:14.4 PCI bridge: ATI Technologies Inc SBx00 PCI to PCI Bridge (rev 40)
> 00:14.5 USB Controller: ATI Technologies Inc SB700/SB800 USB OHCI2
> Controller
> 00:16.0 USB Controller: ATI Technologies Inc SB700/SB800 USB OHCI0
> Controller
> 00:16.2 USB Controller: ATI Technologies Inc SB700/SB800 USB EHCI Controller
> 00:18.0 Host bridge: Advanced Micro Devices [AMD] Family 10h
> Processor HyperTransport Configuration
> 00:18.1 Host bridge: Advanced Micro Devices [AMD] Family 10h
> Processor Address Map
> 00:18.2 Host bridge: Advanced Micro Devices [AMD] Family 10h
> Processor DRAM Controller
> 00:18.3 Host bridge: Advanced Micro Devices [AMD] Family 10h
> Processor Miscellaneous Control
> 00:18.4 Host bridge: Advanced Micro Devices [AMD] Family 10h
> Processor Link Control
> 01:05.0 VGA compatible controller: ATI Technologies Inc RS880
> [Radeon HD 4250]
> 02:00.0 USB Controller: NEC Corporation uPD720200 USB 3.0 Host
> Controller (rev 03)
> 03:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd.
> RTL8111/8168B PCI Express Gigabit Ethernet controller (rev 06)
> 04:00.0 RAID bus controller: LSI Logic / Symbios Logic MegaRAID SAS
> 9240 (rev 02)
> 05:00.0 RAID bus controller: LSI Logic / Symbios Logic MegaRAID SAS
> 9240 (rev 02)
> 07:00.0 SCSI storage controller: Marvell Technology Group Ltd.
> 88SX7042 PCI-e 4-port SATA-II (rev 02)
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/
--
Regards/Gruss,
Boris.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 111+ messages in thread
* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
2011-05-31 5:47 ` Borislav Petkov
@ 2011-05-31 9:26 ` Brad Campbell
-1 siblings, 0 replies; 111+ messages in thread
From: Brad Campbell @ 2011-05-31 9:26 UTC (permalink / raw)
To: Borislav Petkov, linux-kernel, kvm, linux-mm
On 31/05/11 13:47, Borislav Petkov wrote:
> Looks like a KSM issue. Disabling CONFIG_KSM should at least stop your
> machine from oopsing.
>
> Adding linux-mm.
>
I initially thought that, so the second panic was produced with KSM
disabled from boot.
echo 0 > /sys/kernel/mm/ksm/run
If you still think that compiling ksm out of the kernel will prevent it
then I'm willing to give it a go.
It's a production server, so I can only really bounce it around after
about 9PM - GMT+8.
Regards,
Brad
^ permalink raw reply [flat|nested] 111+ messages in thread
* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
@ 2011-05-31 9:26 ` Brad Campbell
0 siblings, 0 replies; 111+ messages in thread
From: Brad Campbell @ 2011-05-31 9:26 UTC (permalink / raw)
To: Borislav Petkov, linux-kernel, kvm, linux-mm
On 31/05/11 13:47, Borislav Petkov wrote:
> Looks like a KSM issue. Disabling CONFIG_KSM should at least stop your
> machine from oopsing.
>
> Adding linux-mm.
>
I initially thought that, so the second panic was produced with KSM
disabled from boot.
echo 0 > /sys/kernel/mm/ksm/run
If you still think that compiling ksm out of the kernel will prevent it
then I'm willing to give it a go.
It's a production server, so I can only really bounce it around after
about 9PM - GMT+8.
Regards,
Brad
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 111+ messages in thread
* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
2011-05-31 9:26 ` Brad Campbell
@ 2011-05-31 10:38 ` Borislav Petkov
-1 siblings, 0 replies; 111+ messages in thread
From: Borislav Petkov @ 2011-05-31 10:38 UTC (permalink / raw)
To: Brad Campbell
Cc: linux-kernel, kvm, linux-mm, Hugh Dickins, Andrea Arcangeli, Izik Eidus
On Tue, May 31, 2011 at 05:26:10PM +0800, Brad Campbell wrote:
> On 31/05/11 13:47, Borislav Petkov wrote:
> >Looks like a KSM issue. Disabling CONFIG_KSM should at least stop your
> >machine from oopsing.
> >
> >Adding linux-mm.
> >
>
> I initially thought that, so the second panic was produced with KSM
> disabled from boot.
>
> echo 0 > /sys/kernel/mm/ksm/run
>
> If you still think that compiling ksm out of the kernel will prevent
> it then I'm willing to give it a go.
Ok, from looking at the code, when KSM inits, it starts the ksm kernel
thread and it looks like your oops comes from the function that is run
in the kernel thread - ksm_scan_thread.
So even if you disable it from sysfs, it runs at least once.
Let's add some more people to Cc and see what happens :).
--
Regards/Gruss,
Boris.
^ permalink raw reply [flat|nested] 111+ messages in thread
* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
@ 2011-05-31 10:38 ` Borislav Petkov
0 siblings, 0 replies; 111+ messages in thread
From: Borislav Petkov @ 2011-05-31 10:38 UTC (permalink / raw)
To: Brad Campbell
Cc: linux-kernel, kvm, linux-mm, Hugh Dickins, Andrea Arcangeli, Izik Eidus
On Tue, May 31, 2011 at 05:26:10PM +0800, Brad Campbell wrote:
> On 31/05/11 13:47, Borislav Petkov wrote:
> >Looks like a KSM issue. Disabling CONFIG_KSM should at least stop your
> >machine from oopsing.
> >
> >Adding linux-mm.
> >
>
> I initially thought that, so the second panic was produced with KSM
> disabled from boot.
>
> echo 0 > /sys/kernel/mm/ksm/run
>
> If you still think that compiling ksm out of the kernel will prevent
> it then I'm willing to give it a go.
Ok, from looking at the code, when KSM inits, it starts the ksm kernel
thread and it looks like your oops comes from the function that is run
in the kernel thread - ksm_scan_thread.
So even if you disable it from sysfs, it runs at least once.
Let's add some more people to Cc and see what happens :).
--
Regards/Gruss,
Boris.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 111+ messages in thread
* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
2011-05-31 10:38 ` Borislav Petkov
(?)
@ 2011-05-31 14:24 ` Brad Campbell
-1 siblings, 0 replies; 111+ messages in thread
From: Brad Campbell @ 2011-05-31 14:24 UTC (permalink / raw)
To: Borislav Petkov, linux-kernel, kvm, linux-mm, Hugh Dickins,
Andrea Arcangeli, Izik Eidus
On 31/05/11 18:38, Borislav Petkov wrote:
> On Tue, May 31, 2011 at 05:26:10PM +0800, Brad Campbell wrote:
>> On 31/05/11 13:47, Borislav Petkov wrote:
>>> Looks like a KSM issue. Disabling CONFIG_KSM should at least stop your
>>> machine from oopsing.
>>>
>>> Adding linux-mm.
>>>
>>
>> I initially thought that, so the second panic was produced with KSM
>> disabled from boot.
>>
>> echo 0> /sys/kernel/mm/ksm/run
>>
>> If you still think that compiling ksm out of the kernel will prevent
>> it then I'm willing to give it a go.
>
> Ok, from looking at the code, when KSM inits, it starts the ksm kernel
> thread and it looks like your oops comes from the function that is run
> in the kernel thread - ksm_scan_thread.
>
> So even if you disable it from sysfs, it runs at least once.
>
Just to confirm, I recompiled 2.6.38.7 without KSM enabled and I've been
unable to reproduce the bug, so it looks like you were on the money.
I've moved back to 2.6.38.7 as 2.6.39 has a painful SCSI bug that panics
about 75% of boots, and the reboot cycle required to get luck my way
into a working kernel is just too much hassle.
It would appear that XP zero's its memory space on bootup, so there
would be lots of pages to merge with a couple of relatively freshly
booted XP machines running.
Regards,
Brad.
^ permalink raw reply [flat|nested] 111+ messages in thread
* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
@ 2011-05-31 14:24 ` Brad Campbell
0 siblings, 0 replies; 111+ messages in thread
From: Brad Campbell @ 2011-05-31 14:24 UTC (permalink / raw)
To: Borislav Petkov, linux-kernel, kvm, linux-mm, Hugh Dickins,
Andrea Arcangeli
On 31/05/11 18:38, Borislav Petkov wrote:
> On Tue, May 31, 2011 at 05:26:10PM +0800, Brad Campbell wrote:
>> On 31/05/11 13:47, Borislav Petkov wrote:
>>> Looks like a KSM issue. Disabling CONFIG_KSM should at least stop your
>>> machine from oopsing.
>>>
>>> Adding linux-mm.
>>>
>>
>> I initially thought that, so the second panic was produced with KSM
>> disabled from boot.
>>
>> echo 0> /sys/kernel/mm/ksm/run
>>
>> If you still think that compiling ksm out of the kernel will prevent
>> it then I'm willing to give it a go.
>
> Ok, from looking at the code, when KSM inits, it starts the ksm kernel
> thread and it looks like your oops comes from the function that is run
> in the kernel thread - ksm_scan_thread.
>
> So even if you disable it from sysfs, it runs at least once.
>
Just to confirm, I recompiled 2.6.38.7 without KSM enabled and I've been
unable to reproduce the bug, so it looks like you were on the money.
I've moved back to 2.6.38.7 as 2.6.39 has a painful SCSI bug that panics
about 75% of boots, and the reboot cycle required to get luck my way
into a working kernel is just too much hassle.
It would appear that XP zero's its memory space on bootup, so there
would be lots of pages to merge with a couple of relatively freshly
booted XP machines running.
Regards,
Brad.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 111+ messages in thread
* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
@ 2011-05-31 14:24 ` Brad Campbell
0 siblings, 0 replies; 111+ messages in thread
From: Brad Campbell @ 2011-05-31 14:24 UTC (permalink / raw)
To: Borislav Petkov, linux-kernel, kvm, linux-mm, Hugh Dickins,
Andrea Arcangeli, Izik Eidus
On 31/05/11 18:38, Borislav Petkov wrote:
> On Tue, May 31, 2011 at 05:26:10PM +0800, Brad Campbell wrote:
>> On 31/05/11 13:47, Borislav Petkov wrote:
>>> Looks like a KSM issue. Disabling CONFIG_KSM should at least stop your
>>> machine from oopsing.
>>>
>>> Adding linux-mm.
>>>
>>
>> I initially thought that, so the second panic was produced with KSM
>> disabled from boot.
>>
>> echo 0> /sys/kernel/mm/ksm/run
>>
>> If you still think that compiling ksm out of the kernel will prevent
>> it then I'm willing to give it a go.
>
> Ok, from looking at the code, when KSM inits, it starts the ksm kernel
> thread and it looks like your oops comes from the function that is run
> in the kernel thread - ksm_scan_thread.
>
> So even if you disable it from sysfs, it runs at least once.
>
Just to confirm, I recompiled 2.6.38.7 without KSM enabled and I've been
unable to reproduce the bug, so it looks like you were on the money.
I've moved back to 2.6.38.7 as 2.6.39 has a painful SCSI bug that panics
about 75% of boots, and the reboot cycle required to get luck my way
into a working kernel is just too much hassle.
It would appear that XP zero's its memory space on bootup, so there
would be lots of pages to merge with a couple of relatively freshly
booted XP machines running.
Regards,
Brad.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 111+ messages in thread
* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
2011-05-31 14:24 ` Brad Campbell
@ 2011-05-31 22:31 ` Hugh Dickins
-1 siblings, 0 replies; 111+ messages in thread
From: Hugh Dickins @ 2011-05-31 22:31 UTC (permalink / raw)
To: Brad Campbell
Cc: Borislav Petkov, linux-kernel, kvm, linux-mm, Andrea Arcangeli,
Izik Eidus
On Tue, 31 May 2011, Brad Campbell wrote:
> On 31/05/11 18:38, Borislav Petkov wrote:
> > On Tue, May 31, 2011 at 05:26:10PM +0800, Brad Campbell wrote:
> > > On 31/05/11 13:47, Borislav Petkov wrote:
> > > > Looks like a KSM issue. Disabling CONFIG_KSM should at least stop your
> > > > machine from oopsing.
> > > >
> > > > Adding linux-mm.
> > > >
> > >
> > > I initially thought that, so the second panic was produced with KSM
> > > disabled from boot.
> > >
> > > echo 0> /sys/kernel/mm/ksm/run
> > >
> > > If you still think that compiling ksm out of the kernel will prevent
> > > it then I'm willing to give it a go.
> >
> > Ok, from looking at the code, when KSM inits, it starts the ksm kernel
> > thread and it looks like your oops comes from the function that is run
> > in the kernel thread - ksm_scan_thread.
> >
> > So even if you disable it from sysfs, it runs at least once.
> >
>
> Just to confirm, I recompiled 2.6.38.7 without KSM enabled and I've been
> unable to reproduce the bug, so it looks like you were on the money.
>
> I've moved back to 2.6.38.7 as 2.6.39 has a painful SCSI bug that panics
> about 75% of boots, and the reboot cycle required to get luck my way into a
> working kernel is just too much hassle.
>
> It would appear that XP zero's its memory space on bootup, so there would be
> lots of pages to merge with a couple of relatively freshly booted XP
> machines running.
Thanks for the Cc, Borislav.
Brad, my suspicion is that in each case the top 16 bits of RDX have been
mysteriously corrupted from ffff to 0000, causing the general protection
faults. I don't understand what that has to do with KSM.
But it's only a suspicion, because I can't make sense of the "Code:"
lines in your traces, they have more than the expected 64 bytes, and
only one of them has a ">" (with no "<") to mark faulting instruction.
I did try compiling the 2.6.39 kernel from your config, but of course
we have different compilers, so although I got close, it wasn't exact.
Would you mind mailing me privately (it's about 73MB) the "objdump -trd"
output for your original vmlinux (with KSM on)? (Those -trd options are
the ones I'm used to typing, I bet not they're not all relevant.)
Of course, it's only a tiny fraction of that output that I need,
might be better to cut it down to remove_rmap_item_from_tree and
dup_fd and ksm_scan_thread, if you have the time to do so.
Thanks,
Hugh
^ permalink raw reply [flat|nested] 111+ messages in thread
* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
@ 2011-05-31 22:31 ` Hugh Dickins
0 siblings, 0 replies; 111+ messages in thread
From: Hugh Dickins @ 2011-05-31 22:31 UTC (permalink / raw)
To: Brad Campbell
Cc: Borislav Petkov, linux-kernel, kvm, linux-mm, Andrea Arcangeli,
Izik Eidus
On Tue, 31 May 2011, Brad Campbell wrote:
> On 31/05/11 18:38, Borislav Petkov wrote:
> > On Tue, May 31, 2011 at 05:26:10PM +0800, Brad Campbell wrote:
> > > On 31/05/11 13:47, Borislav Petkov wrote:
> > > > Looks like a KSM issue. Disabling CONFIG_KSM should at least stop your
> > > > machine from oopsing.
> > > >
> > > > Adding linux-mm.
> > > >
> > >
> > > I initially thought that, so the second panic was produced with KSM
> > > disabled from boot.
> > >
> > > echo 0> /sys/kernel/mm/ksm/run
> > >
> > > If you still think that compiling ksm out of the kernel will prevent
> > > it then I'm willing to give it a go.
> >
> > Ok, from looking at the code, when KSM inits, it starts the ksm kernel
> > thread and it looks like your oops comes from the function that is run
> > in the kernel thread - ksm_scan_thread.
> >
> > So even if you disable it from sysfs, it runs at least once.
> >
>
> Just to confirm, I recompiled 2.6.38.7 without KSM enabled and I've been
> unable to reproduce the bug, so it looks like you were on the money.
>
> I've moved back to 2.6.38.7 as 2.6.39 has a painful SCSI bug that panics
> about 75% of boots, and the reboot cycle required to get luck my way into a
> working kernel is just too much hassle.
>
> It would appear that XP zero's its memory space on bootup, so there would be
> lots of pages to merge with a couple of relatively freshly booted XP
> machines running.
Thanks for the Cc, Borislav.
Brad, my suspicion is that in each case the top 16 bits of RDX have been
mysteriously corrupted from ffff to 0000, causing the general protection
faults. I don't understand what that has to do with KSM.
But it's only a suspicion, because I can't make sense of the "Code:"
lines in your traces, they have more than the expected 64 bytes, and
only one of them has a ">" (with no "<") to mark faulting instruction.
I did try compiling the 2.6.39 kernel from your config, but of course
we have different compilers, so although I got close, it wasn't exact.
Would you mind mailing me privately (it's about 73MB) the "objdump -trd"
output for your original vmlinux (with KSM on)? (Those -trd options are
the ones I'm used to typing, I bet not they're not all relevant.)
Of course, it's only a tiny fraction of that output that I need,
might be better to cut it down to remove_rmap_item_from_tree and
dup_fd and ksm_scan_thread, if you have the time to do so.
Thanks,
Hugh
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 111+ messages in thread
* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
2011-05-31 22:31 ` Hugh Dickins
@ 2011-06-01 0:18 ` Brad Campbell
-1 siblings, 0 replies; 111+ messages in thread
From: Brad Campbell @ 2011-06-01 0:18 UTC (permalink / raw)
To: Hugh Dickins
Cc: Borislav Petkov, linux-kernel, kvm, linux-mm, Andrea Arcangeli,
Izik Eidus
On 01/06/11 06:31, Hugh Dickins wrote:
>
> Brad, my suspicion is that in each case the top 16 bits of RDX have been
> mysteriously corrupted from ffff to 0000, causing the general protection
> faults. I don't understand what that has to do with KSM.
No, nor do I. The panic I reproduced with KSM off was in a completely
unrelated code path. To be honest I would not be surprised if it turns
out I have dodgy RAM, although it has passed multiple memtests and I've
tried clocking it down. Just a gut feeling.
> But it's only a suspicion, because I can't make sense of the "Code:"
> lines in your traces, they have more than the expected 64 bytes, and
> only one of them has a ">" (with no"<") to mark faulting instruction.
Yeah, with hindsight I must have removed them when I re-formatted the
code from the oops. Each byte was one line in the syslog so there was a
lot of deleting to get it to a postable format.
> I did try compiling the 2.6.39 kernel from your config, but of course
> we have different compilers, so although I got close, it wasn't exact.
>
> Would you mind mailing me privately (it's about 73MB) the "objdump -trd"
> output for your original vmlinux (with KSM on)? (Those -trd options are
> the ones I'm used to typing, I bet not they're not all relevant.)
>
> Of course, it's only a tiny fraction of that output that I need,
> might be better to cut it down to remove_rmap_item_from_tree and
> dup_fd and ksm_scan_thread, if you have the time to do so.
Ok, so since my initial posting I've figured out how to get a clean oops
out of netconsole, so tonight (after 9PM GMT+8) I'll reproduce the oops
a couple of times. What about I upload the oops, plus the vmlinux, plus
.config and System.map to a server with a fat pipe and give you a link
to it?
At least I can reproduce it quickly and easily.
^ permalink raw reply [flat|nested] 111+ messages in thread
* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
@ 2011-06-01 0:18 ` Brad Campbell
0 siblings, 0 replies; 111+ messages in thread
From: Brad Campbell @ 2011-06-01 0:18 UTC (permalink / raw)
To: Hugh Dickins
Cc: Borislav Petkov, linux-kernel, kvm, linux-mm, Andrea Arcangeli,
Izik Eidus
On 01/06/11 06:31, Hugh Dickins wrote:
>
> Brad, my suspicion is that in each case the top 16 bits of RDX have been
> mysteriously corrupted from ffff to 0000, causing the general protection
> faults. I don't understand what that has to do with KSM.
No, nor do I. The panic I reproduced with KSM off was in a completely
unrelated code path. To be honest I would not be surprised if it turns
out I have dodgy RAM, although it has passed multiple memtests and I've
tried clocking it down. Just a gut feeling.
> But it's only a suspicion, because I can't make sense of the "Code:"
> lines in your traces, they have more than the expected 64 bytes, and
> only one of them has a ">" (with no"<") to mark faulting instruction.
Yeah, with hindsight I must have removed them when I re-formatted the
code from the oops. Each byte was one line in the syslog so there was a
lot of deleting to get it to a postable format.
> I did try compiling the 2.6.39 kernel from your config, but of course
> we have different compilers, so although I got close, it wasn't exact.
>
> Would you mind mailing me privately (it's about 73MB) the "objdump -trd"
> output for your original vmlinux (with KSM on)? (Those -trd options are
> the ones I'm used to typing, I bet not they're not all relevant.)
>
> Of course, it's only a tiny fraction of that output that I need,
> might be better to cut it down to remove_rmap_item_from_tree and
> dup_fd and ksm_scan_thread, if you have the time to do so.
Ok, so since my initial posting I've figured out how to get a clean oops
out of netconsole, so tonight (after 9PM GMT+8) I'll reproduce the oops
a couple of times. What about I upload the oops, plus the vmlinux, plus
.config and System.map to a server with a fat pipe and give you a link
to it?
At least I can reproduce it quickly and easily.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 111+ messages in thread
* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
2011-05-31 22:31 ` Hugh Dickins
@ 2011-06-01 0:37 ` Brad Campbell
-1 siblings, 0 replies; 111+ messages in thread
From: Brad Campbell @ 2011-06-01 0:37 UTC (permalink / raw)
To: Hugh Dickins
Cc: Borislav Petkov, linux-kernel, kvm, linux-mm, Andrea Arcangeli,
Izik Eidus
On 01/06/11 06:31, Hugh Dickins wrote:
> Brad, my suspicion is that in each case the top 16 bits of RDX have been
> mysteriously corrupted from ffff to 0000, causing the general protection
> faults. I don't understand what that has to do with KSM.
>
> But it's only a suspicion, because I can't make sense of the "Code:"
> lines in your traces, they have more than the expected 64 bytes, and
> only one of them has a ">" (with no"<") to mark faulting instruction.
>
> I did try compiling the 2.6.39 kernel from your config, but of course
> we have different compilers, so although I got close, it wasn't exact.
>
> Would you mind mailing me privately (it's about 73MB) the "objdump -trd"
> output for your original vmlinux (with KSM on)? (Those -trd options are
> the ones I'm used to typing, I bet not they're not all relevant.)
>
> Of course, it's only a tiny fraction of that output that I need,
> might be better to cut it down to remove_rmap_item_from_tree and
> dup_fd and ksm_scan_thread, if you have the time to do so.
Would you believe about 20 seconds after I pressed send the kernel oopsed.
http://www.fnarfbargle.com/private/003_kernel_oops/
oops reproduced here, but an un-munged version is in that directory
alongside the kernel.
[36542.880228] general protection fault: 0000 [#1] SMP
[36542.880271] last sysfs file:
/sys/devices/pci0000:00/0000:00:18.3/temp1_input
[36542.880290] CPU 4
[36542.880301] Modules linked in: xt_iprange xt_DSCP xt_length
xt_CLASSIFY sch_sfq xt_CHECKSUM ipt_REJECT ipt_MASQUER
ADE ipt_REDIRECT xt_recent xt_state iptable_filter iptable_nat nf_nat
nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 x
t_TCPMSS xt_tcpmss xt_tcpudp iptable_mangle ip_tables x_tables pppoe
pppox ppp_generic slhc cls_u32 sch_htb deflate z
lib_deflate des_generic cbc ecb crypto_blkcipher sha1_generic md5 hmac
crypto_hash cryptomgr aead crypto_algapi af_ke
y fuse hwmon_vid netconsole configfs vhost_net powernow_k8 mperf kvm_amd
kvm pl2303 usbserial xhci_hcd k10temp i2c_pi
ix4 ahci usb_storage usb_libusual ohci_hcd ehci_hcd r8169 libahci
usbcore mii sata_mv megaraid_sas [last unloaded: sc
si_wait_scan]
[36542.880842]
[36542.880858] Pid: 13346, comm: bash Not tainted 2.6.38.7 #29 To Be
Filled By O.E.M. To Be Filled By O.E.M./880G Ext
reme3
[36542.880911] RIP: 0010:[<ffffffff810cf0de>] [<ffffffff810cf0de>]
do_vfs_ioctl+0x5e/0x510
[36542.880948] RSP: 0018:ffff8802d25a1ec8 EFLAGS: 00010206
[36542.880965] RAX: fffffffffffffff7 RBX: 000088040eb12840 RCX:
00007fff4fe4a4c0
[36542.880984] RDX: 0000000000005413 RSI: 0000000000005413 RDI:
00000000000000ff
[36542.881002] RBP: 00000000000000ff R08: 00007fff4fe4a400 R09:
0000000000000000
[36542.881020] R10: 00007fff4fe4a380 R11: 0000000000000246 R12:
00007fff4fe4a4c0
[36542.881038] R13: 00007fff4fe4a4c0 R14: 0000000000000000 R15:
0000000000000001
[36542.881058] FS: 00007f65f725b700(0000) GS:ffff8800dbd00000(0000)
knlGS:0000000000000000
[36542.881081] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[36542.881098] CR2: 0000000001f01008 CR3: 00000002d25c3000 CR4:
00000000000006e0
[36542.881116] DR0: 00000000000000a0 DR1: 0000000000000000 DR2:
0000000000000003
[36542.881133] DR3: 00000000000000b0 DR6: 00000000ffff0ff0 DR7:
0000000000000400
[36542.881152] Process bash (pid: 13346, threadinfo ffff8802d25a0000,
task ffff88041df88000)
[36542.881172] Stack:
[36542.881183] 0000000000000000 ffff88041df88218 0000000000100000
0000000000000001
[36542.881225] 0000000000000002 00007fff4fe4a2c0 00007fff4fe4a220
0000000000000002
[36542.881268] 0000000000000000 ffffffff81046d6a 000088040eb12840
00000000000000ff
[36542.881312] Call Trace:
[36542.881333] [<ffffffff81046d6a>] ? sys_rt_sigaction+0x8a/0xc0
[36542.881351] [<ffffffff810cf5d9>] ? sys_ioctl+0x49/0x80
[36542.881373] [<ffffffff810023fb>] ? system_call_fastpath+0x16/0x1b
[36542.881389] Code: 76 7b 81 fa 77 58 04 c0 0f 84 77 01 00 00 0f 1f 80
00 00 00 00 0f 87 a2 00 00 00 81 fa 60 54 00 00 0f 1f 40 00 0f 84 ba 01
00 00 <48> 8b 43 18 48 8b 50 30 0f b7 02 25 00 f0 00 00 3d 00 80 00 00
[36542.881793] RIP [<ffffffff810cf0de>] do_vfs_ioctl+0x5e/0x510
[36542.881818] RSP <ffff8802d25a1ec8>
[36542.882082] ---[ end trace 1b8d730cd479e388 ]---
[36542.882126] Kernel panic - not syncing: Fatal exception
[36542.882175] Pid: 13346, comm: bash Tainted: G D 2.6.38.7 #29
[36542.882222] Call Trace:
[36542.882269] [<ffffffff813c7f42>] ? panic+0x92/0x18a
[36542.882318] [<ffffffff81039a41>] ? kmsg_dump+0x41/0xf0
[36542.882366] [<ffffffff810062bd>] ? oops_end+0x8d/0xa0
[36542.882414] [<ffffffff813caeef>] ? general_protection+0x1f/0x30
[36542.882463] [<ffffffff810cf0de>] ? do_vfs_ioctl+0x5e/0x510
[36542.882511] [<ffffffff81046d6a>] ? sys_rt_sigaction+0x8a/0xc0
[36542.882560] [<ffffffff810cf5d9>] ? sys_ioctl+0x49/0x80
[36542.882608] [<ffffffff810023fb>] ? system_call_fastpath+0x16/0x1b
[36542.882688] Rebooting in 60 seconds..[ 33.104725] fuse init (API
version 7.16)
^ permalink raw reply [flat|nested] 111+ messages in thread
* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
@ 2011-06-01 0:37 ` Brad Campbell
0 siblings, 0 replies; 111+ messages in thread
From: Brad Campbell @ 2011-06-01 0:37 UTC (permalink / raw)
To: Hugh Dickins
Cc: Borislav Petkov, linux-kernel, kvm, linux-mm, Andrea Arcangeli,
Izik Eidus
On 01/06/11 06:31, Hugh Dickins wrote:
> Brad, my suspicion is that in each case the top 16 bits of RDX have been
> mysteriously corrupted from ffff to 0000, causing the general protection
> faults. I don't understand what that has to do with KSM.
>
> But it's only a suspicion, because I can't make sense of the "Code:"
> lines in your traces, they have more than the expected 64 bytes, and
> only one of them has a ">" (with no"<") to mark faulting instruction.
>
> I did try compiling the 2.6.39 kernel from your config, but of course
> we have different compilers, so although I got close, it wasn't exact.
>
> Would you mind mailing me privately (it's about 73MB) the "objdump -trd"
> output for your original vmlinux (with KSM on)? (Those -trd options are
> the ones I'm used to typing, I bet not they're not all relevant.)
>
> Of course, it's only a tiny fraction of that output that I need,
> might be better to cut it down to remove_rmap_item_from_tree and
> dup_fd and ksm_scan_thread, if you have the time to do so.
Would you believe about 20 seconds after I pressed send the kernel oopsed.
http://www.fnarfbargle.com/private/003_kernel_oops/
oops reproduced here, but an un-munged version is in that directory
alongside the kernel.
[36542.880228] general protection fault: 0000 [#1] SMP
[36542.880271] last sysfs file:
/sys/devices/pci0000:00/0000:00:18.3/temp1_input
[36542.880290] CPU 4
[36542.880301] Modules linked in: xt_iprange xt_DSCP xt_length
xt_CLASSIFY sch_sfq xt_CHECKSUM ipt_REJECT ipt_MASQUER
ADE ipt_REDIRECT xt_recent xt_state iptable_filter iptable_nat nf_nat
nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 x
t_TCPMSS xt_tcpmss xt_tcpudp iptable_mangle ip_tables x_tables pppoe
pppox ppp_generic slhc cls_u32 sch_htb deflate z
lib_deflate des_generic cbc ecb crypto_blkcipher sha1_generic md5 hmac
crypto_hash cryptomgr aead crypto_algapi af_ke
y fuse hwmon_vid netconsole configfs vhost_net powernow_k8 mperf kvm_amd
kvm pl2303 usbserial xhci_hcd k10temp i2c_pi
ix4 ahci usb_storage usb_libusual ohci_hcd ehci_hcd r8169 libahci
usbcore mii sata_mv megaraid_sas [last unloaded: sc
si_wait_scan]
[36542.880842]
[36542.880858] Pid: 13346, comm: bash Not tainted 2.6.38.7 #29 To Be
Filled By O.E.M. To Be Filled By O.E.M./880G Ext
reme3
[36542.880911] RIP: 0010:[<ffffffff810cf0de>] [<ffffffff810cf0de>]
do_vfs_ioctl+0x5e/0x510
[36542.880948] RSP: 0018:ffff8802d25a1ec8 EFLAGS: 00010206
[36542.880965] RAX: fffffffffffffff7 RBX: 000088040eb12840 RCX:
00007fff4fe4a4c0
[36542.880984] RDX: 0000000000005413 RSI: 0000000000005413 RDI:
00000000000000ff
[36542.881002] RBP: 00000000000000ff R08: 00007fff4fe4a400 R09:
0000000000000000
[36542.881020] R10: 00007fff4fe4a380 R11: 0000000000000246 R12:
00007fff4fe4a4c0
[36542.881038] R13: 00007fff4fe4a4c0 R14: 0000000000000000 R15:
0000000000000001
[36542.881058] FS: 00007f65f725b700(0000) GS:ffff8800dbd00000(0000)
knlGS:0000000000000000
[36542.881081] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[36542.881098] CR2: 0000000001f01008 CR3: 00000002d25c3000 CR4:
00000000000006e0
[36542.881116] DR0: 00000000000000a0 DR1: 0000000000000000 DR2:
0000000000000003
[36542.881133] DR3: 00000000000000b0 DR6: 00000000ffff0ff0 DR7:
0000000000000400
[36542.881152] Process bash (pid: 13346, threadinfo ffff8802d25a0000,
task ffff88041df88000)
[36542.881172] Stack:
[36542.881183] 0000000000000000 ffff88041df88218 0000000000100000
0000000000000001
[36542.881225] 0000000000000002 00007fff4fe4a2c0 00007fff4fe4a220
0000000000000002
[36542.881268] 0000000000000000 ffffffff81046d6a 000088040eb12840
00000000000000ff
[36542.881312] Call Trace:
[36542.881333] [<ffffffff81046d6a>] ? sys_rt_sigaction+0x8a/0xc0
[36542.881351] [<ffffffff810cf5d9>] ? sys_ioctl+0x49/0x80
[36542.881373] [<ffffffff810023fb>] ? system_call_fastpath+0x16/0x1b
[36542.881389] Code: 76 7b 81 fa 77 58 04 c0 0f 84 77 01 00 00 0f 1f 80
00 00 00 00 0f 87 a2 00 00 00 81 fa 60 54 00 00 0f 1f 40 00 0f 84 ba 01
00 00 <48> 8b 43 18 48 8b 50 30 0f b7 02 25 00 f0 00 00 3d 00 80 00 00
[36542.881793] RIP [<ffffffff810cf0de>] do_vfs_ioctl+0x5e/0x510
[36542.881818] RSP <ffff8802d25a1ec8>
[36542.882082] ---[ end trace 1b8d730cd479e388 ]---
[36542.882126] Kernel panic - not syncing: Fatal exception
[36542.882175] Pid: 13346, comm: bash Tainted: G D 2.6.38.7 #29
[36542.882222] Call Trace:
[36542.882269] [<ffffffff813c7f42>] ? panic+0x92/0x18a
[36542.882318] [<ffffffff81039a41>] ? kmsg_dump+0x41/0xf0
[36542.882366] [<ffffffff810062bd>] ? oops_end+0x8d/0xa0
[36542.882414] [<ffffffff813caeef>] ? general_protection+0x1f/0x30
[36542.882463] [<ffffffff810cf0de>] ? do_vfs_ioctl+0x5e/0x510
[36542.882511] [<ffffffff81046d6a>] ? sys_rt_sigaction+0x8a/0xc0
[36542.882560] [<ffffffff810cf5d9>] ? sys_ioctl+0x49/0x80
[36542.882608] [<ffffffff810023fb>] ? system_call_fastpath+0x16/0x1b
[36542.882688] Rebooting in 60 seconds..[ 33.104725] fuse init (API
version 7.16)
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 111+ messages in thread
* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
2011-06-01 0:37 ` Brad Campbell
@ 2011-06-01 1:15 ` Andrea Arcangeli
-1 siblings, 0 replies; 111+ messages in thread
From: Andrea Arcangeli @ 2011-06-01 1:15 UTC (permalink / raw)
To: Brad Campbell
Cc: Hugh Dickins, Borislav Petkov, linux-kernel, kvm, linux-mm, Izik Eidus
Hello,
On Wed, Jun 01, 2011 at 08:37:25AM +0800, Brad Campbell wrote:
> On 01/06/11 06:31, Hugh Dickins wrote:
> > Brad, my suspicion is that in each case the top 16 bits of RDX have been
> > mysteriously corrupted from ffff to 0000, causing the general protection
> > faults. I don't understand what that has to do with KSM.
> >
> > But it's only a suspicion, because I can't make sense of the "Code:"
> > lines in your traces, they have more than the expected 64 bytes, and
> > only one of them has a ">" (with no"<") to mark faulting instruction.
> >
> > I did try compiling the 2.6.39 kernel from your config, but of course
> > we have different compilers, so although I got close, it wasn't exact.
> >
> > Would you mind mailing me privately (it's about 73MB) the "objdump -trd"
> > output for your original vmlinux (with KSM on)? (Those -trd options are
> > the ones I'm used to typing, I bet not they're not all relevant.)
> >
> > Of course, it's only a tiny fraction of that output that I need,
> > might be better to cut it down to remove_rmap_item_from_tree and
> > dup_fd and ksm_scan_thread, if you have the time to do so.
>
> Would you believe about 20 seconds after I pressed send the kernel oopsed.
>
> http://www.fnarfbargle.com/private/003_kernel_oops/
>
> oops reproduced here, but an un-munged version is in that directory
> alongside the kernel.
>
> [36542.880228] general protection fault: 0000 [#1] SMP
Reminds me of another oops that was reported on the kvm list for
2.6.38.1 with message id 4D8C6110.6090204. There the top 16 bits of
rsi were flipped and it was a general protection too because of
hitting on the not mappable virtual range.
http://www.virtall.com/files/temp/kvm.txt
http://www.virtall.com/files/temp/config-2.6.38.1
http://virtall.com/files/temp/mmu-objdump.txt
That oops happened in kvm_unmap_rmapp though, but it looked memory
corruption (Avi suggested use after free) but it was a production
system so we couldn't debug it further.
I recommend next thing to reproduce again with 2.6.39 or
3.0.0-rc1. Let's fix your scsi trouble if needed but it's better you
test with 2.6.39.
We'd need chmod +r vmlinux on private/003_kernel_oops/
Thanks,
Andrea
^ permalink raw reply [flat|nested] 111+ messages in thread
* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
@ 2011-06-01 1:15 ` Andrea Arcangeli
0 siblings, 0 replies; 111+ messages in thread
From: Andrea Arcangeli @ 2011-06-01 1:15 UTC (permalink / raw)
To: Brad Campbell
Cc: Hugh Dickins, Borislav Petkov, linux-kernel, kvm, linux-mm, Izik Eidus
Hello,
On Wed, Jun 01, 2011 at 08:37:25AM +0800, Brad Campbell wrote:
> On 01/06/11 06:31, Hugh Dickins wrote:
> > Brad, my suspicion is that in each case the top 16 bits of RDX have been
> > mysteriously corrupted from ffff to 0000, causing the general protection
> > faults. I don't understand what that has to do with KSM.
> >
> > But it's only a suspicion, because I can't make sense of the "Code:"
> > lines in your traces, they have more than the expected 64 bytes, and
> > only one of them has a ">" (with no"<") to mark faulting instruction.
> >
> > I did try compiling the 2.6.39 kernel from your config, but of course
> > we have different compilers, so although I got close, it wasn't exact.
> >
> > Would you mind mailing me privately (it's about 73MB) the "objdump -trd"
> > output for your original vmlinux (with KSM on)? (Those -trd options are
> > the ones I'm used to typing, I bet not they're not all relevant.)
> >
> > Of course, it's only a tiny fraction of that output that I need,
> > might be better to cut it down to remove_rmap_item_from_tree and
> > dup_fd and ksm_scan_thread, if you have the time to do so.
>
> Would you believe about 20 seconds after I pressed send the kernel oopsed.
>
> http://www.fnarfbargle.com/private/003_kernel_oops/
>
> oops reproduced here, but an un-munged version is in that directory
> alongside the kernel.
>
> [36542.880228] general protection fault: 0000 [#1] SMP
Reminds me of another oops that was reported on the kvm list for
2.6.38.1 with message id 4D8C6110.6090204. There the top 16 bits of
rsi were flipped and it was a general protection too because of
hitting on the not mappable virtual range.
http://www.virtall.com/files/temp/kvm.txt
http://www.virtall.com/files/temp/config-2.6.38.1
http://virtall.com/files/temp/mmu-objdump.txt
That oops happened in kvm_unmap_rmapp though, but it looked memory
corruption (Avi suggested use after free) but it was a production
system so we couldn't debug it further.
I recommend next thing to reproduce again with 2.6.39 or
3.0.0-rc1. Let's fix your scsi trouble if needed but it's better you
test with 2.6.39.
We'd need chmod +r vmlinux on private/003_kernel_oops/
Thanks,
Andrea
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 111+ messages in thread
* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
2011-06-01 1:15 ` Andrea Arcangeli
@ 2011-06-01 2:03 ` Brad Campbell
-1 siblings, 0 replies; 111+ messages in thread
From: Brad Campbell @ 2011-06-01 2:03 UTC (permalink / raw)
To: Andrea Arcangeli
Cc: Hugh Dickins, Borislav Petkov, linux-kernel, kvm, linux-mm, Izik Eidus
On 01/06/11 09:15, Andrea Arcangeli wrote:
> Hello,
>
> On Wed, Jun 01, 2011 at 08:37:25AM +0800, Brad Campbell wrote:
>> On 01/06/11 06:31, Hugh Dickins wrote:
>>> Brad, my suspicion is that in each case the top 16 bits of RDX have been
>>> mysteriously corrupted from ffff to 0000, causing the general protection
>>> faults. I don't understand what that has to do with KSM.
>>>
>>> But it's only a suspicion, because I can't make sense of the "Code:"
>>> lines in your traces, they have more than the expected 64 bytes, and
>>> only one of them has a ">" (with no"<") to mark faulting instruction.
>>>
>>> I did try compiling the 2.6.39 kernel from your config, but of course
>>> we have different compilers, so although I got close, it wasn't exact.
>>>
>>> Would you mind mailing me privately (it's about 73MB) the "objdump -trd"
>>> output for your original vmlinux (with KSM on)? (Those -trd options are
>>> the ones I'm used to typing, I bet not they're not all relevant.)
>>>
>>> Of course, it's only a tiny fraction of that output that I need,
>>> might be better to cut it down to remove_rmap_item_from_tree and
>>> dup_fd and ksm_scan_thread, if you have the time to do so.
>>
>> Would you believe about 20 seconds after I pressed send the kernel oopsed.
>>
>> http://www.fnarfbargle.com/private/003_kernel_oops/
>>
>> oops reproduced here, but an un-munged version is in that directory
>> alongside the kernel.
>>
>> [36542.880228] general protection fault: 0000 [#1] SMP
>
> Reminds me of another oops that was reported on the kvm list for
> 2.6.38.1 with message id 4D8C6110.6090204. There the top 16 bits of
> rsi were flipped and it was a general protection too because of
> hitting on the not mappable virtual range.
>
> http://www.virtall.com/files/temp/kvm.txt
> http://www.virtall.com/files/temp/config-2.6.38.1
> http://virtall.com/files/temp/mmu-objdump.txt
>
> That oops happened in kvm_unmap_rmapp though, but it looked memory
> corruption (Avi suggested use after free) but it was a production
> system so we couldn't debug it further.
>
> I recommend next thing to reproduce again with 2.6.39 or
> 3.0.0-rc1. Let's fix your scsi trouble if needed but it's better you
> test with 2.6.39.
>
> We'd need chmod +r vmlinux on private/003_kernel_oops/
Ok, here we go then.
http://www.fnarfbargle.com/private/004_kernel_oops/
The permissions are right this time.
2.6.39 + KSM
[ 694.227866] general protection fault: 0000 [#1] SMP
[ 694.228001] last sysfs file: /sys/devices/platform/w83627ehf.656/cpu0_vid
[ 694.228050] CPU 3
[ 694.228091] Modules linked in: xt_iprange xt_DSCP xt_length
xt_CLASSIFY sch_sfq xt_CHECKSUM ipt_REJECT ipt_MASQUERADE ipt_REDIRECT
xt_recent xt_state iptable_filter iptable_nat nf_nat nf_conntrack_ipv4
nf_conntrack nf_defrag_ipv4 xt_TCPMSS xt_tcpmss xt_tcpudp iptable_mangle
ip_tables x_tables pppoe pppox ppp_generic slhc cls_u32 sch_htb deflate
zlib_deflate des_generic cbc ecb crypto_blkcipher sha1_generic md5 hmac
crypto_hash cryptomgr aead crypto_algapi af_key fuse w83627ehf hwmon_vid
netconsole configfs vhost_net powernow_k8 mperf kvm_amd kvm pl2303
usbserial i2c_piix4 k10temp xhci_hcd usb_storage usb_libusual ohci_hcd
r8169 ehci_hcd ahci usbcore sata_mv mii libahci megaraid_sas [last
unloaded: scsi_wait_scan]
[ 694.230897]
[ 694.230944] Pid: 11841, comm: keepalive Not tainted 2.6.39 #3 To Be
Filled By O.E.M. To Be Filled By O.E.M./880G Extreme3
[ 694.231111] RIP: 0010:[<ffffffff810db878>] [<ffffffff810db878>]
dup_fd+0x168/0x300
[ 694.231210] RSP: 0018:ffff8802f524fdd0 EFLAGS: 00010206
[ 694.231258] RAX: 00000000000007f8 RBX: ffff8802f5721b80 RCX:
bfffffffffffffff
[ 694.231308] RDX: 00008802f51cacc0 RSI: 00000000000000ff RDI:
0000000000000800
[ 694.231358] RBP: ffff8803bf419800 R08: ffff88030167f6c0 R09:
0000000000000003
[ 694.231407] R10: 0000000000000001 R11: 4000000000000000 R12:
0000000000000100
[ 694.231457] R13: ffff880417aa9800 R14: ffff88030167f440 R15:
ffff8803bd8c1600
[ 694.231507] FS: 00007f02cfc32700(0000) GS:ffff88041fcc0000(0000)
knlGS:0000000000000000
[ 694.231560] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 694.231609] CR2: 00007f02cf5d4810 CR3: 00000002f52c3000 CR4:
00000000000006e0
[ 694.231657] DR0: 0000000000000045 DR1: 0000000000000000 DR2:
0000000000000000
[ 694.231707] DR3: 0000000000000005 DR6: 00000000ffff0ff0 DR7:
0000000000000400
[ 694.231757] Process keepalive (pid: 11841, threadinfo
ffff8802f524e000, task ffff8802f5143690)
[ 694.231809] Stack:
[ 694.231852] ffff8802f5143690 0000000000000020 ffff8802f56badc0
ffff8802f5721b90
[ 694.232050] ffff880417aa54e0 0000000001200011 ffff880417aa54e0
0000000000000000
[ 694.232248] 00007f02cfc329d0 ffff8802f5143690 0000000000000000
ffffffff81037645
[ 694.232448] Call Trace:
[ 694.232499] [<ffffffff81037645>] ? copy_process+0xa75/0xfd0
[ 694.232549] [<ffffffff81037c0d>] ? do_fork+0x6d/0x2b0
[ 694.232599] [<ffffffff810457a9>] ? sigprocmask+0x69/0x100
[ 694.232651] [<ffffffff813d0ca3>] ? stub_clone+0x13/0x20
[ 694.232699] [<ffffffff813d0a3b>] ? system_call_fastpath+0x16/0x1b
[ 694.232745] Code: 4c 89 c2 e8 6b e5 0f 00 45 85 e4 74 78 41 8d 44 24
ff 31 f6 41 ba 01 00 00 00 48 8d 3c c5 08 00 00 00 31 c0 eb 1a 0f 1f 44
00 00 <f0> 48 ff 42 30 48 89 54 05 00 48 83 c0 08 ff c6 48 39 f8 74 3b
[ 694.235190] RIP [<ffffffff810db878>] dup_fd+0x168/0x300
[ 694.235282] RSP <ffff8802f524fdd0>
[ 694.235379] ---[ end trace 949fad05591fcdb3 ]---
[ 694.235428] Kernel panic - not syncing: Fatal exception
[ 694.235478] Pid: 11841, comm: keepalive Tainted: G D 2.6.39 #3
[ 694.235525] Call Trace:
[ 694.235573] [<ffffffff813cd6f5>] ? panic+0x92/0x18a
[ 694.235624] [<ffffffff81038b61>] ? kmsg_dump+0x41/0xf0
[ 694.235676] [<ffffffff810050ad>] ? oops_end+0x8d/0xa0
[ 694.235726] [<ffffffff813d05ef>] ? general_protection+0x1f/0x30
[ 694.235778] [<ffffffff810db878>] ? dup_fd+0x168/0x300
[ 694.235827] [<ffffffff81037645>] ? copy_process+0xa75/0xfd0
[ 694.235877] [<ffffffff81037c0d>] ? do_fork+0x6d/0x2b0
[ 694.235926] [<ffffffff810457a9>] ? sigprocmask+0x69/0x100
[ 694.235978] [<ffffffff813d0ca3>] ? stub_clone+0x13/0x20
[ 694.236028] [<ffffffff813d0a3b>] ? system_call_fastpath+0x16/0x1b
[ 694.236083] Rebooting in 60 seconds..
^ permalink raw reply [flat|nested] 111+ messages in thread
* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
@ 2011-06-01 2:03 ` Brad Campbell
0 siblings, 0 replies; 111+ messages in thread
From: Brad Campbell @ 2011-06-01 2:03 UTC (permalink / raw)
To: Andrea Arcangeli
Cc: Hugh Dickins, Borislav Petkov, linux-kernel, kvm, linux-mm, Izik Eidus
On 01/06/11 09:15, Andrea Arcangeli wrote:
> Hello,
>
> On Wed, Jun 01, 2011 at 08:37:25AM +0800, Brad Campbell wrote:
>> On 01/06/11 06:31, Hugh Dickins wrote:
>>> Brad, my suspicion is that in each case the top 16 bits of RDX have been
>>> mysteriously corrupted from ffff to 0000, causing the general protection
>>> faults. I don't understand what that has to do with KSM.
>>>
>>> But it's only a suspicion, because I can't make sense of the "Code:"
>>> lines in your traces, they have more than the expected 64 bytes, and
>>> only one of them has a ">" (with no"<") to mark faulting instruction.
>>>
>>> I did try compiling the 2.6.39 kernel from your config, but of course
>>> we have different compilers, so although I got close, it wasn't exact.
>>>
>>> Would you mind mailing me privately (it's about 73MB) the "objdump -trd"
>>> output for your original vmlinux (with KSM on)? (Those -trd options are
>>> the ones I'm used to typing, I bet not they're not all relevant.)
>>>
>>> Of course, it's only a tiny fraction of that output that I need,
>>> might be better to cut it down to remove_rmap_item_from_tree and
>>> dup_fd and ksm_scan_thread, if you have the time to do so.
>>
>> Would you believe about 20 seconds after I pressed send the kernel oopsed.
>>
>> http://www.fnarfbargle.com/private/003_kernel_oops/
>>
>> oops reproduced here, but an un-munged version is in that directory
>> alongside the kernel.
>>
>> [36542.880228] general protection fault: 0000 [#1] SMP
>
> Reminds me of another oops that was reported on the kvm list for
> 2.6.38.1 with message id 4D8C6110.6090204. There the top 16 bits of
> rsi were flipped and it was a general protection too because of
> hitting on the not mappable virtual range.
>
> http://www.virtall.com/files/temp/kvm.txt
> http://www.virtall.com/files/temp/config-2.6.38.1
> http://virtall.com/files/temp/mmu-objdump.txt
>
> That oops happened in kvm_unmap_rmapp though, but it looked memory
> corruption (Avi suggested use after free) but it was a production
> system so we couldn't debug it further.
>
> I recommend next thing to reproduce again with 2.6.39 or
> 3.0.0-rc1. Let's fix your scsi trouble if needed but it's better you
> test with 2.6.39.
>
> We'd need chmod +r vmlinux on private/003_kernel_oops/
Ok, here we go then.
http://www.fnarfbargle.com/private/004_kernel_oops/
The permissions are right this time.
2.6.39 + KSM
[ 694.227866] general protection fault: 0000 [#1] SMP
[ 694.228001] last sysfs file: /sys/devices/platform/w83627ehf.656/cpu0_vid
[ 694.228050] CPU 3
[ 694.228091] Modules linked in: xt_iprange xt_DSCP xt_length
xt_CLASSIFY sch_sfq xt_CHECKSUM ipt_REJECT ipt_MASQUERADE ipt_REDIRECT
xt_recent xt_state iptable_filter iptable_nat nf_nat nf_conntrack_ipv4
nf_conntrack nf_defrag_ipv4 xt_TCPMSS xt_tcpmss xt_tcpudp iptable_mangle
ip_tables x_tables pppoe pppox ppp_generic slhc cls_u32 sch_htb deflate
zlib_deflate des_generic cbc ecb crypto_blkcipher sha1_generic md5 hmac
crypto_hash cryptomgr aead crypto_algapi af_key fuse w83627ehf hwmon_vid
netconsole configfs vhost_net powernow_k8 mperf kvm_amd kvm pl2303
usbserial i2c_piix4 k10temp xhci_hcd usb_storage usb_libusual ohci_hcd
r8169 ehci_hcd ahci usbcore sata_mv mii libahci megaraid_sas [last
unloaded: scsi_wait_scan]
[ 694.230897]
[ 694.230944] Pid: 11841, comm: keepalive Not tainted 2.6.39 #3 To Be
Filled By O.E.M. To Be Filled By O.E.M./880G Extreme3
[ 694.231111] RIP: 0010:[<ffffffff810db878>] [<ffffffff810db878>]
dup_fd+0x168/0x300
[ 694.231210] RSP: 0018:ffff8802f524fdd0 EFLAGS: 00010206
[ 694.231258] RAX: 00000000000007f8 RBX: ffff8802f5721b80 RCX:
bfffffffffffffff
[ 694.231308] RDX: 00008802f51cacc0 RSI: 00000000000000ff RDI:
0000000000000800
[ 694.231358] RBP: ffff8803bf419800 R08: ffff88030167f6c0 R09:
0000000000000003
[ 694.231407] R10: 0000000000000001 R11: 4000000000000000 R12:
0000000000000100
[ 694.231457] R13: ffff880417aa9800 R14: ffff88030167f440 R15:
ffff8803bd8c1600
[ 694.231507] FS: 00007f02cfc32700(0000) GS:ffff88041fcc0000(0000)
knlGS:0000000000000000
[ 694.231560] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 694.231609] CR2: 00007f02cf5d4810 CR3: 00000002f52c3000 CR4:
00000000000006e0
[ 694.231657] DR0: 0000000000000045 DR1: 0000000000000000 DR2:
0000000000000000
[ 694.231707] DR3: 0000000000000005 DR6: 00000000ffff0ff0 DR7:
0000000000000400
[ 694.231757] Process keepalive (pid: 11841, threadinfo
ffff8802f524e000, task ffff8802f5143690)
[ 694.231809] Stack:
[ 694.231852] ffff8802f5143690 0000000000000020 ffff8802f56badc0
ffff8802f5721b90
[ 694.232050] ffff880417aa54e0 0000000001200011 ffff880417aa54e0
0000000000000000
[ 694.232248] 00007f02cfc329d0 ffff8802f5143690 0000000000000000
ffffffff81037645
[ 694.232448] Call Trace:
[ 694.232499] [<ffffffff81037645>] ? copy_process+0xa75/0xfd0
[ 694.232549] [<ffffffff81037c0d>] ? do_fork+0x6d/0x2b0
[ 694.232599] [<ffffffff810457a9>] ? sigprocmask+0x69/0x100
[ 694.232651] [<ffffffff813d0ca3>] ? stub_clone+0x13/0x20
[ 694.232699] [<ffffffff813d0a3b>] ? system_call_fastpath+0x16/0x1b
[ 694.232745] Code: 4c 89 c2 e8 6b e5 0f 00 45 85 e4 74 78 41 8d 44 24
ff 31 f6 41 ba 01 00 00 00 48 8d 3c c5 08 00 00 00 31 c0 eb 1a 0f 1f 44
00 00 <f0> 48 ff 42 30 48 89 54 05 00 48 83 c0 08 ff c6 48 39 f8 74 3b
[ 694.235190] RIP [<ffffffff810db878>] dup_fd+0x168/0x300
[ 694.235282] RSP <ffff8802f524fdd0>
[ 694.235379] ---[ end trace 949fad05591fcdb3 ]---
[ 694.235428] Kernel panic - not syncing: Fatal exception
[ 694.235478] Pid: 11841, comm: keepalive Tainted: G D 2.6.39 #3
[ 694.235525] Call Trace:
[ 694.235573] [<ffffffff813cd6f5>] ? panic+0x92/0x18a
[ 694.235624] [<ffffffff81038b61>] ? kmsg_dump+0x41/0xf0
[ 694.235676] [<ffffffff810050ad>] ? oops_end+0x8d/0xa0
[ 694.235726] [<ffffffff813d05ef>] ? general_protection+0x1f/0x30
[ 694.235778] [<ffffffff810db878>] ? dup_fd+0x168/0x300
[ 694.235827] [<ffffffff81037645>] ? copy_process+0xa75/0xfd0
[ 694.235877] [<ffffffff81037c0d>] ? do_fork+0x6d/0x2b0
[ 694.235926] [<ffffffff810457a9>] ? sigprocmask+0x69/0x100
[ 694.235978] [<ffffffff813d0ca3>] ? stub_clone+0x13/0x20
[ 694.236028] [<ffffffff813d0a3b>] ? system_call_fastpath+0x16/0x1b
[ 694.236083] Rebooting in 60 seconds..
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 111+ messages in thread
* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
2011-06-01 1:15 ` Andrea Arcangeli
@ 2011-06-01 4:52 ` Hugh Dickins
-1 siblings, 0 replies; 111+ messages in thread
From: Hugh Dickins @ 2011-06-01 4:52 UTC (permalink / raw)
To: Andrea Arcangeli
Cc: Brad Campbell, Borislav Petkov, linux-kernel, kvm, linux-mm
On Wed, 1 Jun 2011, Andrea Arcangeli wrote:
> On Wed, Jun 01, 2011 at 08:37:25AM +0800, Brad Campbell wrote:
> > On 01/06/11 06:31, Hugh Dickins wrote:
> > > Brad, my suspicion is that in each case the top 16 bits of RDX have been
> > > mysteriously corrupted from ffff to 0000, causing the general protection
> > > faults. I don't understand what that has to do with KSM.
> > >
> > > But it's only a suspicion, because I can't make sense of the "Code:"
> > > lines in your traces, they have more than the expected 64 bytes, and
> > > only one of them has a ">" (with no"<") to mark faulting instruction.
> > >
> > > I did try compiling the 2.6.39 kernel from your config, but of course
> > > we have different compilers, so although I got close, it wasn't exact.
> > >
> > > Would you mind mailing me privately (it's about 73MB) the "objdump -trd"
> > > output for your original vmlinux (with KSM on)? (Those -trd options are
> > > the ones I'm used to typing, I bet not they're not all relevant.)
> > >
> > > Of course, it's only a tiny fraction of that output that I need,
> > > might be better to cut it down to remove_rmap_item_from_tree and
> > > dup_fd and ksm_scan_thread, if you have the time to do so.
> >
> > Would you believe about 20 seconds after I pressed send the kernel oopsed.
> >
> > http://www.fnarfbargle.com/private/003_kernel_oops/
> >
> > oops reproduced here, but an un-munged version is in that directory
> > alongside the kernel.
> >
> > [36542.880228] general protection fault: 0000 [#1] SMP
>
> Reminds me of another oops that was reported on the kvm list for
> 2.6.38.1 with message id 4D8C6110.6090204. There the top 16 bits of
> rsi were flipped and it was a general protection too because of
> hitting on the not mappable virtual range.
>
> http://www.virtall.com/files/temp/kvm.txt
> http://www.virtall.com/files/temp/config-2.6.38.1
> http://virtall.com/files/temp/mmu-objdump.txt
>
> That oops happened in kvm_unmap_rmapp though, but it looked memory
> corruption (Avi suggested use after free) but it was a production
> system so we couldn't debug it further.
>
> I recommend next thing to reproduce again with 2.6.39 or
> 3.0.0-rc1. Let's fix your scsi trouble if needed but it's better you
> test with 2.6.39.
Brad, thanks for this and the other further crash, with vmlinux etc:
very helpful info.
Andrea, I'm pretty sure you're right to connect Brad's report with
the one above.
In four out of five of Brad's reports (cannot tell in the fifth),
the bad pointer (with top 16 bits 0000 instead of ffff) had been
loaded from SLUB memory at an address offset 0x7f8 (1 case) or
0xff8 (3 cases) i.e. it's the short at 0x7fe or 0xffe that has
been zeroed.
No reason to suspect KSM's rmap_item code, or file table handling:
they just seem to be the victims of corruption from elsewhere.
I notice %rax and %rsi, the corrupted pointer in your kvm.txt
case, is itself a ...7f8 address; and %r13 an ...ff8 address.
I've not even glanced at the code, but I wonder if that implies
that KVM is close to the origin of the corruption.
I doubt I'll be able to spend more time on this, hope you can
take over.
I guess Brad could try SLUB debugging, boot with slub_debug=P
for poisoning perhaps; though it might upset alignments and
drive the problem underground. Or see if the same happens
with SLAB instead of SLUB.
But I rather hope that you or someone will understand the 7fe clue.
Thanks,
Hugh
^ permalink raw reply [flat|nested] 111+ messages in thread
* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
@ 2011-06-01 4:52 ` Hugh Dickins
0 siblings, 0 replies; 111+ messages in thread
From: Hugh Dickins @ 2011-06-01 4:52 UTC (permalink / raw)
To: Andrea Arcangeli
Cc: Brad Campbell, Borislav Petkov, linux-kernel, kvm, linux-mm
On Wed, 1 Jun 2011, Andrea Arcangeli wrote:
> On Wed, Jun 01, 2011 at 08:37:25AM +0800, Brad Campbell wrote:
> > On 01/06/11 06:31, Hugh Dickins wrote:
> > > Brad, my suspicion is that in each case the top 16 bits of RDX have been
> > > mysteriously corrupted from ffff to 0000, causing the general protection
> > > faults. I don't understand what that has to do with KSM.
> > >
> > > But it's only a suspicion, because I can't make sense of the "Code:"
> > > lines in your traces, they have more than the expected 64 bytes, and
> > > only one of them has a ">" (with no"<") to mark faulting instruction.
> > >
> > > I did try compiling the 2.6.39 kernel from your config, but of course
> > > we have different compilers, so although I got close, it wasn't exact.
> > >
> > > Would you mind mailing me privately (it's about 73MB) the "objdump -trd"
> > > output for your original vmlinux (with KSM on)? (Those -trd options are
> > > the ones I'm used to typing, I bet not they're not all relevant.)
> > >
> > > Of course, it's only a tiny fraction of that output that I need,
> > > might be better to cut it down to remove_rmap_item_from_tree and
> > > dup_fd and ksm_scan_thread, if you have the time to do so.
> >
> > Would you believe about 20 seconds after I pressed send the kernel oopsed.
> >
> > http://www.fnarfbargle.com/private/003_kernel_oops/
> >
> > oops reproduced here, but an un-munged version is in that directory
> > alongside the kernel.
> >
> > [36542.880228] general protection fault: 0000 [#1] SMP
>
> Reminds me of another oops that was reported on the kvm list for
> 2.6.38.1 with message id 4D8C6110.6090204. There the top 16 bits of
> rsi were flipped and it was a general protection too because of
> hitting on the not mappable virtual range.
>
> http://www.virtall.com/files/temp/kvm.txt
> http://www.virtall.com/files/temp/config-2.6.38.1
> http://virtall.com/files/temp/mmu-objdump.txt
>
> That oops happened in kvm_unmap_rmapp though, but it looked memory
> corruption (Avi suggested use after free) but it was a production
> system so we couldn't debug it further.
>
> I recommend next thing to reproduce again with 2.6.39 or
> 3.0.0-rc1. Let's fix your scsi trouble if needed but it's better you
> test with 2.6.39.
Brad, thanks for this and the other further crash, with vmlinux etc:
very helpful info.
Andrea, I'm pretty sure you're right to connect Brad's report with
the one above.
In four out of five of Brad's reports (cannot tell in the fifth),
the bad pointer (with top 16 bits 0000 instead of ffff) had been
loaded from SLUB memory at an address offset 0x7f8 (1 case) or
0xff8 (3 cases) i.e. it's the short at 0x7fe or 0xffe that has
been zeroed.
No reason to suspect KSM's rmap_item code, or file table handling:
they just seem to be the victims of corruption from elsewhere.
I notice %rax and %rsi, the corrupted pointer in your kvm.txt
case, is itself a ...7f8 address; and %r13 an ...ff8 address.
I've not even glanced at the code, but I wonder if that implies
that KVM is close to the origin of the corruption.
I doubt I'll be able to spend more time on this, hope you can
take over.
I guess Brad could try SLUB debugging, boot with slub_debug=P
for poisoning perhaps; though it might upset alignments and
drive the problem underground. Or see if the same happens
with SLAB instead of SLUB.
But I rather hope that you or someone will understand the 7fe clue.
Thanks,
Hugh
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 111+ messages in thread
* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
2011-06-01 4:52 ` Hugh Dickins
@ 2011-06-01 6:31 ` Brad Campbell
-1 siblings, 0 replies; 111+ messages in thread
From: Brad Campbell @ 2011-06-01 6:31 UTC (permalink / raw)
To: Hugh Dickins
Cc: Andrea Arcangeli, Borislav Petkov, linux-kernel, kvm, linux-mm
On 01/06/11 12:52, Hugh Dickins wrote:
>
> I guess Brad could try SLUB debugging, boot with slub_debug=P
> for poisoning perhaps; though it might upset alignments and
> drive the problem underground. Or see if the same happens
> with SLAB instead of SLUB.
Not much use I'm afraid.
This is all I get in the log
[ 3161.300073]
=============================================================================
[ 3161.300147] BUG kmalloc-512: Freechain corrupt
The qemu process is then frozen, unkillable but reported in state "R"
13881 ? R 3:27 /usr/bin/qemu -S -M pc-0.13 -enable-kvm -m
1024 -smp 2,sockets=2,cores=1,threads=1 -nam
The machine then progressively dies until it's frozen solid with no
further error messages.
I stupidly forgot to do an alt-sysrq-t prior to doing an alt-sysrq-b,
but at least it responded to that.
On the bright side I can reproduce it at will.
^ permalink raw reply [flat|nested] 111+ messages in thread
* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
@ 2011-06-01 6:31 ` Brad Campbell
0 siblings, 0 replies; 111+ messages in thread
From: Brad Campbell @ 2011-06-01 6:31 UTC (permalink / raw)
To: Hugh Dickins
Cc: Andrea Arcangeli, Borislav Petkov, linux-kernel, kvm, linux-mm
On 01/06/11 12:52, Hugh Dickins wrote:
>
> I guess Brad could try SLUB debugging, boot with slub_debug=P
> for poisoning perhaps; though it might upset alignments and
> drive the problem underground. Or see if the same happens
> with SLAB instead of SLUB.
Not much use I'm afraid.
This is all I get in the log
[ 3161.300073]
=============================================================================
[ 3161.300147] BUG kmalloc-512: Freechain corrupt
The qemu process is then frozen, unkillable but reported in state "R"
13881 ? R 3:27 /usr/bin/qemu -S -M pc-0.13 -enable-kvm -m
1024 -smp 2,sockets=2,cores=1,threads=1 -nam
The machine then progressively dies until it's frozen solid with no
further error messages.
I stupidly forgot to do an alt-sysrq-t prior to doing an alt-sysrq-b,
but at least it responded to that.
On the bright side I can reproduce it at will.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 111+ messages in thread
* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
2011-06-01 6:31 ` Brad Campbell
@ 2011-06-01 6:56 ` Avi Kivity
-1 siblings, 0 replies; 111+ messages in thread
From: Avi Kivity @ 2011-06-01 6:56 UTC (permalink / raw)
To: Brad Campbell
Cc: Hugh Dickins, Andrea Arcangeli, Borislav Petkov, linux-kernel,
kvm, linux-mm
On 06/01/2011 09:31 AM, Brad Campbell wrote:
> On 01/06/11 12:52, Hugh Dickins wrote:
>
>>
>> I guess Brad could try SLUB debugging, boot with slub_debug=P
>> for poisoning perhaps; though it might upset alignments and
>> drive the problem underground. Or see if the same happens
>> with SLAB instead of SLUB.
>
> Not much use I'm afraid.
> This is all I get in the log
>
> [ 3161.300073]
> =============================================================================
> [ 3161.300147] BUG kmalloc-512: Freechain corrupt
>
> The qemu process is then frozen, unkillable but reported in state "R"
>
> 13881 ? R 3:27 /usr/bin/qemu -S -M pc-0.13 -enable-kvm -m
> 1024 -smp 2,sockets=2,cores=1,threads=1 -nam
>
> The machine then progressively dies until it's frozen solid with no
> further error messages.
>
> I stupidly forgot to do an alt-sysrq-t prior to doing an alt-sysrq-b,
> but at least it responded to that.
>
> On the bright side I can reproduce it at will.
Please try slub_debug=FZPU; that should point the finger (hopefully at
somebody else).
--
I have a truly marvellous patch that fixes the bug which this
signature is too narrow to contain.
^ permalink raw reply [flat|nested] 111+ messages in thread
* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
@ 2011-06-01 6:56 ` Avi Kivity
0 siblings, 0 replies; 111+ messages in thread
From: Avi Kivity @ 2011-06-01 6:56 UTC (permalink / raw)
To: Brad Campbell
Cc: Hugh Dickins, Andrea Arcangeli, Borislav Petkov, linux-kernel,
kvm, linux-mm
On 06/01/2011 09:31 AM, Brad Campbell wrote:
> On 01/06/11 12:52, Hugh Dickins wrote:
>
>>
>> I guess Brad could try SLUB debugging, boot with slub_debug=P
>> for poisoning perhaps; though it might upset alignments and
>> drive the problem underground. Or see if the same happens
>> with SLAB instead of SLUB.
>
> Not much use I'm afraid.
> This is all I get in the log
>
> [ 3161.300073]
> =============================================================================
> [ 3161.300147] BUG kmalloc-512: Freechain corrupt
>
> The qemu process is then frozen, unkillable but reported in state "R"
>
> 13881 ? R 3:27 /usr/bin/qemu -S -M pc-0.13 -enable-kvm -m
> 1024 -smp 2,sockets=2,cores=1,threads=1 -nam
>
> The machine then progressively dies until it's frozen solid with no
> further error messages.
>
> I stupidly forgot to do an alt-sysrq-t prior to doing an alt-sysrq-b,
> but at least it responded to that.
>
> On the bright side I can reproduce it at will.
Please try slub_debug=FZPU; that should point the finger (hopefully at
somebody else).
--
I have a truly marvellous patch that fixes the bug which this
signature is too narrow to contain.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 111+ messages in thread
* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
2011-06-01 6:56 ` Avi Kivity
(?)
@ 2011-06-01 9:29 ` Brad Campbell
-1 siblings, 0 replies; 111+ messages in thread
From: Brad Campbell @ 2011-06-01 9:29 UTC (permalink / raw)
To: Avi Kivity
Cc: Hugh Dickins, Andrea Arcangeli, Borislav Petkov, linux-kernel,
kvm, linux-mm
On 01/06/11 14:56, Avi Kivity wrote:
> On 06/01/2011 09:31 AM, Brad Campbell wrote:
>> On 01/06/11 12:52, Hugh Dickins wrote:
>>
>>>
>>> I guess Brad could try SLUB debugging, boot with slub_debug=P
>>> for poisoning perhaps; though it might upset alignments and
>>> drive the problem underground. Or see if the same happens
>>> with SLAB instead of SLUB.
>>
>> Not much use I'm afraid.
>> This is all I get in the log
>>
>> [ 3161.300073]
>> =============================================================================
>>
>> [ 3161.300147] BUG kmalloc-512: Freechain corrupt
>>
>> The qemu process is then frozen, unkillable but reported in state "R"
>>
>> 13881 ? R 3:27 /usr/bin/qemu -S -M pc-0.13 -enable-kvm -m 1024 -smp
>> 2,sockets=2,cores=1,threads=1 -nam
>>
>> The machine then progressively dies until it's frozen solid with no
>> further error messages.
>>
>> I stupidly forgot to do an alt-sysrq-t prior to doing an alt-sysrq-b,
>> but at least it responded to that.
>>
>> On the bright side I can reproduce it at will.
>
> Please try slub_debug=FZPU; that should point the finger (hopefully at
> somebody else).
>
Well the first attempt locked the machine solid. No network, no console..
I saw
"=========================================================================="
on the console.. nothing after that. Would not respond to sysrq-t or any
other sysrq combination other than -b, which rebooted the box.
No output on netconsole at all, I had to walk to the other building to
look at the monitor and reboot it.
The second attempt jammed netconsole again, but I managed to get this
from an ssh session I already had established. The machine died a slow
and horrible death, but remained interactive enough for me to reboot it with
echo b > /proc/sysrq-trigger
Nothing else worked.
[ 376.269051]
=============================================================================
[ 413.755328]
=============================================================================
[ 413.755337] BUG kmalloc-1024: Object padding overwritten
[ 413.755341]
-----------------------------------------------------------------------------
[ 413.755344]
[ 413.755349] INFO: 0xffff880417179566-0xffff880417179567. First byte
0x0 instead of 0x5a
[ 413.755363] INFO: Allocated in tcp_send_ack+0x26/0x120 age=5320 cpu=5
pid=0
[ 413.755374] INFO: Freed in __kfree_skb+0x11/0x90 age=5320 cpu=5 pid=0
[ 413.755380] INFO: Slab 0xffffea000e50d240 objects=29 used=5
fp=0xffff880417179120 flags=0x80000000000040c1
[ 413.755386] INFO: Object 0xffff880417179120 @offset=4384
fp=0xffff8804171799b0
[ 413.755389]
[ 413.755392] Bytes b4 0xffff880417179110: d9 2b 00 00 01 00 00 00 5a
5a 5a 5a 5a 5a 5a 5a Ù+......ZZZZZZZZ
[ 413.755408] Object 0xffff880417179120: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755423] Object 0xffff880417179130: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755438] Object 0xffff880417179140: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755452] Object 0xffff880417179150: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755466] Object 0xffff880417179160: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755480] Object 0xffff880417179170: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755494] Object 0xffff880417179180: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755508] Object 0xffff880417179190: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755522] Object 0xffff8804171791a0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755536] Object 0xffff8804171791b0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755551] Object 0xffff8804171791c0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755565] Object 0xffff8804171791d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755579] Object 0xffff8804171791e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755593] Object 0xffff8804171791f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755607] Object 0xffff880417179200: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755621] Object 0xffff880417179210: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755635] Object 0xffff880417179220: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755650] Object 0xffff880417179230: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755664] Object 0xffff880417179240: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755678] Object 0xffff880417179250: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755692] Object 0xffff880417179260: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755706] Object 0xffff880417179270: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755720] Object 0xffff880417179280: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755734] Object 0xffff880417179290: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755749] Object 0xffff8804171792a0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755763] Object 0xffff8804171792b0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755777] Object 0xffff8804171792c0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755791] Object 0xffff8804171792d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755805] Object 0xffff8804171792e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755819] Object 0xffff8804171792f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755834] Object 0xffff880417179300: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755848] Object 0xffff880417179310: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755862] Object 0xffff880417179320: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755876] Object 0xffff880417179330: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755890] Object 0xffff880417179340: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755904] Object 0xffff880417179350: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755919] Object 0xffff880417179360: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755933] Object 0xffff880417179370: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755947] Object 0xffff880417179380: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755961] Object 0xffff880417179390: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755975] Object 0xffff8804171793a0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755989] Object 0xffff8804171793b0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.756004] Object 0xffff8804171793c0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.756018] Object 0xffff8804171793d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.756032] Object 0xffff8804171793e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.756046] Object 0xffff8804171793f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.756060] Object 0xffff880417179400: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.756074] Object 0xffff880417179410: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.756089] Object 0xffff880417179420: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.756103] Object 0xffff880417179430: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.756117] Object 0xffff880417179440: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.756131] Object 0xffff880417179450: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.756145] Object 0xffff880417179460: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.756160] Object 0xffff880417179470: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.756174] Object 0xffff880417179480: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.756188] Object 0xffff880417179490: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.756202] Object 0xffff8804171794a0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.756216] Object 0xffff8804171794b0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.756230] Object 0xffff8804171794c0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.756245] Object 0xffff8804171794d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.756259] Object 0xffff8804171794e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.756273] Object 0xffff8804171794f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.756287] Object 0xffff880417179500: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.756301] Object 0xffff880417179510: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b a5 kkkkkkkkkkkkkkk¥
[ 413.756316] Redzone 0xffff880417179520: bb bb bb bb bb bb bb bb
»»»»»»»»
[ 413.756329] Padding 0xffff880417179560: 5a 5a 5a 5a 5a 5a 00 00
ZZZZZZ..
[ 413.756345] Pid: 5247, comm: qemu Not tainted 2.6.39 #2
[ 413.756349] Call Trace:
[ 413.756353] <IRQ> [<ffffffff810b7ccd>] ?
check_bytes_and_report+0x10d/0x150
[ 413.756372] [<ffffffff81318f1c>] ? pskb_expand_head+0x15c/0x250
[ 413.756379] [<ffffffff810b7db9>] ? check_object+0xa9/0x260
[ 413.756387] [<ffffffff81318f1c>] ? pskb_expand_head+0x15c/0x250
[ 413.756393] [<ffffffff810b86f4>] ? alloc_debug_processing+0x104/0x190
[ 413.756400] [<ffffffff810b9ac2>] ? T.912+0x272/0x2d0
[ 413.756409] [<ffffffff810ba59d>] ? __kmalloc+0x10d/0x160
[ 413.756416] [<ffffffff81318f1c>] ? pskb_expand_head+0x15c/0x250
[ 413.756424] [<ffffffff813a6c45>] ? nf_bridge_copy_header+0x145/0x160
[ 413.756431] [<ffffffff8139f78d>] ? br_dev_queue_push_xmit+0x6d/0x80
[ 413.756439] [<ffffffff813a55a0>] ? br_nf_post_routing+0x2a0/0x2f0
[ 413.756447] [<ffffffff81346bc4>] ? nf_iterate+0x84/0xb0
[ 413.756453] [<ffffffff8139f720>] ? br_flood_deliver+0x20/0x20
[ 413.756459] [<ffffffff81346c64>] ? nf_hook_slow+0x74/0x120
[ 413.756465] [<ffffffff8139f720>] ? br_flood_deliver+0x20/0x20
[ 413.756472] [<ffffffff8139f7da>] ? br_forward_finish+0x3a/0x60
[ 413.756479] [<ffffffff813a5758>] ? br_nf_forward_finish+0x168/0x170
[ 413.756487] [<ffffffff813a5c90>] ? br_nf_forward_ip+0x360/0x3a0
[ 413.756492] [<ffffffff81346bc4>] ? nf_iterate+0x84/0xb0
[ 413.756498] [<ffffffff8139f7a0>] ? br_dev_queue_push_xmit+0x80/0x80
[ 413.756504] [<ffffffff81346c64>] ? nf_hook_slow+0x74/0x120
[ 413.756510] [<ffffffff8139f7a0>] ? br_dev_queue_push_xmit+0x80/0x80
[ 413.756516] [<ffffffff8139f800>] ? br_forward_finish+0x60/0x60
[ 413.756522] [<ffffffff8139f800>] ? br_forward_finish+0x60/0x60
[ 413.756528] [<ffffffff8139f875>] ? __br_forward+0x75/0xc0
[ 413.756534] [<ffffffff8139f426>] ? deliver_clone+0x36/0x60
[ 413.756540] [<ffffffff8139f69d>] ? br_flood+0xbd/0x100
[ 413.756546] [<ffffffff813a05b0>] ? br_handle_local_finish+0x40/0x40
[ 413.756552] [<ffffffff813a080e>] ? br_handle_frame_finish+0x25e/0x280
[ 413.756560] [<ffffffff813a60f0>] ? br_nf_pre_routing_finish+0x1a0/0x330
[ 413.756568] [<ffffffff813a6958>] ? br_nf_pre_routing+0x6d8/0x800
[ 413.756577] [<ffffffff8102d46a>] ? enqueue_task+0x3a/0x90
[ 413.756582] [<ffffffff81346bc4>] ? nf_iterate+0x84/0xb0
[ 413.756589] [<ffffffff813a05b0>] ? br_handle_local_finish+0x40/0x40
[ 413.756594] [<ffffffff81346c64>] ? nf_hook_slow+0x74/0x120
[ 413.756600] [<ffffffff813a05b0>] ? br_handle_local_finish+0x40/0x40
[ 413.756607] [<ffffffff810339b0>] ? try_to_wake_up+0x2c0/0x2c0
[ 413.756613] [<ffffffff813a09d9>] ? br_handle_frame+0x1a9/0x280
[ 413.756620] [<ffffffff813a0830>] ? br_handle_frame_finish+0x280/0x280
[ 413.756627] [<ffffffff81320ef7>] ? __netif_receive_skb+0x157/0x5c0
[ 413.756634] [<ffffffff81321443>] ? process_backlog+0xe3/0x1d0
[ 413.756641] [<ffffffff81321da5>] ? net_rx_action+0xc5/0x1d0
[ 413.756650] [<ffffffff8103df11>] ? __do_softirq+0x91/0x120
[ 413.756657] [<ffffffff813d838c>] ? call_softirq+0x1c/0x30
[ 413.756660] <EOI> [<ffffffff81003cbd>] ? do_softirq+0x4d/0x80
[ 413.756673] [<ffffffff81321ece>] ? netif_rx_ni+0x1e/0x30
[ 413.756681] [<ffffffff812b3ae2>] ? tun_chr_aio_write+0x332/0x4e0
[ 413.756688] [<ffffffff812b37b0>] ? tun_sendmsg+0x4d0/0x4d0
[ 413.756697] [<ffffffff810c24e9>] ? do_sync_readv_writev+0xa9/0xf0
[ 413.756704] [<ffffffff81063f9c>] ? do_futex+0x13c/0xa70
[ 413.756711] [<ffffffff811d6730>] ? timerqueue_add+0x60/0xb0
[ 413.756719] [<ffffffff81056ab7>] ? __hrtimer_start_range_ns+0x1e7/0x410
[ 413.756726] [<ffffffff810c231b>] ? rw_copy_check_uvector+0x7b/0x140
[ 413.756734] [<ffffffff810c2bcf>] ? do_readv_writev+0xdf/0x210
[ 413.756742] [<ffffffff810c2e7e>] ? sys_writev+0x4e/0xc0
[ 413.756750] [<ffffffff813d753b>] ? system_call_fastpath+0x16/0x1b
[ 413.756756] FIX kmalloc-1024: Restoring
0xffff880417179566-0xffff880417179567=0x5a
[ 413.756760]
[ 556.640033]
=============================================================================
[ 556.640041] BUG kmalloc-512: Object padding overwritten
[ 556.640045]
-----------------------------------------------------------------------------
[ 556.640048]
[ 556.640053] INFO: 0xffff880403bf091e-0xffff880403bf091f. First byte
0x0 instead of 0x5a
[ 556.640069] INFO: Allocated in sock_alloc_send_pskb+0x1d0/0x320
age=21401 cpu=2 pid=5630
[ 556.640080] INFO: Freed in __kfree_skb+0x11/0x90 age=21386 cpu=2 pid=3753
[ 556.640086] INFO: Slab 0xffffea000e0d1c80 objects=28 used=10
fp=0xffff880403bf06d8 flags=0x80000000000040c1
[ 556.640092] INFO: Object 0xffff880403bf06d8 @offset=1752
fp=0xffff880403bf1488
[ 556.640096]
[ 556.640099] Bytes b4 0xffff880403bf06c8: 90 c4 ff ff 00 00 00 00 5a
5a 5a 5a 5a 5a 5a 5a .Äÿÿ....ZZZZZZZZ
[ 556.640115] Object 0xffff880403bf06d8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 556.640130] Object 0xffff880403bf06e8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 556.640144] Object 0xffff880403bf06f8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 556.640158] Object 0xffff880403bf0708: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 556.640172] Object 0xffff880403bf0718: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 556.640187] Object 0xffff880403bf0728: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 556.640201] Object 0xffff880403bf0738: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 556.640215] Object 0xffff880403bf0748: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 556.640229] Object 0xffff880403bf0758: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 556.640243] Object 0xffff880403bf0768: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 556.640257] Object 0xffff880403bf0778: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 556.640271] Object 0xffff880403bf0788: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 556.640286] Object 0xffff880403bf0798: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 556.640300] Object 0xffff880403bf07a8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 556.640314] Object 0xffff880403bf07b8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 556.640328] Object 0xffff880403bf07c8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 556.640343] Object 0xffff880403bf07d8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 556.640357] Object 0xffff880403bf07e8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 556.640371] Object 0xffff880403bf07f8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 556.640385] Object 0xffff880403bf0808: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 556.640399] Object 0xffff880403bf0818: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 556.640413] Object 0xffff880403bf0828: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 556.640428] Object 0xffff880403bf0838: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 556.640442] Object 0xffff880403bf0848: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 556.640456] Object 0xffff880403bf0858: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 556.640471] Object 0xffff880403bf0868: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 556.640485] Object 0xffff880403bf0878: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 556.640499] Object 0xffff880403bf0888: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 556.640513] Object 0xffff880403bf0898: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 556.640527] Object 0xffff880403bf08a8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 556.640542] Object 0xffff880403bf08b8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 556.640556] Object 0xffff880403bf08c8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b a5 kkkkkkkkkkkkkkk¥
[ 556.640570] Redzone 0xffff880403bf08d8: bb bb bb bb bb bb bb bb
»»»»»»»»
[ 556.640583] Padding 0xffff880403bf0918: 5a 5a 5a 5a 5a 5a 00 00
ZZZZZZ..
[ 556.640599] Pid: 4809, comm: qemu Not tainted 2.6.39 #2
[ 556.640603] Call Trace:
[ 556.640607] <IRQ> [<ffffffff810b7ccd>] ?
check_bytes_and_report+0x10d/0x150
[ 556.640626] [<ffffffff8131aae7>] ? __netdev_alloc_skb+0x17/0x40
[ 556.640632] [<ffffffff810b7db9>] ? check_object+0xa9/0x260
[ 556.640640] [<ffffffff8131aae7>] ? __netdev_alloc_skb+0x17/0x40
[ 556.640647] [<ffffffff810b86f4>] ? alloc_debug_processing+0x104/0x190
[ 556.640654] [<ffffffff810b9ac2>] ? T.912+0x272/0x2d0
[ 556.640661] [<ffffffff810bb70d>] ? __kmalloc_track_caller+0x10d/0x160
[ 556.640668] [<ffffffff813198c2>] ? __alloc_skb+0x72/0x160
[ 556.640676] [<ffffffff8131aae7>] ? __netdev_alloc_skb+0x17/0x40
[ 556.640684] [<ffffffff813a7b93>] ?
br_ip4_multicast_alloc_query+0x23/0x1c0
[ 556.640690] [<ffffffff813a7e76>] ? br_multicast_send_query+0x76/0x130
[ 556.640698] [<ffffffff8104dba8>] ? wq_worker_waking_up+0x8/0x30
[ 556.640706] [<ffffffff8102ca81>] ? sched_slice+0x51/0x80
[ 556.640711] [<ffffffff813a7fb5>] ?
br_multicast_port_query_expired+0x45/0x70
[ 556.640719] [<ffffffff81043f77>] ? run_timer_softirq+0x137/0x270
[ 556.640725] [<ffffffff81033fe9>] ? scheduler_tick+0x289/0x2d0
[ 556.640731] [<ffffffff813a7f70>] ? br_multicast_query_expired+0x40/0x40
[ 556.640740] [<ffffffff8103df11>] ? __do_softirq+0x91/0x120
[ 556.640747] [<ffffffff813d838c>] ? call_softirq+0x1c/0x30
[ 556.640755] [<ffffffff81003cbd>] ? do_softirq+0x4d/0x80
[ 556.640762] [<ffffffff8103ddbe>] ? irq_exit+0x8e/0xb0
[ 556.640770] [<ffffffff81019d1a>] ? smp_apic_timer_interrupt+0x6a/0xa0
[ 556.640776] [<ffffffff813d7e53>] ? apic_timer_interrupt+0x13/0x20
[ 556.640780] <EOI> [<ffffffff8102965a>] ?
flush_tlb_others_ipi+0x11a/0x130
[ 556.640791] [<ffffffff81029647>] ? flush_tlb_others_ipi+0x107/0x130
[ 556.640799] [<ffffffff810aadab>] ? ptep_clear_flush+0xb/0x10
[ 556.640806] [<ffffffff8109a40b>] ? do_wp_page+0x34b/0x7a0
[ 556.640836] [<ffffffffa00a188a>] ? kvm_read_guest_page+0x5a/0x70 [kvm]
[ 556.640863] [<ffffffffa00b7db4>] ?
paging64_walk_addr_generic+0x264/0x4c0 [kvm]
[ 556.640871] [<ffffffff8109acad>] ? handle_pte_fault+0x44d/0x990
[ 556.640878] [<ffffffff8109b4e8>] ? follow_page+0x268/0x440
[ 556.640886] [<ffffffff8109bd4d>] ? __get_user_pages+0x12d/0x530
[ 556.640892] [<ffffffff81028dbb>] ? gup_pud_range+0x12b/0x1b0
[ 556.640914] [<ffffffffa00a2d2e>] ? get_user_page_nowait+0x2e/0x40 [kvm]
[ 556.640935] [<ffffffffa00a2e7b>] ? hva_to_pfn+0x13b/0x440 [kvm]
[ 556.640956] [<ffffffffa00a31e9>] ? __gfn_to_pfn+0x39/0xf0 [kvm]
[ 556.640979] [<ffffffffa00b825d>] ? try_async_pf+0x4d/0x190 [kvm]
[ 556.641002] [<ffffffffa00b92be>] ? tdp_page_fault+0x10e/0x200 [kvm]
[ 556.641026] [<ffffffffa00b968c>] ? kvm_mmu_page_fault+0x1c/0x80 [kvm]
[ 556.641050] [<ffffffffa00b0a7a>] ?
kvm_arch_vcpu_ioctl_run+0x3fa/0xcf0 [kvm]
[ 556.641058] [<ffffffff8106239f>] ? futex_wake+0x10f/0x120
[ 556.641065] [<ffffffff81063f7b>] ? do_futex+0x11b/0xa70
[ 556.641086] [<ffffffffa00a517f>] ? kvm_vcpu_ioctl+0x4df/0x5e0 [kvm]
[ 556.641097] [<ffffffff81046671>] ? __dequeue_signal+0xe1/0x170
[ 556.641104] [<ffffffff8104821c>] ? do_send_sig_info+0x6c/0x90
[ 556.641112] [<ffffffff81046aac>] ? dequeue_signal+0x3c/0x170
[ 556.641119] [<ffffffff8104551f>] ? copy_siginfo_to_user+0xef/0x1d0
[ 556.641125] [<ffffffff810d26db>] ? do_vfs_ioctl+0x9b/0x4f0
[ 556.641132] [<ffffffff8106494a>] ? sys_futex+0x7a/0x180
[ 556.641137] [<ffffffff810d2b79>] ? sys_ioctl+0x49/0x80
[ 556.641145] [<ffffffff813d753b>] ? system_call_fastpath+0x16/0x1b
[ 556.641152] FIX kmalloc-512: Restoring
0xffff880403bf091e-0xffff880403bf091f=0x5a
[ 556.641155]
[ 602.610062] INFO: task ksmd:552 blocked for more than 120 seconds.
[ 602.610068] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[ 602.610073] ksmd D 0000000000000000 0 552 2
0x00000000
[ 602.610083] ffff88041db68ca0 0000000000000046 000212d01fc11d00
ffff880400000000
[ 602.610090] ffffffff81593020 ffff88041c1a9fd8 0000000000004000
ffff88041c1a8010
[ 602.610097] ffff88041c1a9fd8 ffff88041db68ca0 0000000000000000
0000000200000002
[ 602.610104] Call Trace:
[ 602.610121] [<ffffffff813d5595>] ? schedule_timeout+0x1c5/0x230
[ 602.610132] [<ffffffff8102f69c>] ? enqueue_task_fair+0x14c/0x190
[ 602.610141] [<ffffffff8102db27>] ? task_rq_lock+0x47/0x90
[ 602.610148] [<ffffffff813d4562>] ? wait_for_common+0xd2/0x180
[ 602.610154] [<ffffffff810339b0>] ? try_to_wake_up+0x2c0/0x2c0
[ 602.610162] [<ffffffff8104cb94>] ? flush_work+0x24/0x30
[ 602.610167] [<ffffffff8104be10>] ? do_work_for_cpu+0x20/0x20
[ 602.610174] [<ffffffff8104e0ab>] ? schedule_on_each_cpu+0xab/0xe0
[ 602.610181] [<ffffffff810b5c05>] ? ksm_scan_thread+0x7f5/0xc20
[ 602.610189] [<ffffffff81052a20>] ? wake_up_bit+0x40/0x40
[ 602.610194] [<ffffffff810b5410>] ?
try_to_merge_with_ksm_page+0x570/0x570
[ 602.610200] [<ffffffff810b5410>] ?
try_to_merge_with_ksm_page+0x570/0x570
[ 602.610207] [<ffffffff810525b6>] ? kthread+0x96/0xa0
[ 602.610214] [<ffffffff813d8294>] ? kernel_thread_helper+0x4/0x10
[ 602.610221] [<ffffffff81052520>] ? kthread_worker_fn+0x120/0x120
[ 602.610227] [<ffffffff813d8290>] ? gs_change+0xb/0xb
[ 602.610233] INFO: task fsnotify_mark:662 blocked for more than 120
seconds.
[ 602.610237] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[ 602.610241] fsnotify_mark D 0000000000000000 0 662 2
0x00000000
[ 602.610248] ffff88041c31d860 0000000000000046 0000000000000000
0000000000000000
[ 602.610255] ffffffff81593020 ffff88041c365fd8 0000000000004000
ffff88041c364010
[ 602.610262] ffff88041c365fd8 ffff88041c31d860 0000000000000000
0000000000000000
[ 602.610268] Call Trace:
[ 602.610275] [<ffffffff81035521>] ? load_balance+0x91/0x5e0
[ 602.610282] [<ffffffff813d5595>] ? schedule_timeout+0x1c5/0x230
[ 602.610289] [<ffffffff8102f323>] ? pick_next_task_fair+0x103/0x190
[ 602.610295] [<ffffffff813d4a6d>] ? schedule+0x28d/0x910
[ 602.610302] [<ffffffff813d4562>] ? wait_for_common+0xd2/0x180
[ 602.610307] [<ffffffff810339b0>] ? try_to_wake_up+0x2c0/0x2c0
[ 602.610314] [<ffffffff81074420>] ? synchronize_rcu_bh+0x50/0x50
[ 602.610320] [<ffffffff8107446a>] ? synchronize_sched+0x4a/0x50
[ 602.610326] [<ffffffff8104f940>] ? find_ge_pid+0x40/0x40
[ 602.610333] [<ffffffff8105742b>] ? __synchronize_srcu+0x5b/0xc0
[ 602.610342] [<ffffffff810f5f63>] ? fsnotify_mark_destroy+0x83/0x150
[ 602.610348] [<ffffffff81052a20>] ? wake_up_bit+0x40/0x40
[ 602.610356] [<ffffffff810f5ee0>] ?
fsnotify_set_mark_ignored_mask_locked+0x20/0x20
[ 602.610364] [<ffffffff810f5ee0>] ?
fsnotify_set_mark_ignored_mask_locked+0x20/0x20
[ 602.610371] [<ffffffff810525b6>] ? kthread+0x96/0xa0
[ 602.610377] [<ffffffff813d8294>] ? kernel_thread_helper+0x4/0x10
[ 602.610385] [<ffffffff81052520>] ? kthread_worker_fn+0x120/0x120
[ 602.610390] [<ffffffff813d8290>] ? gs_change+0xb/0xb
[ 644.689408]
=============================================================================
[ 644.689416] BUG kmalloc-512: Object padding overwritten
[ 644.689420]
-----------------------------------------------------------------------------
[ 644.689423]
[ 644.689428] INFO: 0xffff88041411f476-0xffff88041411f477. First byte
0x0 instead of 0x5a
[ 644.689444] INFO: Allocated in load_elf_binary+0xa6c/0x1c00 age=18648
cpu=3 pid=5915
[ 644.689454] INFO: Freed in load_elf_binary+0xab8/0x1c00 age=18648
cpu=3 pid=5915
[ 644.689461] INFO: Slab 0xffffea000e463e20 objects=28 used=10
fp=0xffff88041411f230 flags=0x80000000000040c1
[ 644.689467] INFO: Object 0xffff88041411f230 @offset=12848
fp=0xffff88041411eda0
[ 644.689470]
[ 644.689474] Bytes b4 0xffff88041411f220: 22 6f 00 00 01 00 00 00 5a
5a 5a 5a 5a 5a 5a 5a "o......ZZZZZZZZ
[ 644.689490] Object 0xffff88041411f230: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 644.689505] Object 0xffff88041411f240: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 644.689519] Object 0xffff88041411f250: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 644.689533] Object 0xffff88041411f260: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 644.689547] Object 0xffff88041411f270: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 644.689562] Object 0xffff88041411f280: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 644.689576] Object 0xffff88041411f290: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 644.689590] Object 0xffff88041411f2a0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 644.689604] Object 0xffff88041411f2b0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 644.689618] Object 0xffff88041411f2c0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 644.689632] Object 0xffff88041411f2d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 644.689647] Object 0xffff88041411f2e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 644.689661] Object 0xffff88041411f2f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 644.689675] Object 0xffff88041411f300: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 644.689689] Object 0xffff88041411f310: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 644.689703] Object 0xffff88041411f320: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 644.689717] Object 0xffff88041411f330: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 644.689732] Object 0xffff88041411f340: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 644.689746] Object 0xffff88041411f350: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 644.689760] Object 0xffff88041411f360: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 644.689774] Object 0xffff88041411f370: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 644.689788] Object 0xffff88041411f380: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 644.689802] Object 0xffff88041411f390: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 644.689817] Object 0xffff88041411f3a0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 644.689831] Object 0xffff88041411f3b0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 644.689845] Object 0xffff88041411f3c0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 644.689859] Object 0xffff88041411f3d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 644.689873] Object 0xffff88041411f3e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 644.689888] Object 0xffff88041411f3f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 644.689902] Object 0xffff88041411f400: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 644.689916] Object 0xffff88041411f410: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 644.689930] Object 0xffff88041411f420: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b a5 kkkkkkkkkkkkkkk¥
[ 644.689945] Redzone 0xffff88041411f430: bb bb bb bb bb bb bb bb
»»»»»»»»
[ 644.689957] Padding 0xffff88041411f470: 5a 5a 5a 5a 5a 5a 00 00
ZZZZZZ..
[ 644.689974] Pid: 6597, comm: awk Not tainted 2.6.39 #2
[ 644.689978] Call Trace:
[ 644.689989] [<ffffffff810b7ccd>] ? check_bytes_and_report+0x10d/0x150
[ 644.689998] [<ffffffff8110773c>] ? load_elf_binary+0xa6c/0x1c00
[ 644.690005] [<ffffffff810b7db9>] ? check_object+0xa9/0x260
[ 644.690012] [<ffffffff8110773c>] ? load_elf_binary+0xa6c/0x1c00
[ 644.690019] [<ffffffff810b86f4>] ? alloc_debug_processing+0x104/0x190
[ 644.690026] [<ffffffff810b9ac2>] ? T.912+0x272/0x2d0
[ 644.690032] [<ffffffff810ba59d>] ? __kmalloc+0x10d/0x160
[ 644.690039] [<ffffffff8110773c>] ? load_elf_binary+0xa6c/0x1c00
[ 644.690047] [<ffffffff8109bd9d>] ? __get_user_pages+0x17d/0x530
[ 644.690055] [<ffffffff810c9556>] ? get_arg_page+0x56/0x100
[ 644.690062] [<ffffffff810c8070>] ? search_binary_handler+0x90/0x240
[ 644.690069] [<ffffffff810c9e9f>] ? do_execve+0x22f/0x2f0
[ 644.690076] [<ffffffff810094a6>] ? sys_execve+0x36/0x60
[ 644.690085] [<ffffffff813d78dc>] ? stub_execve+0x6c/0xc0
[ 644.690092] FIX kmalloc-512: Restoring
0xffff88041411f476-0xffff88041411f477=0x5a
[ 644.690095]
[ 704.979481]
=============================================================================
[ 704.979488] BUG kmalloc-512: Object padding overwritten
[ 704.979492]
-----------------------------------------------------------------------------
[ 704.979496]
[ 704.979501] INFO: 0xffff8804168c1fee-0xffff8804168c1fef. First byte
0x0 instead of 0x5a
[ 704.979517] INFO: Allocated in load_elf_binary+0xa6c/0x1c00 age=24184
cpu=0 pid=5999
[ 704.979527] INFO: Freed in load_elf_binary+0xab8/0x1c00 age=24184
cpu=0 pid=5999
[ 704.979534] INFO: Slab 0xffffea000e4eea00 objects=28 used=10
fp=0xffff8804168c1da8 flags=0x80000000000040c1
[ 704.979540] INFO: Object 0xffff8804168c1da8 @offset=7592
fp=0xffff8804168c2910
[ 704.979543]
[ 704.979547] Bytes b4 0xffff8804168c1d98: 93 98 ff ff 00 00 00 00 5a
5a 5a 5a 5a 5a 5a 5a ..ÿÿ....ZZZZZZZZ
[ 704.979563] Object 0xffff8804168c1da8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 704.979578] Object 0xffff8804168c1db8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 704.979592] Object 0xffff8804168c1dc8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 704.979606] Object 0xffff8804168c1dd8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 704.979620] Object 0xffff8804168c1de8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 704.979634] Object 0xffff8804168c1df8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 704.979648] Object 0xffff8804168c1e08: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 704.979662] Object 0xffff8804168c1e18: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 704.979676] Object 0xffff8804168c1e28: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 704.979690] Object 0xffff8804168c1e38: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 704.979704] Object 0xffff8804168c1e48: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 704.979719] Object 0xffff8804168c1e58: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 704.979733] Object 0xffff8804168c1e68: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 704.979747] Object 0xffff8804168c1e78: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 704.979761] Object 0xffff8804168c1e88: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 704.979775] Object 0xffff8804168c1e98: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 704.979789] Object 0xffff8804168c1ea8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 704.979803] Object 0xffff8804168c1eb8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 704.979817] Object 0xffff8804168c1ec8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 704.979831] Object 0xffff8804168c1ed8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 704.979845] Object 0xffff8804168c1ee8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 704.979859] Object 0xffff8804168c1ef8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 704.979873] Object 0xffff8804168c1f08: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 704.979888] Object 0xffff8804168c1f18: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 704.979902] Object 0xffff8804168c1f28: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 704.979916] Object 0xffff8804168c1f38: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 704.979930] Object 0xffff8804168c1f48: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 704.979944] Object 0xffff8804168c1f58: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 704.979958] Object 0xffff8804168c1f68: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 704.979972] Object 0xffff8804168c1f78: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 704.979986] Object 0xffff8804168c1f88: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 704.980000] Object 0xffff8804168c1f98: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b a5 kkkkkkkkkkkkkkk¥
[ 704.980015] Redzone 0xffff8804168c1fa8: bb bb bb bb bb bb bb bb
»»»»»»»»
[ 704.980028] Padding 0xffff8804168c1fe8: 5a 5a 5a 5a 5a 5a 00 00
ZZZZZZ..
[ 704.980044] Pid: 6812, comm: get-rrdtool-dat Not tainted 2.6.39 #2
[ 704.980048] Call Trace:
[ 704.980060] [<ffffffff810b7ccd>] ? check_bytes_and_report+0x10d/0x150
[ 704.980069] [<ffffffff81106e49>] ? load_elf_binary+0x179/0x1c00
[ 704.980075] [<ffffffff810b7db9>] ? check_object+0xa9/0x260
[ 704.980083] [<ffffffff81106e49>] ? load_elf_binary+0x179/0x1c00
[ 704.980090] [<ffffffff810b86f4>] ? alloc_debug_processing+0x104/0x190
[ 704.980096] [<ffffffff810b9ac2>] ? T.912+0x272/0x2d0
[ 704.980102] [<ffffffff810ba59d>] ? __kmalloc+0x10d/0x160
[ 704.980109] [<ffffffff81106e49>] ? load_elf_binary+0x179/0x1c00
[ 704.980117] [<ffffffff8109b782>] ? __pte_alloc+0x42/0x130
[ 704.980123] [<ffffffff8109b5a5>] ? follow_page+0x325/0x440
[ 704.980130] [<ffffffff8109b5a5>] ? follow_page+0x325/0x440
[ 704.980137] [<ffffffff8109bd9d>] ? __get_user_pages+0x17d/0x530
[ 704.980145] [<ffffffff810c9556>] ? get_arg_page+0x56/0x100
[ 704.980152] [<ffffffff810c8070>] ? search_binary_handler+0x90/0x240
[ 704.980158] [<ffffffff810c9e9f>] ? do_execve+0x22f/0x2f0
[ 704.980166] [<ffffffff810094a6>] ? sys_execve+0x36/0x60
[ 704.980175] [<ffffffff813d78dc>] ? stub_execve+0x6c/0xc0
[ 704.980182] FIX kmalloc-512: Restoring
0xffff8804168c1fee-0xffff8804168c1fef=0x5a
[ 704.980185]
[ 722.610058] INFO: task ksmd:552 blocked for more than 120 seconds.
[ 722.610063] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[ 722.610069] ksmd D 0000000000000000 0 552 2
0x00000000
[ 722.610078] ffff88041db68ca0 0000000000000046 000212d01fc11d00
ffff880400000000
[ 722.610085] ffffffff81593020 ffff88041c1a9fd8 0000000000004000
ffff88041c1a8010
[ 722.610092] ffff88041c1a9fd8 ffff88041db68ca0 0000000000000000
0000000200000002
[ 722.610099] Call Trace:
[ 722.610115] [<ffffffff813d5595>] ? schedule_timeout+0x1c5/0x230
[ 722.610127] [<ffffffff8102f69c>] ? enqueue_task_fair+0x14c/0x190
[ 722.610135] [<ffffffff8102db27>] ? task_rq_lock+0x47/0x90
[ 722.610142] [<ffffffff813d4562>] ? wait_for_common+0xd2/0x180
[ 722.610148] [<ffffffff810339b0>] ? try_to_wake_up+0x2c0/0x2c0
[ 722.610156] [<ffffffff8104cb94>] ? flush_work+0x24/0x30
[ 722.610162] [<ffffffff8104be10>] ? do_work_for_cpu+0x20/0x20
[ 722.610168] [<ffffffff8104e0ab>] ? schedule_on_each_cpu+0xab/0xe0
[ 722.610175] [<ffffffff810b5c05>] ? ksm_scan_thread+0x7f5/0xc20
[ 722.610182] [<ffffffff81052a20>] ? wake_up_bit+0x40/0x40
[ 722.610188] [<ffffffff810b5410>] ?
try_to_merge_with_ksm_page+0x570/0x570
[ 722.610194] [<ffffffff810b5410>] ?
try_to_merge_with_ksm_page+0x570/0x570
[ 722.610201] [<ffffffff810525b6>] ? kthread+0x96/0xa0
[ 722.610208] [<ffffffff813d8294>] ? kernel_thread_helper+0x4/0x10
[ 722.610215] [<ffffffff81052520>] ? kthread_worker_fn+0x120/0x120
[ 722.610221] [<ffffffff813d8290>] ? gs_change+0xb/0xb
[ 722.610226] INFO: task fsnotify_mark:662 blocked for more than 120
seconds.
[ 722.610230] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[ 722.610235] fsnotify_mark D 0000000000000000 0 662 2
0x00000000
[ 722.610241] ffff88041c31d860 0000000000000046 0000000000000000
0000000000000000
[ 722.610248] ffffffff81593020 ffff88041c365fd8 0000000000004000
ffff88041c364010
[ 722.610255] ffff88041c365fd8 ffff88041c31d860 0000000000000000
0000000000000000
[ 722.610261] Call Trace:
[ 722.610268] [<ffffffff81035521>] ? load_balance+0x91/0x5e0
[ 722.610275] [<ffffffff813d5595>] ? schedule_timeout+0x1c5/0x230
[ 722.610282] [<ffffffff8102f323>] ? pick_next_task_fair+0x103/0x190
[ 722.610289] [<ffffffff813d4a6d>] ? schedule+0x28d/0x910
[ 722.610295] [<ffffffff813d4562>] ? wait_for_common+0xd2/0x180
[ 722.610301] [<ffffffff810339b0>] ? try_to_wake_up+0x2c0/0x2c0
[ 722.610308] [<ffffffff81074420>] ? synchronize_rcu_bh+0x50/0x50
[ 722.610313] [<ffffffff8107446a>] ? synchronize_sched+0x4a/0x50
[ 722.610319] [<ffffffff8104f940>] ? find_ge_pid+0x40/0x40
[ 722.610326] [<ffffffff8105742b>] ? __synchronize_srcu+0x5b/0xc0
[ 722.610335] [<ffffffff810f5f63>] ? fsnotify_mark_destroy+0x83/0x150
[ 722.610342] [<ffffffff81052a20>] ? wake_up_bit+0x40/0x40
[ 722.610350] [<ffffffff810f5ee0>] ?
fsnotify_set_mark_ignored_mask_locked+0x20/0x20
[ 722.610358] [<ffffffff810f5ee0>] ?
fsnotify_set_mark_ignored_mask_locked+0x20/0x20
[ 722.610365] [<ffffffff810525b6>] ? kthread+0x96/0xa0
[ 722.610371] [<ffffffff813d8294>] ? kernel_thread_helper+0x4/0x10
[ 722.610378] [<ffffffff81052520>] ? kthread_worker_fn+0x120/0x120
[ 722.610384] [<ffffffff813d8290>] ? gs_change+0xb/0xb
[ 722.610392] INFO: task jbd2/md0-8:2528 blocked for more than 120 seconds.
[ 722.610396] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[ 722.610400] jbd2/md0-8 D 0000000000000005 0 2528 2
0x00000000
[ 722.610407] ffff88041d91c570 0000000000000046 ffff88041baeea28
ffffea0000000000
[ 722.610413] ffff88041d91a5e0 ffff880419f9bfd8 0000000000004000
ffff880419f9a010
[ 722.610420] ffff880419f9bfd8 ffff88041d91c570 ffff88041baeea28
ffffffff810b8495
[ 722.610426] Call Trace:
[ 722.610432] [<ffffffff810b8495>] ? init_object+0x85/0xa0
[ 722.610438] [<ffffffff810b8916>] ? free_debug_processing+0x196/0x250
[ 722.610447] [<ffffffff8105adae>] ? ktime_get_ts+0x6e/0xf0
[ 722.610455] [<ffffffff810810f0>] ? __lock_page+0x70/0x70
[ 722.610461] [<ffffffff813d5174>] ? io_schedule+0x84/0xd0
[ 722.610469] [<ffffffff811d4403>] ?
radix_tree_gang_lookup_tag_slot+0x93/0xf0
[ 722.610476] [<ffffffff810810f9>] ? sleep_on_page+0x9/0x10
[ 722.610482] [<ffffffff813d57df>] ? __wait_on_bit+0x4f/0x80
[ 722.610489] [<ffffffff810812eb>] ? wait_on_page_bit+0x6b/0x80
[ 722.610496] [<ffffffff81052a50>] ? autoremove_wake_function+0x30/0x30
[ 722.610504] [<ffffffff8108a458>] ? pagevec_lookup_tag+0x18/0x20
[ 722.610509] [<ffffffff81081f2a>] ? filemap_fdatawait_range+0xfa/0x180
[ 722.610518] [<ffffffff811be09f>] ? submit_bio+0x6f/0xf0
[ 722.610526] [<ffffffff81176276>] ?
jbd2_journal_commit_transaction+0x796/0x1270
[ 722.610536] [<ffffffff81179ed1>] ? kjournald2+0xb1/0x1e0
[ 722.610542] [<ffffffff81052a20>] ? wake_up_bit+0x40/0x40
[ 722.610549] [<ffffffff81179e20>] ? commit_timeout+0x10/0x10
[ 722.610556] [<ffffffff81179e20>] ? commit_timeout+0x10/0x10
[ 722.610563] [<ffffffff810525b6>] ? kthread+0x96/0xa0
[ 722.610569] [<ffffffff813d8294>] ? kernel_thread_helper+0x4/0x10
[ 722.610576] [<ffffffff81052520>] ? kthread_worker_fn+0x120/0x120
[ 722.610582] [<ffffffff813d8290>] ? gs_change+0xb/0xb
[ 722.610591] INFO: task nfsd:4326 blocked for more than 120 seconds.
[ 722.610594] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[ 722.610598] nfsd D 0000000000000000 0 4326 2
0x00000000
[ 722.610605] ffff88041b604bc0 0000000000000046 0000000000000016
ffffffff00000000
[ 722.610611] ffff88041d8fa5e0 ffff88041c527fd8 0000000000004000
ffff88041c526010
[ 722.610617] ffff88041c527fd8 ffff88041b604bc0 000000000000009c
ffff880000000000
[ 722.610623] Call Trace:
[ 722.610630] [<ffffffff8101e845>] ? amd_flush_garts+0x105/0x140
[ 722.610637] [<ffffffff8101fa80>] ? gart_map_sg+0x480/0x480
[ 722.610643] [<ffffffff8101f5d3>] ? flush_gart+0x23/0x50
[ 722.610650] [<ffffffff81080f98>] ? find_get_page+0x18/0x90
[ 722.610657] [<ffffffff81174b95>] ? do_get_write_access+0x265/0x4a0
[ 722.610665] [<ffffffff81052a50>] ? autoremove_wake_function+0x30/0x30
[ 722.610672] [<ffffffff81174ef9>] ?
jbd2_journal_get_write_access+0x29/0x50
[ 722.610680] [<ffffffff8115f122>] ?
__ext4_journal_get_write_access+0x32/0x80
[ 722.610689] [<ffffffff81143908>] ? ext4_reserve_inode_write+0x78/0xa0
[ 722.610696] [<ffffffff81143970>] ? ext4_mark_inode_dirty+0x40/0x1e0
[ 722.610703] [<ffffffff81156c0b>] ? ext4_journal_start_sb+0x6b/0x160
[ 722.610711] [<ffffffff81322645>] ? dev_hard_start_xmit+0x305/0x5f0
[ 722.610719] [<ffffffff81352fd0>] ? ip_finish_output2+0x290/0x290
[ 722.610726] [<ffffffff81143c65>] ? ext4_dirty_inode+0x35/0x70
[ 722.610733] [<ffffffff810e4a08>] ? __mark_inode_dirty+0x38/0x210
[ 722.610741] [<ffffffff810d9317>] ? file_update_time+0xf7/0x180
[ 722.610747] [<ffffffff81082458>] ? __generic_file_aio_write+0x1f8/0x430
[ 722.610755] [<ffffffff81373ef9>] ? udp_sendmsg+0x3c9/0x7e0
[ 722.610763] [<ffffffff81314260>] ? sock_alloc_send_pskb+0x1d0/0x320
[ 722.610769] [<ffffffff81082703>] ? generic_file_aio_write+0x73/0xf0
[ 722.610776] [<ffffffff8113f74e>] ? ext4_file_write+0x6e/0x2b0
[ 722.610783] [<ffffffff810da8ac>] ? iget_locked+0x4c/0x140
[ 722.610789] [<ffffffff8119a590>] ? fh_compose+0x4c0/0x4c0
[ 722.610795] [<ffffffff8113f6e0>] ? ext4_llseek+0x110/0x110
[ 722.610803] [<ffffffff810c24e9>] ? do_sync_readv_writev+0xa9/0xf0
[ 722.610811] [<ffffffff810c231b>] ? rw_copy_check_uvector+0x7b/0x140
[ 722.610819] [<ffffffff810c2bcf>] ? do_readv_writev+0xdf/0x210
[ 722.610825] [<ffffffff810b8735>] ? alloc_debug_processing+0x145/0x190
[ 722.610831] [<ffffffff810b9bc5>] ? kmem_cache_alloc+0xa5/0xb0
[ 722.610838] [<ffffffff8113f463>] ? ext4_file_open+0x63/0x180
[ 722.610844] [<ffffffff8119c21d>] ? nfsd_vfs_write+0xed/0x3a0
[ 722.610851] [<ffffffff810c1127>] ? __dentry_open+0x1f7/0x2b0
[ 722.610857] [<ffffffff8119c892>] ? nfsd_open+0xf2/0x1b0
[ 722.610862] [<ffffffff8119cd34>] ? nfsd_write+0xf4/0x110
[ 722.610868] [<ffffffff81199930>] ? nfsd_proc_write+0xb0/0x120
[ 722.610876] [<ffffffff811971c5>] ? nfsd_dispatch+0xf5/0x230
[ 722.610882] [<ffffffff813b456f>] ? svc_process+0x4af/0x820
[ 722.610887] [<ffffffff810339b0>] ? try_to_wake_up+0x2c0/0x2c0
[ 722.610894] [<ffffffff811977a0>] ? nfsd_svc+0x1b0/0x1b0
[ 722.610901] [<ffffffff8119784d>] ? nfsd+0xad/0x150
[ 722.610907] [<ffffffff810525b6>] ? kthread+0x96/0xa0
[ 722.610913] [<ffffffff813d8294>] ? kernel_thread_helper+0x4/0x10
[ 722.610921] [<ffffffff81052520>] ? kthread_worker_fn+0x120/0x120
[ 722.610926] [<ffffffff813d8290>] ? gs_change+0xb/0xb
[ 765.241265]
=============================================================================
[ 765.241267] BUG kmalloc-512: Object padding overwritten
[ 765.241268]
-----------------------------------------------------------------------------
[ 765.241269]
[ 765.241271] INFO: 0xffff8804168c1486-0xffff8804168c1487. First byte
0x0 instead of 0x5a
[ 765.241277] INFO: Allocated in bio_kmalloc+0x2d/0x70 age=73010 cpu=5
pid=2495
[ 765.241281] INFO: Freed in r10buf_pool_free+0x71/0x90 age=72999 cpu=5
pid=1728
[ 765.241283] INFO: Slab 0xffffea000e4eea00 objects=28 used=13
fp=0xffff8804168c1240 flags=0x80000000000040c1
[ 765.241284] INFO: Object 0xffff8804168c1240 @offset=4672
fp=0xffff8804168c0ff8
[ 765.241285]
[ 765.241286] Bytes b4 0xffff8804168c1230: 95 98 ff ff 00 00 00 00 5a
5a 5a 5a 5a 5a 5a 5a ..ÿÿ....ZZZZZZZZ
[ 765.241291] Object 0xffff8804168c1240: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 765.241294] Object 0xffff8804168c1250: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 765.241298] Object 0xffff8804168c1260: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 765.241301] Object 0xffff8804168c1270: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 765.241305] Object 0xffff8804168c1280: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 765.241308] Object 0xffff8804168c1290: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 765.241312] Object 0xffff8804168c12a0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 765.241315] Object 0xffff8804168c12b0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 765.241319] Object 0xffff8804168c12c0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 765.241322] Object 0xffff8804168c12d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 765.241326] Object 0xffff8804168c12e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 765.241329] Object 0xffff8804168c12f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 765.241332] Object 0xffff8804168c1300: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 765.241336] Object 0xffff8804168c1310: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 765.241339] Object 0xffff8804168c1320: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 765.241343] Object 0xffff8804168c1330: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 765.241346] Object 0xffff8804168c1340: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 765.241350] Object 0xffff8804168c1350: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 765.241353] Object 0xffff8804168c1360: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 765.241357] Object 0xffff8804168c1370: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 765.241360] Object 0xffff8804168c1380: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 765.241364] Object 0xffff8804168c1390: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 765.241367] Object 0xffff8804168c13a0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 765.241371] Object 0xffff8804168c13b0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 765.241374] Object 0xffff8804168c13c0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 765.241378] Object 0xffff8804168c13d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 765.241381] Object 0xffff8804168c13e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 765.241385] Object 0xffff8804168c13f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 765.241388] Object 0xffff8804168c1400: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 765.241392] Object 0xffff8804168c1410: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 765.241395] Object 0xffff8804168c1420: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 765.241399] Object 0xffff8804168c1430: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b a5 kkkkkkkkkkkkkkk¥
[ 765.241402] Redzone 0xffff8804168c1440: bb bb bb bb bb bb bb bb
»»»»»»»»
[ 765.241406] Padding 0xffff8804168c1480: 5a 5a 5a 5a 5a 5a 00 00
ZZZZZZ..
[ 765.241410] Pid: 7147, comm: apcaccess Not tainted 2.6.39 #2
[ 765.241411] Call Trace:
[ 765.241416] [<ffffffff810b7ccd>] ? check_bytes_and_report+0x10d/0x150
[ 765.241419] [<ffffffff8110773c>] ? load_elf_binary+0xa6c/0x1c00
[ 765.241421] [<ffffffff810b7db9>] ? check_object+0xa9/0x260
[ 765.241423] [<ffffffff8110773c>] ? load_elf_binary+0xa6c/0x1c00
[ 765.241424] [<ffffffff810b86f4>] ? alloc_debug_processing+0x104/0x190
[ 765.241426] [<ffffffff810b9ac2>] ? T.912+0x272/0x2d0
[ 765.241428] [<ffffffff810ba59d>] ? __kmalloc+0x10d/0x160
[ 765.241429] [<ffffffff8110773c>] ? load_elf_binary+0xa6c/0x1c00
[ 765.241432] [<ffffffff8109bd9d>] ? __get_user_pages+0x17d/0x530
[ 765.241434] [<ffffffff810c9556>] ? get_arg_page+0x56/0x100
[ 765.241437] [<ffffffff811da4ad>] ? strnlen_user+0x2d/0x80
[ 765.241439] [<ffffffff810c8070>] ? search_binary_handler+0x90/0x240
[ 765.241440] [<ffffffff810c9e9f>] ? do_execve+0x22f/0x2f0
[ 765.241443] [<ffffffff810094a6>] ? sys_execve+0x36/0x60
[ 765.241446] [<ffffffff813d78dc>] ? stub_execve+0x6c/0xc0
[ 765.241448] FIX kmalloc-512: Restoring
0xffff8804168c1486-0xffff8804168c1487=0x5a
[ 765.241449]
[ 789.536436]
=============================================================================
[ 789.536444] BUG kmalloc-2048: Object padding overwritten
[ 789.536448]
-----------------------------------------------------------------------------
[ 789.536451]
[ 789.536456] INFO: 0xffff8802c8b1a11e-0xffff8802c8b1a11f. First byte
0x0 instead of 0x5a
[ 789.536471] INFO: Allocated in sk_stream_alloc_skb+0x3a/0x110 age=63
cpu=3 pid=5169
[ 789.536482] INFO: Freed in __kfree_skb+0x11/0x90 age=63 cpu=5 pid=0
[ 789.536488] INFO: Slab 0xffffea0009be6d40 objects=15 used=4
fp=0xffff8802c8b198d8 flags=0x80000000000040c1
[ 789.536494] INFO: Object 0xffff8802c8b198d8 @offset=6360
fp=0xffff8802c8b1a968
[ 789.536497]
[ 789.536501] Bytes b4 0xffff8802c8b198c8: cc be 00 00 01 00 00 00 5a
5a 5a 5a 5a 5a 5a 5a ÌŸ......ZZZZZZZZ
[ 789.536517] Object 0xffff8802c8b198d8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.536532] Object 0xffff8802c8b198e8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.536546] Object 0xffff8802c8b198f8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.536560] Object 0xffff8802c8b19908: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.536575] Object 0xffff8802c8b19918: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.536589] Object 0xffff8802c8b19928: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.536603] Object 0xffff8802c8b19938: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.536617] Object 0xffff8802c8b19948: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.536631] Object 0xffff8802c8b19958: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.536645] Object 0xffff8802c8b19968: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.536660] Object 0xffff8802c8b19978: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.536674] Object 0xffff8802c8b19988: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.536688] Object 0xffff8802c8b19998: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.536702] Object 0xffff8802c8b199a8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.536716] Object 0xffff8802c8b199b8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.536730] Object 0xffff8802c8b199c8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.536745] Object 0xffff8802c8b199d8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.536759] Object 0xffff8802c8b199e8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.536773] Object 0xffff8802c8b199f8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.536787] Object 0xffff8802c8b19a08: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.536801] Object 0xffff8802c8b19a18: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.536816] Object 0xffff8802c8b19a28: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.536830] Object 0xffff8802c8b19a38: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.536844] Object 0xffff8802c8b19a48: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.536858] Object 0xffff8802c8b19a58: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.536872] Object 0xffff8802c8b19a68: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.536887] Object 0xffff8802c8b19a78: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.536901] Object 0xffff8802c8b19a88: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.536915] Object 0xffff8802c8b19a98: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.536929] Object 0xffff8802c8b19aa8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.536943] Object 0xffff8802c8b19ab8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.536958] Object 0xffff8802c8b19ac8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.536972] Object 0xffff8802c8b19ad8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.536986] Object 0xffff8802c8b19ae8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537000] Object 0xffff8802c8b19af8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537014] Object 0xffff8802c8b19b08: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537029] Object 0xffff8802c8b19b18: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537043] Object 0xffff8802c8b19b28: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537057] Object 0xffff8802c8b19b38: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537071] Object 0xffff8802c8b19b48: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537085] Object 0xffff8802c8b19b58: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537100] Object 0xffff8802c8b19b68: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537114] Object 0xffff8802c8b19b78: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537128] Object 0xffff8802c8b19b88: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537143] Object 0xffff8802c8b19b98: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537158] Object 0xffff8802c8b19ba8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537172] Object 0xffff8802c8b19bb8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537187] Object 0xffff8802c8b19bc8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537201] Object 0xffff8802c8b19bd8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537215] Object 0xffff8802c8b19be8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537229] Object 0xffff8802c8b19bf8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537244] Object 0xffff8802c8b19c08: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537258] Object 0xffff8802c8b19c18: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537272] Object 0xffff8802c8b19c28: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537286] Object 0xffff8802c8b19c38: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537301] Object 0xffff8802c8b19c48: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537315] Object 0xffff8802c8b19c58: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537329] Object 0xffff8802c8b19c68: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537343] Object 0xffff8802c8b19c78: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537357] Object 0xffff8802c8b19c88: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537372] Object 0xffff8802c8b19c98: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537386] Object 0xffff8802c8b19ca8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537400] Object 0xffff8802c8b19cb8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537414] Object 0xffff8802c8b19cc8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537428] Object 0xffff8802c8b19cd8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537443] Object 0xffff8802c8b19ce8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537457] Object 0xffff8802c8b19cf8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537471] Object 0xffff8802c8b19d08: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537485] Object 0xffff8802c8b19d18: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537500] Object 0xffff8802c8b19d28: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537514] Object 0xffff8802c8b19d38: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537528] Object 0xffff8802c8b19d48: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537542] Object 0xffff8802c8b19d58: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537557] Object 0xffff8802c8b19d68: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537571] Object 0xffff8802c8b19d78: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537585] Object 0xffff8802c8b19d88: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537599] Object 0xffff8802c8b19d98: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537613] Object 0xffff8802c8b19da8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537628] Object 0xffff8802c8b19db8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537642] Object 0xffff8802c8b19dc8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537656] Object 0xffff8802c8b19dd8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537670] Object 0xffff8802c8b19de8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537685] Object 0xffff8802c8b19df8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537699] Object 0xffff8802c8b19e08: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537713] Object 0xffff8802c8b19e18: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537727] Object 0xffff8802c8b19e28: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537742] Object 0xffff8802c8b19e38: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537756] Object 0xffff8802c8b19e48: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537770] Object 0xffff8802c8b19e58: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537784] Object 0xffff8802c8b19e68: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537799] Object 0xffff8802c8b19e78: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537813] Object 0xffff8802c8b19e88: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537827] Object 0xffff8802c8b19e98: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537841] Object 0xffff8802c8b19ea8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537856] Object 0xffff8802c8b19eb8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537870] Object 0xffff8802c8b19ec8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537884] Object 0xffff8802c8b19ed8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537898] Object 0xffff8802c8b19ee8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537912] Object 0xffff8802c8b19ef8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537927] Object 0xffff8802c8b19f08: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537941] Object 0xffff8802c8b19f18: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537955] Object 0xffff8802c8b19f28: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537969] Object 0xffff8802c8b19f38: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537984] Object 0xffff8802c8b19f48: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537998] Object 0xffff8802c8b19f58: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.538012] Object 0xffff8802c8b19f68: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.538026] Object 0xffff8802c8b19f78: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.538041] Object 0xffff8802c8b19f88: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.538055] Object 0xffff8802c8b19f98: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.538069] Object 0xffff8802c8b19fa8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.538083] Object 0xffff8802c8b19fb8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.538097] Object 0xffff8802c8b19fc8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.538112] Object 0xffff8802c8b19fd8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.538126] Object 0xffff8802c8b19fe8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.538140] Object 0xffff8802c8b19ff8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.538154] Object 0xffff8802c8b1a008: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.538169] Object 0xffff8802c8b1a018: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.538183] Object 0xffff8802c8b1a028: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.538197] Object 0xffff8802c8b1a038: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.538211] Object 0xffff8802c8b1a048: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.538226] Object 0xffff8802c8b1a058: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.538240] Object 0xffff8802c8b1a068: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.538254] Object 0xffff8802c8b1a078: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.538268] Object 0xffff8802c8b1a088: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.538283] Object 0xffff8802c8b1a098: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.538297] Object 0xffff8802c8b1a0a8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.538311] Object 0xffff8802c8b1a0b8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.538325] Object 0xffff8802c8b1a0c8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b a5 kkkkkkkkkkkkkkk¥
[ 789.538340] Redzone 0xffff8802c8b1a0d8: bb bb bb bb bb bb bb bb
»»»»»»»»
[ 789.538353] Padding 0xffff8802c8b1a118: 5a 5a 5a 5a 5a 5a 00 00
ZZZZZZ..
[ 789.538369] Pid: 5247, comm: qemu Not tainted 2.6.39 #2
[ 789.538373] Call Trace:
[ 789.538385] [<ffffffff810b7ccd>] ? check_bytes_and_report+0x10d/0x150
[ 789.538394] [<ffffffff81314260>] ? sock_alloc_send_pskb+0x1d0/0x320
[ 789.538400] [<ffffffff810b7db9>] ? check_object+0xa9/0x260
[ 789.538407] [<ffffffff81314260>] ? sock_alloc_send_pskb+0x1d0/0x320
[ 789.538414] [<ffffffff810b86f4>] ? alloc_debug_processing+0x104/0x190
[ 789.538420] [<ffffffff810b9ac2>] ? T.912+0x272/0x2d0
[ 789.538427] [<ffffffff810bb70d>] ? __kmalloc_track_caller+0x10d/0x160
[ 789.538434] [<ffffffff813198c2>] ? __alloc_skb+0x72/0x160
[ 789.538441] [<ffffffff81314260>] ? sock_alloc_send_pskb+0x1d0/0x320
[ 789.538450] [<ffffffff8105af7b>] ? getnstimeofday+0x4b/0xd0
[ 789.538457] [<ffffffff81320d32>] ? netif_rx+0xb2/0x120
[ 789.538465] [<ffffffff812b393e>] ? tun_chr_aio_write+0x18e/0x4e0
[ 789.538472] [<ffffffff812b37b0>] ? tun_sendmsg+0x4d0/0x4d0
[ 789.538481] [<ffffffff810c24e9>] ? do_sync_readv_writev+0xa9/0xf0
[ 789.538489] [<ffffffff810c231b>] ? rw_copy_check_uvector+0x7b/0x140
[ 789.538496] [<ffffffff810c2bcf>] ? do_readv_writev+0xdf/0x210
[ 789.538504] [<ffffffff810c2e7e>] ? sys_writev+0x4e/0xc0
[ 789.538513] [<ffffffff813d753b>] ? system_call_fastpath+0x16/0x1b
[ 789.538519] FIX kmalloc-2048: Restoring
0xffff8802c8b1a11e-0xffff8802c8b1a11f=0x5a
[ 789.538522]
[ 825.601291]
=============================================================================
[ 825.601298] BUG kmalloc-512: Object padding overwritten
[ 825.601302]
-----------------------------------------------------------------------------
[ 825.601306]
[ 825.601311] INFO: 0xffff88041411efe6-0xffff88041411efe7. First byte
0x0 instead of 0x5a
[ 825.601327] INFO: Allocated in sock_alloc_send_pskb+0x1d0/0x320
age=36742 cpu=3 pid=4846
[ 825.601338] INFO: Freed in __kfree_skb+0x11/0x90 age=36742 cpu=3 pid=3781
[ 825.601344] INFO: Slab 0xffffea000e463e20 objects=28 used=11
fp=0xffff88041411eda0 flags=0x80000000000040c1
[ 825.601350] INFO: Object 0xffff88041411eda0 @offset=11680
fp=0xffff88041411cdb0
[ 825.601353]
[ 825.601357] Bytes b4 0xffff88041411ed90: e2 96 ff ff 00 00 00 00 5a
5a 5a 5a 5a 5a 5a 5a â.ÿÿ....ZZZZZZZZ
[ 825.601373] Object 0xffff88041411eda0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 825.601388] Object 0xffff88041411edb0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 825.601402] Object 0xffff88041411edc0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 825.601416] Object 0xffff88041411edd0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 825.601430] Object 0xffff88041411ede0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 825.601444] Object 0xffff88041411edf0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 825.601459] Object 0xffff88041411ee00: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 825.601473] Object 0xffff88041411ee10: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 825.601487] Object 0xffff88041411ee20: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 825.601501] Object 0xffff88041411ee30: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 825.601515] Object 0xffff88041411ee40: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 825.601529] Object 0xffff88041411ee50: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 825.601543] Object 0xffff88041411ee60: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 825.601557] Object 0xffff88041411ee70: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 825.601571] Object 0xffff88041411ee80: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 825.601586] Object 0xffff88041411ee90: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 825.601600] Object 0xffff88041411eea0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 825.601614] Object 0xffff88041411eeb0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 825.601628] Object 0xffff88041411eec0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 825.601642] Object 0xffff88041411eed0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 825.601656] Object 0xffff88041411eee0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 825.601670] Object 0xffff88041411eef0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 825.601684] Object 0xffff88041411ef00: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 825.601699] Object 0xffff88041411ef10: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 825.601713] Object 0xffff88041411ef20: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 825.601727] Object 0xffff88041411ef30: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 825.601741] Object 0xffff88041411ef40: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 825.601755] Object 0xffff88041411ef50: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 825.601769] Object 0xffff88041411ef60: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 825.601783] Object 0xffff88041411ef70: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 825.601797] Object 0xffff88041411ef80: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 825.601812] Object 0xffff88041411ef90: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b a5 kkkkkkkkkkkkkkk¥
[ 825.601826] Redzone 0xffff88041411efa0: bb bb bb bb bb bb bb bb
»»»»»»»»
[ 825.601839] Padding 0xffff88041411efe0: 5a 5a 5a 5a 5a 5a 00 00
ZZZZZZ..
[ 825.601855] Pid: 7353, comm: apcaccess Not tainted 2.6.39 #2
[ 825.601859] Call Trace:
[ 825.601871] [<ffffffff810b7ccd>] ? check_bytes_and_report+0x10d/0x150
[ 825.601881] [<ffffffff8110773c>] ? load_elf_binary+0xa6c/0x1c00
[ 825.601887] [<ffffffff810b7db9>] ? check_object+0xa9/0x260
[ 825.601895] [<ffffffff8110773c>] ? load_elf_binary+0xa6c/0x1c00
[ 825.601902] [<ffffffff810b86f4>] ? alloc_debug_processing+0x104/0x190
[ 825.601908] [<ffffffff810b9ac2>] ? T.912+0x272/0x2d0
[ 825.601914] [<ffffffff810ba59d>] ? __kmalloc+0x10d/0x160
[ 825.601922] [<ffffffff8110773c>] ? load_elf_binary+0xa6c/0x1c00
[ 825.601930] [<ffffffff8109bd9d>] ? __get_user_pages+0x17d/0x530
[ 825.601938] [<ffffffff810c9556>] ? get_arg_page+0x56/0x100
[ 825.601945] [<ffffffff810c8070>] ? search_binary_handler+0x90/0x240
[ 825.601951] [<ffffffff810c9e9f>] ? do_execve+0x22f/0x2f0
[ 825.601959] [<ffffffff810094a6>] ? sys_execve+0x36/0x60
[ 825.601968] [<ffffffff813d78dc>] ? stub_execve+0x6c/0xc0
[ 825.601975] FIX kmalloc-512: Restoring
0xffff88041411efe6-0xffff88041411efe7=0x5a
[ 825.601978]
[ 842.610057] INFO: task ksmd:552 blocked for more than 120 seconds.
[ 842.610063] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[ 842.610069] ksmd D 0000000000000000 0 552 2
0x00000000
[ 842.610078] ffff88041db68ca0 0000000000000046 000212d01fc11d00
ffff880400000000
[ 842.610086] ffffffff81593020 ffff88041c1a9fd8 0000000000004000
ffff88041c1a8010
[ 842.610093] ffff88041c1a9fd8 ffff88041db68ca0 0000000000000000
0000000200000002
[ 842.610100] Call Trace:
[ 842.610116] [<ffffffff813d5595>] ? schedule_timeout+0x1c5/0x230
[ 842.610128] [<ffffffff8102f69c>] ? enqueue_task_fair+0x14c/0x190
[ 842.610137] [<ffffffff8102db27>] ? task_rq_lock+0x47/0x90
[ 842.610143] [<ffffffff813d4562>] ? wait_for_common+0xd2/0x180
[ 842.610150] [<ffffffff810339b0>] ? try_to_wake_up+0x2c0/0x2c0
[ 842.610157] [<ffffffff8104cb94>] ? flush_work+0x24/0x30
[ 842.610163] [<ffffffff8104be10>] ? do_work_for_cpu+0x20/0x20
[ 842.610170] [<ffffffff8104e0ab>] ? schedule_on_each_cpu+0xab/0xe0
[ 842.610177] [<ffffffff810b5c05>] ? ksm_scan_thread+0x7f5/0xc20
[ 842.610184] [<ffffffff81052a20>] ? wake_up_bit+0x40/0x40
[ 842.610190] [<ffffffff810b5410>] ?
try_to_merge_with_ksm_page+0x570/0x570
[ 842.610196] [<ffffffff810b5410>] ?
try_to_merge_with_ksm_page+0x570/0x570
[ 842.610203] [<ffffffff810525b6>] ? kthread+0x96/0xa0
[ 842.610210] [<ffffffff813d8294>] ? kernel_thread_helper+0x4/0x10
[ 842.610218] [<ffffffff81052520>] ? kthread_worker_fn+0x120/0x120
[ 842.610223] [<ffffffff813d8290>] ? gs_change+0xb/0xb
[ 842.610229] INFO: task fsnotify_mark:662 blocked for more than 120
seconds.
[ 842.610233] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[ 842.610237] fsnotify_mark D 0000000000000000 0 662 2
0x00000000
[ 842.610244] ffff88041c31d860 0000000000000046 0000000000000000
0000000000000000
[ 842.610251] ffffffff81593020 ffff88041c365fd8 0000000000004000
ffff88041c364010
[ 842.610257] ffff88041c365fd8 ffff88041c31d860 0000000000000000
0000000000000000
[ 842.610263] Call Trace:
[ 842.610270] [<ffffffff81035521>] ? load_balance+0x91/0x5e0
[ 842.610278] [<ffffffff813d5595>] ? schedule_timeout+0x1c5/0x230
[ 842.610285] [<ffffffff8102f323>] ? pick_next_task_fair+0x103/0x190
[ 842.610291] [<ffffffff813d4a6d>] ? schedule+0x28d/0x910
[ 842.610297] [<ffffffff813d4562>] ? wait_for_common+0xd2/0x180
[ 842.610303] [<ffffffff810339b0>] ? try_to_wake_up+0x2c0/0x2c0
[ 842.610310] [<ffffffff81074420>] ? synchronize_rcu_bh+0x50/0x50
[ 842.610316] [<ffffffff8107446a>] ? synchronize_sched+0x4a/0x50
[ 842.610322] [<ffffffff8104f940>] ? find_ge_pid+0x40/0x40
[ 842.610329] [<ffffffff8105742b>] ? __synchronize_srcu+0x5b/0xc0
[ 842.610338] [<ffffffff810f5f63>] ? fsnotify_mark_destroy+0x83/0x150
[ 842.610344] [<ffffffff81052a20>] ? wake_up_bit+0x40/0x40
[ 842.610352] [<ffffffff810f5ee0>] ?
fsnotify_set_mark_ignored_mask_locked+0x20/0x20
[ 842.610360] [<ffffffff810f5ee0>] ?
fsnotify_set_mark_ignored_mask_locked+0x20/0x20
[ 842.610367] [<ffffffff810525b6>] ? kthread+0x96/0xa0
[ 842.610373] [<ffffffff813d8294>] ? kernel_thread_helper+0x4/0x10
[ 842.610381] [<ffffffff81052520>] ? kthread_worker_fn+0x120/0x120
[ 842.610386] [<ffffffff813d8290>] ? gs_change+0xb/0xb
[ 842.610395] INFO: task jbd2/md0-8:2528 blocked for more than 120 seconds.
[ 842.610399] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[ 842.610403] jbd2/md0-8 D 0000000000000005 0 2528 2
0x00000000
[ 842.610410] ffff88041d91c570 0000000000000046 ffff88041baeea28
ffffea0000000000
[ 842.610417] ffff88041d91a5e0 ffff880419f9bfd8 0000000000004000
ffff880419f9a010
[ 842.610423] ffff880419f9bfd8 ffff88041d91c570 ffff88041baeea28
ffffffff810b8495
[ 842.610429] Call Trace:
[ 842.610435] [<ffffffff810b8495>] ? init_object+0x85/0xa0
[ 842.610442] [<ffffffff810b8916>] ? free_debug_processing+0x196/0x250
[ 842.610450] [<ffffffff8105adae>] ? ktime_get_ts+0x6e/0xf0
[ 842.610458] [<ffffffff810810f0>] ? __lock_page+0x70/0x70
[ 842.610464] [<ffffffff813d5174>] ? io_schedule+0x84/0xd0
[ 842.610473] [<ffffffff811d4403>] ?
radix_tree_gang_lookup_tag_slot+0x93/0xf0
[ 842.610480] [<ffffffff810810f9>] ? sleep_on_page+0x9/0x10
[ 842.610486] [<ffffffff813d57df>] ? __wait_on_bit+0x4f/0x80
[ 842.610492] [<ffffffff810812eb>] ? wait_on_page_bit+0x6b/0x80
[ 842.610499] [<ffffffff81052a50>] ? autoremove_wake_function+0x30/0x30
[ 842.610507] [<ffffffff8108a458>] ? pagevec_lookup_tag+0x18/0x20
[ 842.610512] [<ffffffff81081f2a>] ? filemap_fdatawait_range+0xfa/0x180
[ 842.610521] [<ffffffff811be09f>] ? submit_bio+0x6f/0xf0
[ 842.610530] [<ffffffff81176276>] ?
jbd2_journal_commit_transaction+0x796/0x1270
[ 842.610539] [<ffffffff81179ed1>] ? kjournald2+0xb1/0x1e0
[ 842.610546] [<ffffffff81052a20>] ? wake_up_bit+0x40/0x40
[ 842.610553] [<ffffffff81179e20>] ? commit_timeout+0x10/0x10
[ 842.610560] [<ffffffff81179e20>] ? commit_timeout+0x10/0x10
[ 842.610566] [<ffffffff810525b6>] ? kthread+0x96/0xa0
[ 842.610572] [<ffffffff813d8294>] ? kernel_thread_helper+0x4/0x10
[ 842.610580] [<ffffffff81052520>] ? kthread_worker_fn+0x120/0x120
[ 842.610585] [<ffffffff813d8290>] ? gs_change+0xb/0xb
[ 842.610594] INFO: task nfsd:4326 blocked for more than 120 seconds.
[ 842.610598] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[ 842.610602] nfsd D 0000000000000000 0 4326 2
0x00000000
[ 842.610608] ffff88041b604bc0 0000000000000046 0000000000000016
ffffffff00000000
[ 842.610614] ffff88041d8fa5e0 ffff88041c527fd8 0000000000004000
ffff88041c526010
[ 842.610621] ffff88041c527fd8 ffff88041b604bc0 000000000000009c
ffff880000000000
[ 842.610626] Call Trace:
[ 842.610634] [<ffffffff8101e845>] ? amd_flush_garts+0x105/0x140
[ 842.610640] [<ffffffff8101fa80>] ? gart_map_sg+0x480/0x480
[ 842.610646] [<ffffffff8101f5d3>] ? flush_gart+0x23/0x50
[ 842.610653] [<ffffffff81080f98>] ? find_get_page+0x18/0x90
[ 842.610660] [<ffffffff81174b95>] ? do_get_write_access+0x265/0x4a0
[ 842.610668] [<ffffffff81052a50>] ? autoremove_wake_function+0x30/0x30
[ 842.610675] [<ffffffff81174ef9>] ?
jbd2_journal_get_write_access+0x29/0x50
[ 842.610684] [<ffffffff8115f122>] ?
__ext4_journal_get_write_access+0x32/0x80
[ 842.610692] [<ffffffff81143908>] ? ext4_reserve_inode_write+0x78/0xa0
[ 842.610700] [<ffffffff81143970>] ? ext4_mark_inode_dirty+0x40/0x1e0
[ 842.610706] [<ffffffff81156c0b>] ? ext4_journal_start_sb+0x6b/0x160
[ 842.610715] [<ffffffff81322645>] ? dev_hard_start_xmit+0x305/0x5f0
[ 842.610723] [<ffffffff81352fd0>] ? ip_finish_output2+0x290/0x290
[ 842.610730] [<ffffffff81143c65>] ? ext4_dirty_inode+0x35/0x70
[ 842.610738] [<ffffffff810e4a08>] ? __mark_inode_dirty+0x38/0x210
[ 842.610745] [<ffffffff810d9317>] ? file_update_time+0xf7/0x180
[ 842.610751] [<ffffffff81082458>] ? __generic_file_aio_write+0x1f8/0x430
[ 842.610760] [<ffffffff81373ef9>] ? udp_sendmsg+0x3c9/0x7e0
[ 842.610767] [<ffffffff81314260>] ? sock_alloc_send_pskb+0x1d0/0x320
[ 842.610774] [<ffffffff81082703>] ? generic_file_aio_write+0x73/0xf0
[ 842.610781] [<ffffffff8113f74e>] ? ext4_file_write+0x6e/0x2b0
[ 842.610788] [<ffffffff810da8ac>] ? iget_locked+0x4c/0x140
[ 842.610794] [<ffffffff8119a590>] ? fh_compose+0x4c0/0x4c0
[ 842.610800] [<ffffffff8113f6e0>] ? ext4_llseek+0x110/0x110
[ 842.610808] [<ffffffff810c24e9>] ? do_sync_readv_writev+0xa9/0xf0
[ 842.610816] [<ffffffff810c231b>] ? rw_copy_check_uvector+0x7b/0x140
[ 842.610824] [<ffffffff810c2bcf>] ? do_readv_writev+0xdf/0x210
[ 842.610830] [<ffffffff810b8735>] ? alloc_debug_processing+0x145/0x190
[ 842.610836] [<ffffffff810b9bc5>] ? kmem_cache_alloc+0xa5/0xb0
[ 842.610843] [<ffffffff8113f463>] ? ext4_file_open+0x63/0x180
[ 842.610849] [<ffffffff8119c21d>] ? nfsd_vfs_write+0xed/0x3a0
[ 842.610856] [<ffffffff810c1127>] ? __dentry_open+0x1f7/0x2b0
[ 842.610862] [<ffffffff8119c892>] ? nfsd_open+0xf2/0x1b0
[ 842.610867] [<ffffffff8119cd34>] ? nfsd_write+0xf4/0x110
[ 842.610873] [<ffffffff81199930>] ? nfsd_proc_write+0xb0/0x120
[ 842.610880] [<ffffffff811971c5>] ? nfsd_dispatch+0xf5/0x230
[ 842.610886] [<ffffffff813b456f>] ? svc_process+0x4af/0x820
[ 842.610892] [<ffffffff810339b0>] ? try_to_wake_up+0x2c0/0x2c0
[ 842.610899] [<ffffffff811977a0>] ? nfsd_svc+0x1b0/0x1b0
[ 842.610906] [<ffffffff8119784d>] ? nfsd+0xad/0x150
[ 842.610912] [<ffffffff810525b6>] ? kthread+0x96/0xa0
[ 842.610918] [<ffffffff813d8294>] ? kernel_thread_helper+0x4/0x10
[ 842.610926] [<ffffffff81052520>] ? kthread_worker_fn+0x120/0x120
[ 842.610931] [<ffffffff813d8290>] ? gs_change+0xb/0xb
[ 854.241450]
=============================================================================
[ 854.241459] BUG kmalloc-1024: Object padding overwritten
[ 854.241463]
-----------------------------------------------------------------------------
[ 854.241466]
[ 854.241471] INFO: 0xffff8802c86059e6-0xffff8802c86059e7. First byte
0x0 instead of 0x5a
[ 854.241488] INFO: Allocated in __blockdev_direct_IO+0x16c/0xa90
age=518 cpu=5 pid=7094
[ 854.241500] INFO: Freed in __kfree_skb+0x11/0x90 age=519 cpu=5 pid=5247
[ 854.241507] INFO: Slab 0xffffea0009bd5000 objects=29 used=22
fp=0xffff8802c8606f50 flags=0x80000000000040c1
[ 854.241513] INFO: Object 0xffff8802c86055a0 @offset=21920
fp=0xffff8802c86066c0
[ 854.241516]
[ 854.241520] Bytes b4 0xffff8802c8605590: 00 00 00 00 00 00 00 00 5a
5a 5a 5a 5a 5a 5a 5a ........ZZZZZZZZ
[ 854.241536] Object 0xffff8802c86055a0: 00 00 00 00 00 00 00 00 e0
61 29 fa 03 88 ff ff ........àa)ú..ÿÿ
[ 854.241551] Object 0xffff8802c86055b0: 00 00 00 00 00 00 00 00 00
00 41 ce 04 00 00 00 ..........AÎ....
[ 854.241565] Object 0xffff8802c86055c0: 03 00 00 00 0c 00 00 00 00
00 00 00 01 00 00 00 ................
[ 854.241579] Object 0xffff8802c86055d0: 00 00 00 00 00 00 00 00 00
60 00 00 00 00 00 00 .........`......
[ 854.241593] Object 0xffff8802c86055e0: b7 2c 0a 00 00 00 00 00 00
00 00 00 00 00 00 00 ·,..............
[ 854.241607] Object 0xffff8802c86055f0: b7 2c 0a 00 00 00 00 00 00
00 00 00 00 00 00 00 ·,..............
[ 854.241621] Object 0xffff8802c8605600: 01 00 00 00 00 00 00 00 f0
77 14 81 ff ff ff ff ........ðw..ÿÿÿÿ
[ 854.241635] Object 0xffff8802c8605610: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 854.241649] Object 0xffff8802c8605620: 00 00 00 00 00 00 00 00 b7
54 3b 02 00 00 00 00 ........·T;.....
[ 854.241663] Object 0xffff8802c8605630: b7 54 3b 02 00 00 00 00 20
00 00 00 00 00 00 00 ·T;.............
[ 854.241677] Object 0xffff8802c8605640: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 854.241691] Object 0xffff8802c8605650: b6 54 3b 02 00 00 00 00 00
10 00 00 00 00 00 00 ¶T;.............
[ 854.241705] Object 0xffff8802c8605660: 00 00 00 00 00 00 00 00 00
80 8e 1a 04 88 ff ff ..............ÿÿ
[ 854.241719] Object 0xffff8802c8605670: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 854.241733] Object 0xffff8802c8605680: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 854.241747] Object 0xffff8802c8605690: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 854.241761] Object 0xffff8802c86056a0: 00 00 00 00 00 00 00 00 00
00 00 00 00 10 00 00 ................
[ 854.241775] Object 0xffff8802c86056b0: b6 54 3b 02 00 00 00 00 00
60 cb a2 00 00 00 00 ¶T;......`Ë¢....
[ 854.241789] Object 0xffff8802c86056c0: 06 06 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 854.241803] Object 0xffff8802c86056d0: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 854.241817] Object 0xffff8802c86056e0: 68 fd 1c c9 02 88 ff ff 00
00 00 00 00 00 00 00 hý.É..ÿÿ........
[ 854.241831] Object 0xffff8802c86056f0: 00 60 00 00 00 00 00 00 01
00 00 00 01 00 00 00 .`..............
[ 854.241845] Object 0xffff8802c8605700: 00 80 c9 dd 39 7f 00 00 01
00 00 00 01 00 00 00 ..ÉÝ9...........
[ 854.241860] Object 0xffff8802c8605710: 00 00 00 00 00 00 00 00 08
81 bb 0c 00 ea ff ff ..........»..êÿÿ
[ 854.241874] Object 0xffff8802c8605720: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.241888] Object 0xffff8802c8605730: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.241902] Object 0xffff8802c8605740: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.241916] Object 0xffff8802c8605750: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.241930] Object 0xffff8802c8605760: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.241944] Object 0xffff8802c8605770: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.241959] Object 0xffff8802c8605780: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.241973] Object 0xffff8802c8605790: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.241987] Object 0xffff8802c86057a0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.242001] Object 0xffff8802c86057b0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.242015] Object 0xffff8802c86057c0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.242029] Object 0xffff8802c86057d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.242043] Object 0xffff8802c86057e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.242058] Object 0xffff8802c86057f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.242072] Object 0xffff8802c8605800: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.242086] Object 0xffff8802c8605810: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.242100] Object 0xffff8802c8605820: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.242114] Object 0xffff8802c8605830: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.242128] Object 0xffff8802c8605840: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.242142] Object 0xffff8802c8605850: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.242157] Object 0xffff8802c8605860: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.242171] Object 0xffff8802c8605870: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.242185] Object 0xffff8802c8605880: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.242199] Object 0xffff8802c8605890: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.242213] Object 0xffff8802c86058a0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.242227] Object 0xffff8802c86058b0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.242241] Object 0xffff8802c86058c0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.242256] Object 0xffff8802c86058d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.242270] Object 0xffff8802c86058e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.242284] Object 0xffff8802c86058f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.242298] Object 0xffff8802c8605900: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.242312] Object 0xffff8802c8605910: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.242326] Object 0xffff8802c8605920: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.242340] Object 0xffff8802c8605930: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.242355] Object 0xffff8802c8605940: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.242369] Object 0xffff8802c8605950: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.242383] Object 0xffff8802c8605960: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.242397] Object 0xffff8802c8605970: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.242411] Object 0xffff8802c8605980: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.242425] Object 0xffff8802c8605990: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b a5 kkkkkkkkkkkkkkk¥
[ 854.242440] Redzone 0xffff8802c86059a0: cc cc cc cc cc cc cc cc
ÌÌÌÌÌÌÌÌ
[ 854.242453] Padding 0xffff8802c86059e0: 5a 5a 5a 5a 5a 5a 00 00
ZZZZZZ..
[ 854.242470] Pid: 7094, comm: qemu Not tainted 2.6.39 #2
[ 854.242474] Call Trace:
[ 854.242485] [<ffffffff810b7ccd>] ? check_bytes_and_report+0x10d/0x150
[ 854.242492] [<ffffffff810b7db9>] ? check_object+0xa9/0x260
[ 854.242500] [<ffffffff810f3126>] ? __blockdev_direct_IO+0xa16/0xa90
[ 854.242507] [<ffffffff810b88b3>] ? free_debug_processing+0x133/0x250
[ 854.242513] [<ffffffff810b8afb>] ? __slab_free+0x12b/0x140
[ 854.242520] [<ffffffff810f3126>] ? __blockdev_direct_IO+0xa16/0xa90
[ 854.242530] [<ffffffff81144a17>] ? ext4_ind_direct_IO+0xf7/0x410
[ 854.242536] [<ffffffff811477f0>] ? noalloc_get_block_write+0x30/0x30
[ 854.242543] [<ffffffff81082029>] ? __filemap_fdatawrite_range+0x49/0x50
[ 854.242550] [<ffffffff810830c3>] ? generic_file_aio_read+0x653/0x6d0
[ 854.242558] [<ffffffff810567a2>] ? hrtimer_cancel+0x12/0x20
[ 854.242565] [<ffffffff81062137>] ? futex_wait+0x197/0x240
[ 854.242572] [<ffffffff81082a70>] ? delete_from_page_cache+0x70/0x70
[ 854.242580] [<ffffffff810c24e9>] ? do_sync_readv_writev+0xa9/0xf0
[ 854.242587] [<ffffffff8106239f>] ? futex_wake+0x10f/0x120
[ 854.242594] [<ffffffff81063f7b>] ? do_futex+0x11b/0xa70
[ 854.242601] [<ffffffff810471cc>] ? T.680+0x13c/0x280
[ 854.242608] [<ffffffff810c231b>] ? rw_copy_check_uvector+0x7b/0x140
[ 854.242616] [<ffffffff810c2bcf>] ? do_readv_writev+0xdf/0x210
[ 854.242624] [<ffffffff810c3003>] ? sys_preadv+0xc3/0xd0
[ 854.242633] [<ffffffff813d753b>] ? system_call_fastpath+0x16/0x1b
[ 854.242640] FIX kmalloc-1024: Restoring
0xffff8802c86059e6-0xffff8802c86059e7=0x5a
[ 854.242643]
[ 860.203918]
=============================================================================
[ 860.203927] BUG kmalloc-1024: Object padding overwritten
[ 860.203931]
-----------------------------------------------------------------------------
[ 860.203934]
[ 860.203939] INFO: 0xffff88041ad15156-0xffff88041ad15157. First byte
0x0 instead of 0x5a
[ 860.203955] INFO: Allocated in __blockdev_direct_IO+0x16c/0xa90
age=1609 cpu=3 pid=7094
[ 860.203965] INFO: Freed in __blockdev_direct_IO+0xa16/0xa90 age=1609
cpu=3 pid=7094
[ 860.203972] INFO: Slab 0xffffea000e5ddb80 objects=29 used=25
fp=0xffff88041ad14d10 flags=0x80000000000040c1
[ 860.203977] INFO: Object 0xffff88041ad14d10 @offset=19728
fp=0xffff88041ad10000
[ 860.203981]
[ 860.203984] Bytes b4 0xffff88041ad14d00: 62 28 00 00 01 00 00 00 5a
5a 5a 5a 5a 5a 5a 5a b(......ZZZZZZZZ
[ 860.204000] Object 0xffff88041ad14d10: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204015] Object 0xffff88041ad14d20: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204030] Object 0xffff88041ad14d30: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204044] Object 0xffff88041ad14d40: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204058] Object 0xffff88041ad14d50: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204073] Object 0xffff88041ad14d60: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204087] Object 0xffff88041ad14d70: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204101] Object 0xffff88041ad14d80: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204116] Object 0xffff88041ad14d90: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204130] Object 0xffff88041ad14da0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204144] Object 0xffff88041ad14db0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204159] Object 0xffff88041ad14dc0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204173] Object 0xffff88041ad14dd0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204187] Object 0xffff88041ad14de0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204202] Object 0xffff88041ad14df0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204216] Object 0xffff88041ad14e00: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204230] Object 0xffff88041ad14e10: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204245] Object 0xffff88041ad14e20: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204259] Object 0xffff88041ad14e30: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204273] Object 0xffff88041ad14e40: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204288] Object 0xffff88041ad14e50: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204302] Object 0xffff88041ad14e60: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204316] Object 0xffff88041ad14e70: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204331] Object 0xffff88041ad14e80: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204345] Object 0xffff88041ad14e90: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204359] Object 0xffff88041ad14ea0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204374] Object 0xffff88041ad14eb0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204388] Object 0xffff88041ad14ec0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204402] Object 0xffff88041ad14ed0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204417] Object 0xffff88041ad14ee0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204431] Object 0xffff88041ad14ef0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204445] Object 0xffff88041ad14f00: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204460] Object 0xffff88041ad14f10: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204474] Object 0xffff88041ad14f20: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204488] Object 0xffff88041ad14f30: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204503] Object 0xffff88041ad14f40: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204517] Object 0xffff88041ad14f50: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204531] Object 0xffff88041ad14f60: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204546] Object 0xffff88041ad14f70: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204560] Object 0xffff88041ad14f80: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204574] Object 0xffff88041ad14f90: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204589] Object 0xffff88041ad14fa0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204603] Object 0xffff88041ad14fb0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204618] Object 0xffff88041ad14fc0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204632] Object 0xffff88041ad14fd0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204646] Object 0xffff88041ad14fe0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204661] Object 0xffff88041ad14ff0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204675] Object 0xffff88041ad15000: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204689] Object 0xffff88041ad15010: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204704] Object 0xffff88041ad15020: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204718] Object 0xffff88041ad15030: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204732] Object 0xffff88041ad15040: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204747] Object 0xffff88041ad15050: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204761] Object 0xffff88041ad15060: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204775] Object 0xffff88041ad15070: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204790] Object 0xffff88041ad15080: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204804] Object 0xffff88041ad15090: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204818] Object 0xffff88041ad150a0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204833] Object 0xffff88041ad150b0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204847] Object 0xffff88041ad150c0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204861] Object 0xffff88041ad150d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204876] Object 0xffff88041ad150e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204890] Object 0xffff88041ad150f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204904] Object 0xffff88041ad15100: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b a5 kkkkkkkkkkkkkkk¥
[ 860.204919] Redzone 0xffff88041ad15110: bb bb bb bb bb bb bb bb
»»»»»»»»
[ 860.204932] Padding 0xffff88041ad15150: 5a 5a 5a 5a 5a 5a 00 00
ZZZZZZ..
[ 860.204949] Pid: 7100, comm: qemu Not tainted 2.6.39 #2
[ 860.204953] Call Trace:
[ 860.204964] [<ffffffff810b7ccd>] ? check_bytes_and_report+0x10d/0x150
[ 860.204973] [<ffffffff810f287c>] ? __blockdev_direct_IO+0x16c/0xa90
[ 860.204979] [<ffffffff810b7db9>] ? check_object+0xa9/0x260
[ 860.204986] [<ffffffff810f287c>] ? __blockdev_direct_IO+0x16c/0xa90
[ 860.204993] [<ffffffff810b86f4>] ? alloc_debug_processing+0x104/0x190
[ 860.204999] [<ffffffff810b9ac2>] ? T.912+0x272/0x2d0
[ 860.205006] [<ffffffff810b9bc5>] ? kmem_cache_alloc+0xa5/0xb0
[ 860.205013] [<ffffffff810f287c>] ? __blockdev_direct_IO+0x16c/0xa90
[ 860.205022] [<ffffffff8108a475>] ? pagevec_lookup+0x15/0x20
[ 860.205029] [<ffffffff8108b5b7>] ?
invalidate_inode_pages2_range+0x87/0x300
[ 860.205039] [<ffffffff81144e41>] ? ext4_direct_IO+0x111/0x1f0
[ 860.205045] [<ffffffff811477a0>] ? _ext4_get_block+0x160/0x160
[ 860.205052] [<ffffffff81144f20>] ? ext4_direct_IO+0x1f0/0x1f0
[ 860.205058] [<ffffffff81082182>] ? generic_file_direct_write+0xd2/0x1b0
[ 860.205064] [<ffffffff8108252b>] ? __generic_file_aio_write+0x2cb/0x430
[ 860.205071] [<ffffffff81082703>] ? generic_file_aio_write+0x73/0xf0
[ 860.205079] [<ffffffff8113f74e>] ? ext4_file_write+0x6e/0x2b0
[ 860.205087] [<ffffffff810c25ef>] ? do_sync_write+0xbf/0x100
[ 860.205095] [<ffffffff810471cc>] ? T.680+0x13c/0x280
[ 860.205103] [<ffffffff810485a8>] ? kill_pid_info+0x38/0x60
[ 860.205110] [<ffffffff81048854>] ? sys_kill+0x94/0x1b0
[ 860.205117] [<ffffffff810c3198>] ? vfs_write+0xc8/0x190
[ 860.205124] [<ffffffff810c32fb>] ? sys_pwrite64+0x9b/0xb0
[ 860.205134] [<ffffffff813d753b>] ? system_call_fastpath+0x16/0x1b
[ 860.205140] FIX kmalloc-1024: Restoring
0xffff88041ad15156-0xffff88041ad15157=0x5a
[ 860.205143]
^ permalink raw reply [flat|nested] 111+ messages in thread
* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
@ 2011-06-01 9:29 ` Brad Campbell
0 siblings, 0 replies; 111+ messages in thread
From: Brad Campbell @ 2011-06-01 9:29 UTC (permalink / raw)
To: Avi Kivity
Cc: Hugh Dickins, Andrea Arcangeli, Borislav Petkov, linux-kernel,
kvm, linux-mm
On 01/06/11 14:56, Avi Kivity wrote:
> On 06/01/2011 09:31 AM, Brad Campbell wrote:
>> On 01/06/11 12:52, Hugh Dickins wrote:
>>
>>>
>>> I guess Brad could try SLUB debugging, boot with slub_debug=P
>>> for poisoning perhaps; though it might upset alignments and
>>> drive the problem underground. Or see if the same happens
>>> with SLAB instead of SLUB.
>>
>> Not much use I'm afraid.
>> This is all I get in the log
>>
>> [ 3161.300073]
>> =============================================================================
>>
>> [ 3161.300147] BUG kmalloc-512: Freechain corrupt
>>
>> The qemu process is then frozen, unkillable but reported in state "R"
>>
>> 13881 ? R 3:27 /usr/bin/qemu -S -M pc-0.13 -enable-kvm -m 1024 -smp
>> 2,sockets=2,cores=1,threads=1 -nam
>>
>> The machine then progressively dies until it's frozen solid with no
>> further error messages.
>>
>> I stupidly forgot to do an alt-sysrq-t prior to doing an alt-sysrq-b,
>> but at least it responded to that.
>>
>> On the bright side I can reproduce it at will.
>
> Please try slub_debug=FZPU; that should point the finger (hopefully at
> somebody else).
>
Well the first attempt locked the machine solid. No network, no console..
I saw
"=========================================================================="
on the console.. nothing after that. Would not respond to sysrq-t or any
other sysrq combination other than -b, which rebooted the box.
No output on netconsole at all, I had to walk to the other building to
look at the monitor and reboot it.
The second attempt jammed netconsole again, but I managed to get this
from an ssh session I already had established. The machine died a slow
and horrible death, but remained interactive enough for me to reboot it with
echo b > /proc/sysrq-trigger
Nothing else worked.
[ 376.269051]
=============================================================================
[ 413.755328]
=============================================================================
[ 413.755337] BUG kmalloc-1024: Object padding overwritten
[ 413.755341]
-----------------------------------------------------------------------------
[ 413.755344]
[ 413.755349] INFO: 0xffff880417179566-0xffff880417179567. First byte
0x0 instead of 0x5a
[ 413.755363] INFO: Allocated in tcp_send_ack+0x26/0x120 age=5320 cpu=5
pid=0
[ 413.755374] INFO: Freed in __kfree_skb+0x11/0x90 age=5320 cpu=5 pid=0
[ 413.755380] INFO: Slab 0xffffea000e50d240 objects=29 used=5
fp=0xffff880417179120 flags=0x80000000000040c1
[ 413.755386] INFO: Object 0xffff880417179120 @offset=4384
fp=0xffff8804171799b0
[ 413.755389]
[ 413.755392] Bytes b4 0xffff880417179110: d9 2b 00 00 01 00 00 00 5a
5a 5a 5a 5a 5a 5a 5a Ù+......ZZZZZZZZ
[ 413.755408] Object 0xffff880417179120: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755423] Object 0xffff880417179130: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755438] Object 0xffff880417179140: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755452] Object 0xffff880417179150: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755466] Object 0xffff880417179160: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755480] Object 0xffff880417179170: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755494] Object 0xffff880417179180: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755508] Object 0xffff880417179190: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755522] Object 0xffff8804171791a0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755536] Object 0xffff8804171791b0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755551] Object 0xffff8804171791c0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755565] Object 0xffff8804171791d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755579] Object 0xffff8804171791e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755593] Object 0xffff8804171791f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755607] Object 0xffff880417179200: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755621] Object 0xffff880417179210: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755635] Object 0xffff880417179220: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755650] Object 0xffff880417179230: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755664] Object 0xffff880417179240: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755678] Object 0xffff880417179250: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755692] Object 0xffff880417179260: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755706] Object 0xffff880417179270: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755720] Object 0xffff880417179280: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755734] Object 0xffff880417179290: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755749] Object 0xffff8804171792a0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755763] Object 0xffff8804171792b0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755777] Object 0xffff8804171792c0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755791] Object 0xffff8804171792d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755805] Object 0xffff8804171792e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755819] Object 0xffff8804171792f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755834] Object 0xffff880417179300: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755848] Object 0xffff880417179310: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755862] Object 0xffff880417179320: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755876] Object 0xffff880417179330: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755890] Object 0xffff880417179340: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755904] Object 0xffff880417179350: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755919] Object 0xffff880417179360: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755933] Object 0xffff880417179370: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755947] Object 0xffff880417179380: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755961] Object 0xffff880417179390: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755975] Object 0xffff8804171793a0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755989] Object 0xffff8804171793b0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.756004] Object 0xffff8804171793c0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.756018] Object 0xffff8804171793d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.756032] Object 0xffff8804171793e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.756046] Object 0xffff8804171793f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.756060] Object 0xffff880417179400: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.756074] Object 0xffff880417179410: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.756089] Object 0xffff880417179420: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.756103] Object 0xffff880417179430: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.756117] Object 0xffff880417179440: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.756131] Object 0xffff880417179450: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.756145] Object 0xffff880417179460: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.756160] Object 0xffff880417179470: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.756174] Object 0xffff880417179480: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.756188] Object 0xffff880417179490: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.756202] Object 0xffff8804171794a0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.756216] Object 0xffff8804171794b0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.756230] Object 0xffff8804171794c0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.756245] Object 0xffff8804171794d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.756259] Object 0xffff8804171794e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.756273] Object 0xffff8804171794f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.756287] Object 0xffff880417179500: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.756301] Object 0xffff880417179510: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b a5 kkkkkkkkkkkkkkk¥
[ 413.756316] Redzone 0xffff880417179520: bb bb bb bb bb bb bb bb
»»»»»»»»
[ 413.756329] Padding 0xffff880417179560: 5a 5a 5a 5a 5a 5a 00 00
ZZZZZZ..
[ 413.756345] Pid: 5247, comm: qemu Not tainted 2.6.39 #2
[ 413.756349] Call Trace:
[ 413.756353] <IRQ> [<ffffffff810b7ccd>] ?
check_bytes_and_report+0x10d/0x150
[ 413.756372] [<ffffffff81318f1c>] ? pskb_expand_head+0x15c/0x250
[ 413.756379] [<ffffffff810b7db9>] ? check_object+0xa9/0x260
[ 413.756387] [<ffffffff81318f1c>] ? pskb_expand_head+0x15c/0x250
[ 413.756393] [<ffffffff810b86f4>] ? alloc_debug_processing+0x104/0x190
[ 413.756400] [<ffffffff810b9ac2>] ? T.912+0x272/0x2d0
[ 413.756409] [<ffffffff810ba59d>] ? __kmalloc+0x10d/0x160
[ 413.756416] [<ffffffff81318f1c>] ? pskb_expand_head+0x15c/0x250
[ 413.756424] [<ffffffff813a6c45>] ? nf_bridge_copy_header+0x145/0x160
[ 413.756431] [<ffffffff8139f78d>] ? br_dev_queue_push_xmit+0x6d/0x80
[ 413.756439] [<ffffffff813a55a0>] ? br_nf_post_routing+0x2a0/0x2f0
[ 413.756447] [<ffffffff81346bc4>] ? nf_iterate+0x84/0xb0
[ 413.756453] [<ffffffff8139f720>] ? br_flood_deliver+0x20/0x20
[ 413.756459] [<ffffffff81346c64>] ? nf_hook_slow+0x74/0x120
[ 413.756465] [<ffffffff8139f720>] ? br_flood_deliver+0x20/0x20
[ 413.756472] [<ffffffff8139f7da>] ? br_forward_finish+0x3a/0x60
[ 413.756479] [<ffffffff813a5758>] ? br_nf_forward_finish+0x168/0x170
[ 413.756487] [<ffffffff813a5c90>] ? br_nf_forward_ip+0x360/0x3a0
[ 413.756492] [<ffffffff81346bc4>] ? nf_iterate+0x84/0xb0
[ 413.756498] [<ffffffff8139f7a0>] ? br_dev_queue_push_xmit+0x80/0x80
[ 413.756504] [<ffffffff81346c64>] ? nf_hook_slow+0x74/0x120
[ 413.756510] [<ffffffff8139f7a0>] ? br_dev_queue_push_xmit+0x80/0x80
[ 413.756516] [<ffffffff8139f800>] ? br_forward_finish+0x60/0x60
[ 413.756522] [<ffffffff8139f800>] ? br_forward_finish+0x60/0x60
[ 413.756528] [<ffffffff8139f875>] ? __br_forward+0x75/0xc0
[ 413.756534] [<ffffffff8139f426>] ? deliver_clone+0x36/0x60
[ 413.756540] [<ffffffff8139f69d>] ? br_flood+0xbd/0x100
[ 413.756546] [<ffffffff813a05b0>] ? br_handle_local_finish+0x40/0x40
[ 413.756552] [<ffffffff813a080e>] ? br_handle_frame_finish+0x25e/0x280
[ 413.756560] [<ffffffff813a60f0>] ? br_nf_pre_routing_finish+0x1a0/0x330
[ 413.756568] [<ffffffff813a6958>] ? br_nf_pre_routing+0x6d8/0x800
[ 413.756577] [<ffffffff8102d46a>] ? enqueue_task+0x3a/0x90
[ 413.756582] [<ffffffff81346bc4>] ? nf_iterate+0x84/0xb0
[ 413.756589] [<ffffffff813a05b0>] ? br_handle_local_finish+0x40/0x40
[ 413.756594] [<ffffffff81346c64>] ? nf_hook_slow+0x74/0x120
[ 413.756600] [<ffffffff813a05b0>] ? br_handle_local_finish+0x40/0x40
[ 413.756607] [<ffffffff810339b0>] ? try_to_wake_up+0x2c0/0x2c0
[ 413.756613] [<ffffffff813a09d9>] ? br_handle_frame+0x1a9/0x280
[ 413.756620] [<ffffffff813a0830>] ? br_handle_frame_finish+0x280/0x280
[ 413.756627] [<ffffffff81320ef7>] ? __netif_receive_skb+0x157/0x5c0
[ 413.756634] [<ffffffff81321443>] ? process_backlog+0xe3/0x1d0
[ 413.756641] [<ffffffff81321da5>] ? net_rx_action+0xc5/0x1d0
[ 413.756650] [<ffffffff8103df11>] ? __do_softirq+0x91/0x120
[ 413.756657] [<ffffffff813d838c>] ? call_softirq+0x1c/0x30
[ 413.756660] <EOI> [<ffffffff81003cbd>] ? do_softirq+0x4d/0x80
[ 413.756673] [<ffffffff81321ece>] ? netif_rx_ni+0x1e/0x30
[ 413.756681] [<ffffffff812b3ae2>] ? tun_chr_aio_write+0x332/0x4e0
[ 413.756688] [<ffffffff812b37b0>] ? tun_sendmsg+0x4d0/0x4d0
[ 413.756697] [<ffffffff810c24e9>] ? do_sync_readv_writev+0xa9/0xf0
[ 413.756704] [<ffffffff81063f9c>] ? do_futex+0x13c/0xa70
[ 413.756711] [<ffffffff811d6730>] ? timerqueue_add+0x60/0xb0
[ 413.756719] [<ffffffff81056ab7>] ? __hrtimer_start_range_ns+0x1e7/0x410
[ 413.756726] [<ffffffff810c231b>] ? rw_copy_check_uvector+0x7b/0x140
[ 413.756734] [<ffffffff810c2bcf>] ? do_readv_writev+0xdf/0x210
[ 413.756742] [<ffffffff810c2e7e>] ? sys_writev+0x4e/0xc0
[ 413.756750] [<ffffffff813d753b>] ? system_call_fastpath+0x16/0x1b
[ 413.756756] FIX kmalloc-1024: Restoring
0xffff880417179566-0xffff880417179567=0x5a
[ 413.756760]
[ 556.640033]
=============================================================================
[ 556.640041] BUG kmalloc-512: Object padding overwritten
[ 556.640045]
-----------------------------------------------------------------------------
[ 556.640048]
[ 556.640053] INFO: 0xffff880403bf091e-0xffff880403bf091f. First byte
0x0 instead of 0x5a
[ 556.640069] INFO: Allocated in sock_alloc_send_pskb+0x1d0/0x320
age=21401 cpu=2 pid=5630
[ 556.640080] INFO: Freed in __kfree_skb+0x11/0x90 age=21386 cpu=2 pid=3753
[ 556.640086] INFO: Slab 0xffffea000e0d1c80 objects=28 used=10
fp=0xffff880403bf06d8 flags=0x80000000000040c1
[ 556.640092] INFO: Object 0xffff880403bf06d8 @offset=1752
fp=0xffff880403bf1488
[ 556.640096]
[ 556.640099] Bytes b4 0xffff880403bf06c8: 90 c4 ff ff 00 00 00 00 5a
5a 5a 5a 5a 5a 5a 5a .Äÿÿ....ZZZZZZZZ
[ 556.640115] Object 0xffff880403bf06d8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 556.640130] Object 0xffff880403bf06e8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 556.640144] Object 0xffff880403bf06f8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 556.640158] Object 0xffff880403bf0708: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 556.640172] Object 0xffff880403bf0718: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 556.640187] Object 0xffff880403bf0728: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 556.640201] Object 0xffff880403bf0738: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 556.640215] Object 0xffff880403bf0748: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 556.640229] Object 0xffff880403bf0758: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 556.640243] Object 0xffff880403bf0768: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 556.640257] Object 0xffff880403bf0778: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 556.640271] Object 0xffff880403bf0788: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 556.640286] Object 0xffff880403bf0798: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 556.640300] Object 0xffff880403bf07a8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 556.640314] Object 0xffff880403bf07b8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 556.640328] Object 0xffff880403bf07c8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 556.640343] Object 0xffff880403bf07d8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 556.640357] Object 0xffff880403bf07e8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 556.640371] Object 0xffff880403bf07f8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 556.640385] Object 0xffff880403bf0808: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 556.640399] Object 0xffff880403bf0818: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 556.640413] Object 0xffff880403bf0828: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 556.640428] Object 0xffff880403bf0838: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 556.640442] Object 0xffff880403bf0848: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 556.640456] Object 0xffff880403bf0858: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 556.640471] Object 0xffff880403bf0868: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 556.640485] Object 0xffff880403bf0878: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 556.640499] Object 0xffff880403bf0888: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 556.640513] Object 0xffff880403bf0898: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 556.640527] Object 0xffff880403bf08a8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 556.640542] Object 0xffff880403bf08b8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 556.640556] Object 0xffff880403bf08c8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b a5 kkkkkkkkkkkkkkk¥
[ 556.640570] Redzone 0xffff880403bf08d8: bb bb bb bb bb bb bb bb
»»»»»»»»
[ 556.640583] Padding 0xffff880403bf0918: 5a 5a 5a 5a 5a 5a 00 00
ZZZZZZ..
[ 556.640599] Pid: 4809, comm: qemu Not tainted 2.6.39 #2
[ 556.640603] Call Trace:
[ 556.640607] <IRQ> [<ffffffff810b7ccd>] ?
check_bytes_and_report+0x10d/0x150
[ 556.640626] [<ffffffff8131aae7>] ? __netdev_alloc_skb+0x17/0x40
[ 556.640632] [<ffffffff810b7db9>] ? check_object+0xa9/0x260
[ 556.640640] [<ffffffff8131aae7>] ? __netdev_alloc_skb+0x17/0x40
[ 556.640647] [<ffffffff810b86f4>] ? alloc_debug_processing+0x104/0x190
[ 556.640654] [<ffffffff810b9ac2>] ? T.912+0x272/0x2d0
[ 556.640661] [<ffffffff810bb70d>] ? __kmalloc_track_caller+0x10d/0x160
[ 556.640668] [<ffffffff813198c2>] ? __alloc_skb+0x72/0x160
[ 556.640676] [<ffffffff8131aae7>] ? __netdev_alloc_skb+0x17/0x40
[ 556.640684] [<ffffffff813a7b93>] ?
br_ip4_multicast_alloc_query+0x23/0x1c0
[ 556.640690] [<ffffffff813a7e76>] ? br_multicast_send_query+0x76/0x130
[ 556.640698] [<ffffffff8104dba8>] ? wq_worker_waking_up+0x8/0x30
[ 556.640706] [<ffffffff8102ca81>] ? sched_slice+0x51/0x80
[ 556.640711] [<ffffffff813a7fb5>] ?
br_multicast_port_query_expired+0x45/0x70
[ 556.640719] [<ffffffff81043f77>] ? run_timer_softirq+0x137/0x270
[ 556.640725] [<ffffffff81033fe9>] ? scheduler_tick+0x289/0x2d0
[ 556.640731] [<ffffffff813a7f70>] ? br_multicast_query_expired+0x40/0x40
[ 556.640740] [<ffffffff8103df11>] ? __do_softirq+0x91/0x120
[ 556.640747] [<ffffffff813d838c>] ? call_softirq+0x1c/0x30
[ 556.640755] [<ffffffff81003cbd>] ? do_softirq+0x4d/0x80
[ 556.640762] [<ffffffff8103ddbe>] ? irq_exit+0x8e/0xb0
[ 556.640770] [<ffffffff81019d1a>] ? smp_apic_timer_interrupt+0x6a/0xa0
[ 556.640776] [<ffffffff813d7e53>] ? apic_timer_interrupt+0x13/0x20
[ 556.640780] <EOI> [<ffffffff8102965a>] ?
flush_tlb_others_ipi+0x11a/0x130
[ 556.640791] [<ffffffff81029647>] ? flush_tlb_others_ipi+0x107/0x130
[ 556.640799] [<ffffffff810aadab>] ? ptep_clear_flush+0xb/0x10
[ 556.640806] [<ffffffff8109a40b>] ? do_wp_page+0x34b/0x7a0
[ 556.640836] [<ffffffffa00a188a>] ? kvm_read_guest_page+0x5a/0x70 [kvm]
[ 556.640863] [<ffffffffa00b7db4>] ?
paging64_walk_addr_generic+0x264/0x4c0 [kvm]
[ 556.640871] [<ffffffff8109acad>] ? handle_pte_fault+0x44d/0x990
[ 556.640878] [<ffffffff8109b4e8>] ? follow_page+0x268/0x440
[ 556.640886] [<ffffffff8109bd4d>] ? __get_user_pages+0x12d/0x530
[ 556.640892] [<ffffffff81028dbb>] ? gup_pud_range+0x12b/0x1b0
[ 556.640914] [<ffffffffa00a2d2e>] ? get_user_page_nowait+0x2e/0x40 [kvm]
[ 556.640935] [<ffffffffa00a2e7b>] ? hva_to_pfn+0x13b/0x440 [kvm]
[ 556.640956] [<ffffffffa00a31e9>] ? __gfn_to_pfn+0x39/0xf0 [kvm]
[ 556.640979] [<ffffffffa00b825d>] ? try_async_pf+0x4d/0x190 [kvm]
[ 556.641002] [<ffffffffa00b92be>] ? tdp_page_fault+0x10e/0x200 [kvm]
[ 556.641026] [<ffffffffa00b968c>] ? kvm_mmu_page_fault+0x1c/0x80 [kvm]
[ 556.641050] [<ffffffffa00b0a7a>] ?
kvm_arch_vcpu_ioctl_run+0x3fa/0xcf0 [kvm]
[ 556.641058] [<ffffffff8106239f>] ? futex_wake+0x10f/0x120
[ 556.641065] [<ffffffff81063f7b>] ? do_futex+0x11b/0xa70
[ 556.641086] [<ffffffffa00a517f>] ? kvm_vcpu_ioctl+0x4df/0x5e0 [kvm]
[ 556.641097] [<ffffffff81046671>] ? __dequeue_signal+0xe1/0x170
[ 556.641104] [<ffffffff8104821c>] ? do_send_sig_info+0x6c/0x90
[ 556.641112] [<ffffffff81046aac>] ? dequeue_signal+0x3c/0x170
[ 556.641119] [<ffffffff8104551f>] ? copy_siginfo_to_user+0xef/0x1d0
[ 556.641125] [<ffffffff810d26db>] ? do_vfs_ioctl+0x9b/0x4f0
[ 556.641132] [<ffffffff8106494a>] ? sys_futex+0x7a/0x180
[ 556.641137] [<ffffffff810d2b79>] ? sys_ioctl+0x49/0x80
[ 556.641145] [<ffffffff813d753b>] ? system_call_fastpath+0x16/0x1b
[ 556.641152] FIX kmalloc-512: Restoring
0xffff880403bf091e-0xffff880403bf091f=0x5a
[ 556.641155]
[ 602.610062] INFO: task ksmd:552 blocked for more than 120 seconds.
[ 602.610068] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[ 602.610073] ksmd D 0000000000000000 0 552 2
0x00000000
[ 602.610083] ffff88041db68ca0 0000000000000046 000212d01fc11d00
ffff880400000000
[ 602.610090] ffffffff81593020 ffff88041c1a9fd8 0000000000004000
ffff88041c1a8010
[ 602.610097] ffff88041c1a9fd8 ffff88041db68ca0 0000000000000000
0000000200000002
[ 602.610104] Call Trace:
[ 602.610121] [<ffffffff813d5595>] ? schedule_timeout+0x1c5/0x230
[ 602.610132] [<ffffffff8102f69c>] ? enqueue_task_fair+0x14c/0x190
[ 602.610141] [<ffffffff8102db27>] ? task_rq_lock+0x47/0x90
[ 602.610148] [<ffffffff813d4562>] ? wait_for_common+0xd2/0x180
[ 602.610154] [<ffffffff810339b0>] ? try_to_wake_up+0x2c0/0x2c0
[ 602.610162] [<ffffffff8104cb94>] ? flush_work+0x24/0x30
[ 602.610167] [<ffffffff8104be10>] ? do_work_for_cpu+0x20/0x20
[ 602.610174] [<ffffffff8104e0ab>] ? schedule_on_each_cpu+0xab/0xe0
[ 602.610181] [<ffffffff810b5c05>] ? ksm_scan_thread+0x7f5/0xc20
[ 602.610189] [<ffffffff81052a20>] ? wake_up_bit+0x40/0x40
[ 602.610194] [<ffffffff810b5410>] ?
try_to_merge_with_ksm_page+0x570/0x570
[ 602.610200] [<ffffffff810b5410>] ?
try_to_merge_with_ksm_page+0x570/0x570
[ 602.610207] [<ffffffff810525b6>] ? kthread+0x96/0xa0
[ 602.610214] [<ffffffff813d8294>] ? kernel_thread_helper+0x4/0x10
[ 602.610221] [<ffffffff81052520>] ? kthread_worker_fn+0x120/0x120
[ 602.610227] [<ffffffff813d8290>] ? gs_change+0xb/0xb
[ 602.610233] INFO: task fsnotify_mark:662 blocked for more than 120
seconds.
[ 602.610237] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[ 602.610241] fsnotify_mark D 0000000000000000 0 662 2
0x00000000
[ 602.610248] ffff88041c31d860 0000000000000046 0000000000000000
0000000000000000
[ 602.610255] ffffffff81593020 ffff88041c365fd8 0000000000004000
ffff88041c364010
[ 602.610262] ffff88041c365fd8 ffff88041c31d860 0000000000000000
0000000000000000
[ 602.610268] Call Trace:
[ 602.610275] [<ffffffff81035521>] ? load_balance+0x91/0x5e0
[ 602.610282] [<ffffffff813d5595>] ? schedule_timeout+0x1c5/0x230
[ 602.610289] [<ffffffff8102f323>] ? pick_next_task_fair+0x103/0x190
[ 602.610295] [<ffffffff813d4a6d>] ? schedule+0x28d/0x910
[ 602.610302] [<ffffffff813d4562>] ? wait_for_common+0xd2/0x180
[ 602.610307] [<ffffffff810339b0>] ? try_to_wake_up+0x2c0/0x2c0
[ 602.610314] [<ffffffff81074420>] ? synchronize_rcu_bh+0x50/0x50
[ 602.610320] [<ffffffff8107446a>] ? synchronize_sched+0x4a/0x50
[ 602.610326] [<ffffffff8104f940>] ? find_ge_pid+0x40/0x40
[ 602.610333] [<ffffffff8105742b>] ? __synchronize_srcu+0x5b/0xc0
[ 602.610342] [<ffffffff810f5f63>] ? fsnotify_mark_destroy+0x83/0x150
[ 602.610348] [<ffffffff81052a20>] ? wake_up_bit+0x40/0x40
[ 602.610356] [<ffffffff810f5ee0>] ?
fsnotify_set_mark_ignored_mask_locked+0x20/0x20
[ 602.610364] [<ffffffff810f5ee0>] ?
fsnotify_set_mark_ignored_mask_locked+0x20/0x20
[ 602.610371] [<ffffffff810525b6>] ? kthread+0x96/0xa0
[ 602.610377] [<ffffffff813d8294>] ? kernel_thread_helper+0x4/0x10
[ 602.610385] [<ffffffff81052520>] ? kthread_worker_fn+0x120/0x120
[ 602.610390] [<ffffffff813d8290>] ? gs_change+0xb/0xb
[ 644.689408]
=============================================================================
[ 644.689416] BUG kmalloc-512: Object padding overwritten
[ 644.689420]
-----------------------------------------------------------------------------
[ 644.689423]
[ 644.689428] INFO: 0xffff88041411f476-0xffff88041411f477. First byte
0x0 instead of 0x5a
[ 644.689444] INFO: Allocated in load_elf_binary+0xa6c/0x1c00 age=18648
cpu=3 pid=5915
[ 644.689454] INFO: Freed in load_elf_binary+0xab8/0x1c00 age=18648
cpu=3 pid=5915
[ 644.689461] INFO: Slab 0xffffea000e463e20 objects=28 used=10
fp=0xffff88041411f230 flags=0x80000000000040c1
[ 644.689467] INFO: Object 0xffff88041411f230 @offset=12848
fp=0xffff88041411eda0
[ 644.689470]
[ 644.689474] Bytes b4 0xffff88041411f220: 22 6f 00 00 01 00 00 00 5a
5a 5a 5a 5a 5a 5a 5a "o......ZZZZZZZZ
[ 644.689490] Object 0xffff88041411f230: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 644.689505] Object 0xffff88041411f240: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 644.689519] Object 0xffff88041411f250: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 644.689533] Object 0xffff88041411f260: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 644.689547] Object 0xffff88041411f270: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 644.689562] Object 0xffff88041411f280: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 644.689576] Object 0xffff88041411f290: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 644.689590] Object 0xffff88041411f2a0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 644.689604] Object 0xffff88041411f2b0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 644.689618] Object 0xffff88041411f2c0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 644.689632] Object 0xffff88041411f2d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 644.689647] Object 0xffff88041411f2e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 644.689661] Object 0xffff88041411f2f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 644.689675] Object 0xffff88041411f300: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 644.689689] Object 0xffff88041411f310: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 644.689703] Object 0xffff88041411f320: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 644.689717] Object 0xffff88041411f330: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 644.689732] Object 0xffff88041411f340: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 644.689746] Object 0xffff88041411f350: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 644.689760] Object 0xffff88041411f360: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 644.689774] Object 0xffff88041411f370: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 644.689788] Object 0xffff88041411f380: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 644.689802] Object 0xffff88041411f390: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 644.689817] Object 0xffff88041411f3a0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 644.689831] Object 0xffff88041411f3b0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 644.689845] Object 0xffff88041411f3c0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 644.689859] Object 0xffff88041411f3d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 644.689873] Object 0xffff88041411f3e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 644.689888] Object 0xffff88041411f3f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 644.689902] Object 0xffff88041411f400: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 644.689916] Object 0xffff88041411f410: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 644.689930] Object 0xffff88041411f420: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b a5 kkkkkkkkkkkkkkk¥
[ 644.689945] Redzone 0xffff88041411f430: bb bb bb bb bb bb bb bb
»»»»»»»»
[ 644.689957] Padding 0xffff88041411f470: 5a 5a 5a 5a 5a 5a 00 00
ZZZZZZ..
[ 644.689974] Pid: 6597, comm: awk Not tainted 2.6.39 #2
[ 644.689978] Call Trace:
[ 644.689989] [<ffffffff810b7ccd>] ? check_bytes_and_report+0x10d/0x150
[ 644.689998] [<ffffffff8110773c>] ? load_elf_binary+0xa6c/0x1c00
[ 644.690005] [<ffffffff810b7db9>] ? check_object+0xa9/0x260
[ 644.690012] [<ffffffff8110773c>] ? load_elf_binary+0xa6c/0x1c00
[ 644.690019] [<ffffffff810b86f4>] ? alloc_debug_processing+0x104/0x190
[ 644.690026] [<ffffffff810b9ac2>] ? T.912+0x272/0x2d0
[ 644.690032] [<ffffffff810ba59d>] ? __kmalloc+0x10d/0x160
[ 644.690039] [<ffffffff8110773c>] ? load_elf_binary+0xa6c/0x1c00
[ 644.690047] [<ffffffff8109bd9d>] ? __get_user_pages+0x17d/0x530
[ 644.690055] [<ffffffff810c9556>] ? get_arg_page+0x56/0x100
[ 644.690062] [<ffffffff810c8070>] ? search_binary_handler+0x90/0x240
[ 644.690069] [<ffffffff810c9e9f>] ? do_execve+0x22f/0x2f0
[ 644.690076] [<ffffffff810094a6>] ? sys_execve+0x36/0x60
[ 644.690085] [<ffffffff813d78dc>] ? stub_execve+0x6c/0xc0
[ 644.690092] FIX kmalloc-512: Restoring
0xffff88041411f476-0xffff88041411f477=0x5a
[ 644.690095]
[ 704.979481]
=============================================================================
[ 704.979488] BUG kmalloc-512: Object padding overwritten
[ 704.979492]
-----------------------------------------------------------------------------
[ 704.979496]
[ 704.979501] INFO: 0xffff8804168c1fee-0xffff8804168c1fef. First byte
0x0 instead of 0x5a
[ 704.979517] INFO: Allocated in load_elf_binary+0xa6c/0x1c00 age=24184
cpu=0 pid=5999
[ 704.979527] INFO: Freed in load_elf_binary+0xab8/0x1c00 age=24184
cpu=0 pid=5999
[ 704.979534] INFO: Slab 0xffffea000e4eea00 objects=28 used=10
fp=0xffff8804168c1da8 flags=0x80000000000040c1
[ 704.979540] INFO: Object 0xffff8804168c1da8 @offset=7592
fp=0xffff8804168c2910
[ 704.979543]
[ 704.979547] Bytes b4 0xffff8804168c1d98: 93 98 ff ff 00 00 00 00 5a
5a 5a 5a 5a 5a 5a 5a ..ÿÿ....ZZZZZZZZ
[ 704.979563] Object 0xffff8804168c1da8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 704.979578] Object 0xffff8804168c1db8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 704.979592] Object 0xffff8804168c1dc8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 704.979606] Object 0xffff8804168c1dd8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 704.979620] Object 0xffff8804168c1de8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 704.979634] Object 0xffff8804168c1df8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 704.979648] Object 0xffff8804168c1e08: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 704.979662] Object 0xffff8804168c1e18: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 704.979676] Object 0xffff8804168c1e28: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 704.979690] Object 0xffff8804168c1e38: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 704.979704] Object 0xffff8804168c1e48: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 704.979719] Object 0xffff8804168c1e58: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 704.979733] Object 0xffff8804168c1e68: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 704.979747] Object 0xffff8804168c1e78: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 704.979761] Object 0xffff8804168c1e88: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 704.979775] Object 0xffff8804168c1e98: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 704.979789] Object 0xffff8804168c1ea8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 704.979803] Object 0xffff8804168c1eb8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 704.979817] Object 0xffff8804168c1ec8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 704.979831] Object 0xffff8804168c1ed8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 704.979845] Object 0xffff8804168c1ee8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 704.979859] Object 0xffff8804168c1ef8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 704.979873] Object 0xffff8804168c1f08: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 704.979888] Object 0xffff8804168c1f18: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 704.979902] Object 0xffff8804168c1f28: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 704.979916] Object 0xffff8804168c1f38: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 704.979930] Object 0xffff8804168c1f48: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 704.979944] Object 0xffff8804168c1f58: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 704.979958] Object 0xffff8804168c1f68: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 704.979972] Object 0xffff8804168c1f78: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 704.979986] Object 0xffff8804168c1f88: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 704.980000] Object 0xffff8804168c1f98: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b a5 kkkkkkkkkkkkkkk¥
[ 704.980015] Redzone 0xffff8804168c1fa8: bb bb bb bb bb bb bb bb
»»»»»»»»
[ 704.980028] Padding 0xffff8804168c1fe8: 5a 5a 5a 5a 5a 5a 00 00
ZZZZZZ..
[ 704.980044] Pid: 6812, comm: get-rrdtool-dat Not tainted 2.6.39 #2
[ 704.980048] Call Trace:
[ 704.980060] [<ffffffff810b7ccd>] ? check_bytes_and_report+0x10d/0x150
[ 704.980069] [<ffffffff81106e49>] ? load_elf_binary+0x179/0x1c00
[ 704.980075] [<ffffffff810b7db9>] ? check_object+0xa9/0x260
[ 704.980083] [<ffffffff81106e49>] ? load_elf_binary+0x179/0x1c00
[ 704.980090] [<ffffffff810b86f4>] ? alloc_debug_processing+0x104/0x190
[ 704.980096] [<ffffffff810b9ac2>] ? T.912+0x272/0x2d0
[ 704.980102] [<ffffffff810ba59d>] ? __kmalloc+0x10d/0x160
[ 704.980109] [<ffffffff81106e49>] ? load_elf_binary+0x179/0x1c00
[ 704.980117] [<ffffffff8109b782>] ? __pte_alloc+0x42/0x130
[ 704.980123] [<ffffffff8109b5a5>] ? follow_page+0x325/0x440
[ 704.980130] [<ffffffff8109b5a5>] ? follow_page+0x325/0x440
[ 704.980137] [<ffffffff8109bd9d>] ? __get_user_pages+0x17d/0x530
[ 704.980145] [<ffffffff810c9556>] ? get_arg_page+0x56/0x100
[ 704.980152] [<ffffffff810c8070>] ? search_binary_handler+0x90/0x240
[ 704.980158] [<ffffffff810c9e9f>] ? do_execve+0x22f/0x2f0
[ 704.980166] [<ffffffff810094a6>] ? sys_execve+0x36/0x60
[ 704.980175] [<ffffffff813d78dc>] ? stub_execve+0x6c/0xc0
[ 704.980182] FIX kmalloc-512: Restoring
0xffff8804168c1fee-0xffff8804168c1fef=0x5a
[ 704.980185]
[ 722.610058] INFO: task ksmd:552 blocked for more than 120 seconds.
[ 722.610063] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[ 722.610069] ksmd D 0000000000000000 0 552 2
0x00000000
[ 722.610078] ffff88041db68ca0 0000000000000046 000212d01fc11d00
ffff880400000000
[ 722.610085] ffffffff81593020 ffff88041c1a9fd8 0000000000004000
ffff88041c1a8010
[ 722.610092] ffff88041c1a9fd8 ffff88041db68ca0 0000000000000000
0000000200000002
[ 722.610099] Call Trace:
[ 722.610115] [<ffffffff813d5595>] ? schedule_timeout+0x1c5/0x230
[ 722.610127] [<ffffffff8102f69c>] ? enqueue_task_fair+0x14c/0x190
[ 722.610135] [<ffffffff8102db27>] ? task_rq_lock+0x47/0x90
[ 722.610142] [<ffffffff813d4562>] ? wait_for_common+0xd2/0x180
[ 722.610148] [<ffffffff810339b0>] ? try_to_wake_up+0x2c0/0x2c0
[ 722.610156] [<ffffffff8104cb94>] ? flush_work+0x24/0x30
[ 722.610162] [<ffffffff8104be10>] ? do_work_for_cpu+0x20/0x20
[ 722.610168] [<ffffffff8104e0ab>] ? schedule_on_each_cpu+0xab/0xe0
[ 722.610175] [<ffffffff810b5c05>] ? ksm_scan_thread+0x7f5/0xc20
[ 722.610182] [<ffffffff81052a20>] ? wake_up_bit+0x40/0x40
[ 722.610188] [<ffffffff810b5410>] ?
try_to_merge_with_ksm_page+0x570/0x570
[ 722.610194] [<ffffffff810b5410>] ?
try_to_merge_with_ksm_page+0x570/0x570
[ 722.610201] [<ffffffff810525b6>] ? kthread+0x96/0xa0
[ 722.610208] [<ffffffff813d8294>] ? kernel_thread_helper+0x4/0x10
[ 722.610215] [<ffffffff81052520>] ? kthread_worker_fn+0x120/0x120
[ 722.610221] [<ffffffff813d8290>] ? gs_change+0xb/0xb
[ 722.610226] INFO: task fsnotify_mark:662 blocked for more than 120
seconds.
[ 722.610230] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[ 722.610235] fsnotify_mark D 0000000000000000 0 662 2
0x00000000
[ 722.610241] ffff88041c31d860 0000000000000046 0000000000000000
0000000000000000
[ 722.610248] ffffffff81593020 ffff88041c365fd8 0000000000004000
ffff88041c364010
[ 722.610255] ffff88041c365fd8 ffff88041c31d860 0000000000000000
0000000000000000
[ 722.610261] Call Trace:
[ 722.610268] [<ffffffff81035521>] ? load_balance+0x91/0x5e0
[ 722.610275] [<ffffffff813d5595>] ? schedule_timeout+0x1c5/0x230
[ 722.610282] [<ffffffff8102f323>] ? pick_next_task_fair+0x103/0x190
[ 722.610289] [<ffffffff813d4a6d>] ? schedule+0x28d/0x910
[ 722.610295] [<ffffffff813d4562>] ? wait_for_common+0xd2/0x180
[ 722.610301] [<ffffffff810339b0>] ? try_to_wake_up+0x2c0/0x2c0
[ 722.610308] [<ffffffff81074420>] ? synchronize_rcu_bh+0x50/0x50
[ 722.610313] [<ffffffff8107446a>] ? synchronize_sched+0x4a/0x50
[ 722.610319] [<ffffffff8104f940>] ? find_ge_pid+0x40/0x40
[ 722.610326] [<ffffffff8105742b>] ? __synchronize_srcu+0x5b/0xc0
[ 722.610335] [<ffffffff810f5f63>] ? fsnotify_mark_destroy+0x83/0x150
[ 722.610342] [<ffffffff81052a20>] ? wake_up_bit+0x40/0x40
[ 722.610350] [<ffffffff810f5ee0>] ?
fsnotify_set_mark_ignored_mask_locked+0x20/0x20
[ 722.610358] [<ffffffff810f5ee0>] ?
fsnotify_set_mark_ignored_mask_locked+0x20/0x20
[ 722.610365] [<ffffffff810525b6>] ? kthread+0x96/0xa0
[ 722.610371] [<ffffffff813d8294>] ? kernel_thread_helper+0x4/0x10
[ 722.610378] [<ffffffff81052520>] ? kthread_worker_fn+0x120/0x120
[ 722.610384] [<ffffffff813d8290>] ? gs_change+0xb/0xb
[ 722.610392] INFO: task jbd2/md0-8:2528 blocked for more than 120 seconds.
[ 722.610396] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[ 722.610400] jbd2/md0-8 D 0000000000000005 0 2528 2
0x00000000
[ 722.610407] ffff88041d91c570 0000000000000046 ffff88041baeea28
ffffea0000000000
[ 722.610413] ffff88041d91a5e0 ffff880419f9bfd8 0000000000004000
ffff880419f9a010
[ 722.610420] ffff880419f9bfd8 ffff88041d91c570 ffff88041baeea28
ffffffff810b8495
[ 722.610426] Call Trace:
[ 722.610432] [<ffffffff810b8495>] ? init_object+0x85/0xa0
[ 722.610438] [<ffffffff810b8916>] ? free_debug_processing+0x196/0x250
[ 722.610447] [<ffffffff8105adae>] ? ktime_get_ts+0x6e/0xf0
[ 722.610455] [<ffffffff810810f0>] ? __lock_page+0x70/0x70
[ 722.610461] [<ffffffff813d5174>] ? io_schedule+0x84/0xd0
[ 722.610469] [<ffffffff811d4403>] ?
radix_tree_gang_lookup_tag_slot+0x93/0xf0
[ 722.610476] [<ffffffff810810f9>] ? sleep_on_page+0x9/0x10
[ 722.610482] [<ffffffff813d57df>] ? __wait_on_bit+0x4f/0x80
[ 722.610489] [<ffffffff810812eb>] ? wait_on_page_bit+0x6b/0x80
[ 722.610496] [<ffffffff81052a50>] ? autoremove_wake_function+0x30/0x30
[ 722.610504] [<ffffffff8108a458>] ? pagevec_lookup_tag+0x18/0x20
[ 722.610509] [<ffffffff81081f2a>] ? filemap_fdatawait_range+0xfa/0x180
[ 722.610518] [<ffffffff811be09f>] ? submit_bio+0x6f/0xf0
[ 722.610526] [<ffffffff81176276>] ?
jbd2_journal_commit_transaction+0x796/0x1270
[ 722.610536] [<ffffffff81179ed1>] ? kjournald2+0xb1/0x1e0
[ 722.610542] [<ffffffff81052a20>] ? wake_up_bit+0x40/0x40
[ 722.610549] [<ffffffff81179e20>] ? commit_timeout+0x10/0x10
[ 722.610556] [<ffffffff81179e20>] ? commit_timeout+0x10/0x10
[ 722.610563] [<ffffffff810525b6>] ? kthread+0x96/0xa0
[ 722.610569] [<ffffffff813d8294>] ? kernel_thread_helper+0x4/0x10
[ 722.610576] [<ffffffff81052520>] ? kthread_worker_fn+0x120/0x120
[ 722.610582] [<ffffffff813d8290>] ? gs_change+0xb/0xb
[ 722.610591] INFO: task nfsd:4326 blocked for more than 120 seconds.
[ 722.610594] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[ 722.610598] nfsd D 0000000000000000 0 4326 2
0x00000000
[ 722.610605] ffff88041b604bc0 0000000000000046 0000000000000016
ffffffff00000000
[ 722.610611] ffff88041d8fa5e0 ffff88041c527fd8 0000000000004000
ffff88041c526010
[ 722.610617] ffff88041c527fd8 ffff88041b604bc0 000000000000009c
ffff880000000000
[ 722.610623] Call Trace:
[ 722.610630] [<ffffffff8101e845>] ? amd_flush_garts+0x105/0x140
[ 722.610637] [<ffffffff8101fa80>] ? gart_map_sg+0x480/0x480
[ 722.610643] [<ffffffff8101f5d3>] ? flush_gart+0x23/0x50
[ 722.610650] [<ffffffff81080f98>] ? find_get_page+0x18/0x90
[ 722.610657] [<ffffffff81174b95>] ? do_get_write_access+0x265/0x4a0
[ 722.610665] [<ffffffff81052a50>] ? autoremove_wake_function+0x30/0x30
[ 722.610672] [<ffffffff81174ef9>] ?
jbd2_journal_get_write_access+0x29/0x50
[ 722.610680] [<ffffffff8115f122>] ?
__ext4_journal_get_write_access+0x32/0x80
[ 722.610689] [<ffffffff81143908>] ? ext4_reserve_inode_write+0x78/0xa0
[ 722.610696] [<ffffffff81143970>] ? ext4_mark_inode_dirty+0x40/0x1e0
[ 722.610703] [<ffffffff81156c0b>] ? ext4_journal_start_sb+0x6b/0x160
[ 722.610711] [<ffffffff81322645>] ? dev_hard_start_xmit+0x305/0x5f0
[ 722.610719] [<ffffffff81352fd0>] ? ip_finish_output2+0x290/0x290
[ 722.610726] [<ffffffff81143c65>] ? ext4_dirty_inode+0x35/0x70
[ 722.610733] [<ffffffff810e4a08>] ? __mark_inode_dirty+0x38/0x210
[ 722.610741] [<ffffffff810d9317>] ? file_update_time+0xf7/0x180
[ 722.610747] [<ffffffff81082458>] ? __generic_file_aio_write+0x1f8/0x430
[ 722.610755] [<ffffffff81373ef9>] ? udp_sendmsg+0x3c9/0x7e0
[ 722.610763] [<ffffffff81314260>] ? sock_alloc_send_pskb+0x1d0/0x320
[ 722.610769] [<ffffffff81082703>] ? generic_file_aio_write+0x73/0xf0
[ 722.610776] [<ffffffff8113f74e>] ? ext4_file_write+0x6e/0x2b0
[ 722.610783] [<ffffffff810da8ac>] ? iget_locked+0x4c/0x140
[ 722.610789] [<ffffffff8119a590>] ? fh_compose+0x4c0/0x4c0
[ 722.610795] [<ffffffff8113f6e0>] ? ext4_llseek+0x110/0x110
[ 722.610803] [<ffffffff810c24e9>] ? do_sync_readv_writev+0xa9/0xf0
[ 722.610811] [<ffffffff810c231b>] ? rw_copy_check_uvector+0x7b/0x140
[ 722.610819] [<ffffffff810c2bcf>] ? do_readv_writev+0xdf/0x210
[ 722.610825] [<ffffffff810b8735>] ? alloc_debug_processing+0x145/0x190
[ 722.610831] [<ffffffff810b9bc5>] ? kmem_cache_alloc+0xa5/0xb0
[ 722.610838] [<ffffffff8113f463>] ? ext4_file_open+0x63/0x180
[ 722.610844] [<ffffffff8119c21d>] ? nfsd_vfs_write+0xed/0x3a0
[ 722.610851] [<ffffffff810c1127>] ? __dentry_open+0x1f7/0x2b0
[ 722.610857] [<ffffffff8119c892>] ? nfsd_open+0xf2/0x1b0
[ 722.610862] [<ffffffff8119cd34>] ? nfsd_write+0xf4/0x110
[ 722.610868] [<ffffffff81199930>] ? nfsd_proc_write+0xb0/0x120
[ 722.610876] [<ffffffff811971c5>] ? nfsd_dispatch+0xf5/0x230
[ 722.610882] [<ffffffff813b456f>] ? svc_process+0x4af/0x820
[ 722.610887] [<ffffffff810339b0>] ? try_to_wake_up+0x2c0/0x2c0
[ 722.610894] [<ffffffff811977a0>] ? nfsd_svc+0x1b0/0x1b0
[ 722.610901] [<ffffffff8119784d>] ? nfsd+0xad/0x150
[ 722.610907] [<ffffffff810525b6>] ? kthread+0x96/0xa0
[ 722.610913] [<ffffffff813d8294>] ? kernel_thread_helper+0x4/0x10
[ 722.610921] [<ffffffff81052520>] ? kthread_worker_fn+0x120/0x120
[ 722.610926] [<ffffffff813d8290>] ? gs_change+0xb/0xb
[ 765.241265]
=============================================================================
[ 765.241267] BUG kmalloc-512: Object padding overwritten
[ 765.241268]
-----------------------------------------------------------------------------
[ 765.241269]
[ 765.241271] INFO: 0xffff8804168c1486-0xffff8804168c1487. First byte
0x0 instead of 0x5a
[ 765.241277] INFO: Allocated in bio_kmalloc+0x2d/0x70 age=73010 cpu=5
pid=2495
[ 765.241281] INFO: Freed in r10buf_pool_free+0x71/0x90 age=72999 cpu=5
pid=1728
[ 765.241283] INFO: Slab 0xffffea000e4eea00 objects=28 used=13
fp=0xffff8804168c1240 flags=0x80000000000040c1
[ 765.241284] INFO: Object 0xffff8804168c1240 @offset=4672
fp=0xffff8804168c0ff8
[ 765.241285]
[ 765.241286] Bytes b4 0xffff8804168c1230: 95 98 ff ff 00 00 00 00 5a
5a 5a 5a 5a 5a 5a 5a ..ÿÿ....ZZZZZZZZ
[ 765.241291] Object 0xffff8804168c1240: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 765.241294] Object 0xffff8804168c1250: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 765.241298] Object 0xffff8804168c1260: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 765.241301] Object 0xffff8804168c1270: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 765.241305] Object 0xffff8804168c1280: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 765.241308] Object 0xffff8804168c1290: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 765.241312] Object 0xffff8804168c12a0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 765.241315] Object 0xffff8804168c12b0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 765.241319] Object 0xffff8804168c12c0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 765.241322] Object 0xffff8804168c12d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 765.241326] Object 0xffff8804168c12e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 765.241329] Object 0xffff8804168c12f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 765.241332] Object 0xffff8804168c1300: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 765.241336] Object 0xffff8804168c1310: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 765.241339] Object 0xffff8804168c1320: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 765.241343] Object 0xffff8804168c1330: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 765.241346] Object 0xffff8804168c1340: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 765.241350] Object 0xffff8804168c1350: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 765.241353] Object 0xffff8804168c1360: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 765.241357] Object 0xffff8804168c1370: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 765.241360] Object 0xffff8804168c1380: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 765.241364] Object 0xffff8804168c1390: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 765.241367] Object 0xffff8804168c13a0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 765.241371] Object 0xffff8804168c13b0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 765.241374] Object 0xffff8804168c13c0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 765.241378] Object 0xffff8804168c13d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 765.241381] Object 0xffff8804168c13e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 765.241385] Object 0xffff8804168c13f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 765.241388] Object 0xffff8804168c1400: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 765.241392] Object 0xffff8804168c1410: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 765.241395] Object 0xffff8804168c1420: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 765.241399] Object 0xffff8804168c1430: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b a5 kkkkkkkkkkkkkkk¥
[ 765.241402] Redzone 0xffff8804168c1440: bb bb bb bb bb bb bb bb
»»»»»»»»
[ 765.241406] Padding 0xffff8804168c1480: 5a 5a 5a 5a 5a 5a 00 00
ZZZZZZ..
[ 765.241410] Pid: 7147, comm: apcaccess Not tainted 2.6.39 #2
[ 765.241411] Call Trace:
[ 765.241416] [<ffffffff810b7ccd>] ? check_bytes_and_report+0x10d/0x150
[ 765.241419] [<ffffffff8110773c>] ? load_elf_binary+0xa6c/0x1c00
[ 765.241421] [<ffffffff810b7db9>] ? check_object+0xa9/0x260
[ 765.241423] [<ffffffff8110773c>] ? load_elf_binary+0xa6c/0x1c00
[ 765.241424] [<ffffffff810b86f4>] ? alloc_debug_processing+0x104/0x190
[ 765.241426] [<ffffffff810b9ac2>] ? T.912+0x272/0x2d0
[ 765.241428] [<ffffffff810ba59d>] ? __kmalloc+0x10d/0x160
[ 765.241429] [<ffffffff8110773c>] ? load_elf_binary+0xa6c/0x1c00
[ 765.241432] [<ffffffff8109bd9d>] ? __get_user_pages+0x17d/0x530
[ 765.241434] [<ffffffff810c9556>] ? get_arg_page+0x56/0x100
[ 765.241437] [<ffffffff811da4ad>] ? strnlen_user+0x2d/0x80
[ 765.241439] [<ffffffff810c8070>] ? search_binary_handler+0x90/0x240
[ 765.241440] [<ffffffff810c9e9f>] ? do_execve+0x22f/0x2f0
[ 765.241443] [<ffffffff810094a6>] ? sys_execve+0x36/0x60
[ 765.241446] [<ffffffff813d78dc>] ? stub_execve+0x6c/0xc0
[ 765.241448] FIX kmalloc-512: Restoring
0xffff8804168c1486-0xffff8804168c1487=0x5a
[ 765.241449]
[ 789.536436]
=============================================================================
[ 789.536444] BUG kmalloc-2048: Object padding overwritten
[ 789.536448]
-----------------------------------------------------------------------------
[ 789.536451]
[ 789.536456] INFO: 0xffff8802c8b1a11e-0xffff8802c8b1a11f. First byte
0x0 instead of 0x5a
[ 789.536471] INFO: Allocated in sk_stream_alloc_skb+0x3a/0x110 age=63
cpu=3 pid=5169
[ 789.536482] INFO: Freed in __kfree_skb+0x11/0x90 age=63 cpu=5 pid=0
[ 789.536488] INFO: Slab 0xffffea0009be6d40 objects=15 used=4
fp=0xffff8802c8b198d8 flags=0x80000000000040c1
[ 789.536494] INFO: Object 0xffff8802c8b198d8 @offset=6360
fp=0xffff8802c8b1a968
[ 789.536497]
[ 789.536501] Bytes b4 0xffff8802c8b198c8: cc be 00 00 01 00 00 00 5a
5a 5a 5a 5a 5a 5a 5a ÌŸ......ZZZZZZZZ
[ 789.536517] Object 0xffff8802c8b198d8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.536532] Object 0xffff8802c8b198e8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.536546] Object 0xffff8802c8b198f8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.536560] Object 0xffff8802c8b19908: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.536575] Object 0xffff8802c8b19918: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.536589] Object 0xffff8802c8b19928: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.536603] Object 0xffff8802c8b19938: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.536617] Object 0xffff8802c8b19948: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.536631] Object 0xffff8802c8b19958: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.536645] Object 0xffff8802c8b19968: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.536660] Object 0xffff8802c8b19978: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.536674] Object 0xffff8802c8b19988: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.536688] Object 0xffff8802c8b19998: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.536702] Object 0xffff8802c8b199a8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.536716] Object 0xffff8802c8b199b8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.536730] Object 0xffff8802c8b199c8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.536745] Object 0xffff8802c8b199d8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.536759] Object 0xffff8802c8b199e8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.536773] Object 0xffff8802c8b199f8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.536787] Object 0xffff8802c8b19a08: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.536801] Object 0xffff8802c8b19a18: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.536816] Object 0xffff8802c8b19a28: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.536830] Object 0xffff8802c8b19a38: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.536844] Object 0xffff8802c8b19a48: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.536858] Object 0xffff8802c8b19a58: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.536872] Object 0xffff8802c8b19a68: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.536887] Object 0xffff8802c8b19a78: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.536901] Object 0xffff8802c8b19a88: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.536915] Object 0xffff8802c8b19a98: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.536929] Object 0xffff8802c8b19aa8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.536943] Object 0xffff8802c8b19ab8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.536958] Object 0xffff8802c8b19ac8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.536972] Object 0xffff8802c8b19ad8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.536986] Object 0xffff8802c8b19ae8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537000] Object 0xffff8802c8b19af8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537014] Object 0xffff8802c8b19b08: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537029] Object 0xffff8802c8b19b18: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537043] Object 0xffff8802c8b19b28: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537057] Object 0xffff8802c8b19b38: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537071] Object 0xffff8802c8b19b48: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537085] Object 0xffff8802c8b19b58: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537100] Object 0xffff8802c8b19b68: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537114] Object 0xffff8802c8b19b78: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537128] Object 0xffff8802c8b19b88: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537143] Object 0xffff8802c8b19b98: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537158] Object 0xffff8802c8b19ba8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537172] Object 0xffff8802c8b19bb8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537187] Object 0xffff8802c8b19bc8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537201] Object 0xffff8802c8b19bd8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537215] Object 0xffff8802c8b19be8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537229] Object 0xffff8802c8b19bf8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537244] Object 0xffff8802c8b19c08: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537258] Object 0xffff8802c8b19c18: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537272] Object 0xffff8802c8b19c28: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537286] Object 0xffff8802c8b19c38: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537301] Object 0xffff8802c8b19c48: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537315] Object 0xffff8802c8b19c58: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537329] Object 0xffff8802c8b19c68: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537343] Object 0xffff8802c8b19c78: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537357] Object 0xffff8802c8b19c88: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537372] Object 0xffff8802c8b19c98: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537386] Object 0xffff8802c8b19ca8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537400] Object 0xffff8802c8b19cb8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537414] Object 0xffff8802c8b19cc8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537428] Object 0xffff8802c8b19cd8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537443] Object 0xffff8802c8b19ce8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537457] Object 0xffff8802c8b19cf8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537471] Object 0xffff8802c8b19d08: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537485] Object 0xffff8802c8b19d18: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537500] Object 0xffff8802c8b19d28: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537514] Object 0xffff8802c8b19d38: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537528] Object 0xffff8802c8b19d48: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537542] Object 0xffff8802c8b19d58: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537557] Object 0xffff8802c8b19d68: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537571] Object 0xffff8802c8b19d78: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537585] Object 0xffff8802c8b19d88: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537599] Object 0xffff8802c8b19d98: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537613] Object 0xffff8802c8b19da8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537628] Object 0xffff8802c8b19db8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537642] Object 0xffff8802c8b19dc8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537656] Object 0xffff8802c8b19dd8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537670] Object 0xffff8802c8b19de8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537685] Object 0xffff8802c8b19df8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537699] Object 0xffff8802c8b19e08: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537713] Object 0xffff8802c8b19e18: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537727] Object 0xffff8802c8b19e28: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537742] Object 0xffff8802c8b19e38: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537756] Object 0xffff8802c8b19e48: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537770] Object 0xffff8802c8b19e58: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537784] Object 0xffff8802c8b19e68: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537799] Object 0xffff8802c8b19e78: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537813] Object 0xffff8802c8b19e88: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537827] Object 0xffff8802c8b19e98: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537841] Object 0xffff8802c8b19ea8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537856] Object 0xffff8802c8b19eb8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537870] Object 0xffff8802c8b19ec8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537884] Object 0xffff8802c8b19ed8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537898] Object 0xffff8802c8b19ee8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537912] Object 0xffff8802c8b19ef8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537927] Object 0xffff8802c8b19f08: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537941] Object 0xffff8802c8b19f18: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537955] Object 0xffff8802c8b19f28: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537969] Object 0xffff8802c8b19f38: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537984] Object 0xffff8802c8b19f48: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537998] Object 0xffff8802c8b19f58: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.538012] Object 0xffff8802c8b19f68: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.538026] Object 0xffff8802c8b19f78: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.538041] Object 0xffff8802c8b19f88: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.538055] Object 0xffff8802c8b19f98: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.538069] Object 0xffff8802c8b19fa8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.538083] Object 0xffff8802c8b19fb8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.538097] Object 0xffff8802c8b19fc8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.538112] Object 0xffff8802c8b19fd8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.538126] Object 0xffff8802c8b19fe8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.538140] Object 0xffff8802c8b19ff8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.538154] Object 0xffff8802c8b1a008: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.538169] Object 0xffff8802c8b1a018: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.538183] Object 0xffff8802c8b1a028: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.538197] Object 0xffff8802c8b1a038: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.538211] Object 0xffff8802c8b1a048: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.538226] Object 0xffff8802c8b1a058: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.538240] Object 0xffff8802c8b1a068: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.538254] Object 0xffff8802c8b1a078: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.538268] Object 0xffff8802c8b1a088: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.538283] Object 0xffff8802c8b1a098: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.538297] Object 0xffff8802c8b1a0a8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.538311] Object 0xffff8802c8b1a0b8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.538325] Object 0xffff8802c8b1a0c8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b a5 kkkkkkkkkkkkkkk¥
[ 789.538340] Redzone 0xffff8802c8b1a0d8: bb bb bb bb bb bb bb bb
»»»»»»»»
[ 789.538353] Padding 0xffff8802c8b1a118: 5a 5a 5a 5a 5a 5a 00 00
ZZZZZZ..
[ 789.538369] Pid: 5247, comm: qemu Not tainted 2.6.39 #2
[ 789.538373] Call Trace:
[ 789.538385] [<ffffffff810b7ccd>] ? check_bytes_and_report+0x10d/0x150
[ 789.538394] [<ffffffff81314260>] ? sock_alloc_send_pskb+0x1d0/0x320
[ 789.538400] [<ffffffff810b7db9>] ? check_object+0xa9/0x260
[ 789.538407] [<ffffffff81314260>] ? sock_alloc_send_pskb+0x1d0/0x320
[ 789.538414] [<ffffffff810b86f4>] ? alloc_debug_processing+0x104/0x190
[ 789.538420] [<ffffffff810b9ac2>] ? T.912+0x272/0x2d0
[ 789.538427] [<ffffffff810bb70d>] ? __kmalloc_track_caller+0x10d/0x160
[ 789.538434] [<ffffffff813198c2>] ? __alloc_skb+0x72/0x160
[ 789.538441] [<ffffffff81314260>] ? sock_alloc_send_pskb+0x1d0/0x320
[ 789.538450] [<ffffffff8105af7b>] ? getnstimeofday+0x4b/0xd0
[ 789.538457] [<ffffffff81320d32>] ? netif_rx+0xb2/0x120
[ 789.538465] [<ffffffff812b393e>] ? tun_chr_aio_write+0x18e/0x4e0
[ 789.538472] [<ffffffff812b37b0>] ? tun_sendmsg+0x4d0/0x4d0
[ 789.538481] [<ffffffff810c24e9>] ? do_sync_readv_writev+0xa9/0xf0
[ 789.538489] [<ffffffff810c231b>] ? rw_copy_check_uvector+0x7b/0x140
[ 789.538496] [<ffffffff810c2bcf>] ? do_readv_writev+0xdf/0x210
[ 789.538504] [<ffffffff810c2e7e>] ? sys_writev+0x4e/0xc0
[ 789.538513] [<ffffffff813d753b>] ? system_call_fastpath+0x16/0x1b
[ 789.538519] FIX kmalloc-2048: Restoring
0xffff8802c8b1a11e-0xffff8802c8b1a11f=0x5a
[ 789.538522]
[ 825.601291]
=============================================================================
[ 825.601298] BUG kmalloc-512: Object padding overwritten
[ 825.601302]
-----------------------------------------------------------------------------
[ 825.601306]
[ 825.601311] INFO: 0xffff88041411efe6-0xffff88041411efe7. First byte
0x0 instead of 0x5a
[ 825.601327] INFO: Allocated in sock_alloc_send_pskb+0x1d0/0x320
age=36742 cpu=3 pid=4846
[ 825.601338] INFO: Freed in __kfree_skb+0x11/0x90 age=36742 cpu=3 pid=3781
[ 825.601344] INFO: Slab 0xffffea000e463e20 objects=28 used=11
fp=0xffff88041411eda0 flags=0x80000000000040c1
[ 825.601350] INFO: Object 0xffff88041411eda0 @offset=11680
fp=0xffff88041411cdb0
[ 825.601353]
[ 825.601357] Bytes b4 0xffff88041411ed90: e2 96 ff ff 00 00 00 00 5a
5a 5a 5a 5a 5a 5a 5a â.ÿÿ....ZZZZZZZZ
[ 825.601373] Object 0xffff88041411eda0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 825.601388] Object 0xffff88041411edb0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 825.601402] Object 0xffff88041411edc0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 825.601416] Object 0xffff88041411edd0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 825.601430] Object 0xffff88041411ede0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 825.601444] Object 0xffff88041411edf0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 825.601459] Object 0xffff88041411ee00: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 825.601473] Object 0xffff88041411ee10: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 825.601487] Object 0xffff88041411ee20: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 825.601501] Object 0xffff88041411ee30: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 825.601515] Object 0xffff88041411ee40: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 825.601529] Object 0xffff88041411ee50: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 825.601543] Object 0xffff88041411ee60: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 825.601557] Object 0xffff88041411ee70: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 825.601571] Object 0xffff88041411ee80: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 825.601586] Object 0xffff88041411ee90: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 825.601600] Object 0xffff88041411eea0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 825.601614] Object 0xffff88041411eeb0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 825.601628] Object 0xffff88041411eec0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 825.601642] Object 0xffff88041411eed0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 825.601656] Object 0xffff88041411eee0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 825.601670] Object 0xffff88041411eef0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 825.601684] Object 0xffff88041411ef00: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 825.601699] Object 0xffff88041411ef10: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 825.601713] Object 0xffff88041411ef20: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 825.601727] Object 0xffff88041411ef30: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 825.601741] Object 0xffff88041411ef40: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 825.601755] Object 0xffff88041411ef50: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 825.601769] Object 0xffff88041411ef60: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 825.601783] Object 0xffff88041411ef70: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 825.601797] Object 0xffff88041411ef80: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 825.601812] Object 0xffff88041411ef90: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b a5 kkkkkkkkkkkkkkk¥
[ 825.601826] Redzone 0xffff88041411efa0: bb bb bb bb bb bb bb bb
»»»»»»»»
[ 825.601839] Padding 0xffff88041411efe0: 5a 5a 5a 5a 5a 5a 00 00
ZZZZZZ..
[ 825.601855] Pid: 7353, comm: apcaccess Not tainted 2.6.39 #2
[ 825.601859] Call Trace:
[ 825.601871] [<ffffffff810b7ccd>] ? check_bytes_and_report+0x10d/0x150
[ 825.601881] [<ffffffff8110773c>] ? load_elf_binary+0xa6c/0x1c00
[ 825.601887] [<ffffffff810b7db9>] ? check_object+0xa9/0x260
[ 825.601895] [<ffffffff8110773c>] ? load_elf_binary+0xa6c/0x1c00
[ 825.601902] [<ffffffff810b86f4>] ? alloc_debug_processing+0x104/0x190
[ 825.601908] [<ffffffff810b9ac2>] ? T.912+0x272/0x2d0
[ 825.601914] [<ffffffff810ba59d>] ? __kmalloc+0x10d/0x160
[ 825.601922] [<ffffffff8110773c>] ? load_elf_binary+0xa6c/0x1c00
[ 825.601930] [<ffffffff8109bd9d>] ? __get_user_pages+0x17d/0x530
[ 825.601938] [<ffffffff810c9556>] ? get_arg_page+0x56/0x100
[ 825.601945] [<ffffffff810c8070>] ? search_binary_handler+0x90/0x240
[ 825.601951] [<ffffffff810c9e9f>] ? do_execve+0x22f/0x2f0
[ 825.601959] [<ffffffff810094a6>] ? sys_execve+0x36/0x60
[ 825.601968] [<ffffffff813d78dc>] ? stub_execve+0x6c/0xc0
[ 825.601975] FIX kmalloc-512: Restoring
0xffff88041411efe6-0xffff88041411efe7=0x5a
[ 825.601978]
[ 842.610057] INFO: task ksmd:552 blocked for more than 120 seconds.
[ 842.610063] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[ 842.610069] ksmd D 0000000000000000 0 552 2
0x00000000
[ 842.610078] ffff88041db68ca0 0000000000000046 000212d01fc11d00
ffff880400000000
[ 842.610086] ffffffff81593020 ffff88041c1a9fd8 0000000000004000
ffff88041c1a8010
[ 842.610093] ffff88041c1a9fd8 ffff88041db68ca0 0000000000000000
0000000200000002
[ 842.610100] Call Trace:
[ 842.610116] [<ffffffff813d5595>] ? schedule_timeout+0x1c5/0x230
[ 842.610128] [<ffffffff8102f69c>] ? enqueue_task_fair+0x14c/0x190
[ 842.610137] [<ffffffff8102db27>] ? task_rq_lock+0x47/0x90
[ 842.610143] [<ffffffff813d4562>] ? wait_for_common+0xd2/0x180
[ 842.610150] [<ffffffff810339b0>] ? try_to_wake_up+0x2c0/0x2c0
[ 842.610157] [<ffffffff8104cb94>] ? flush_work+0x24/0x30
[ 842.610163] [<ffffffff8104be10>] ? do_work_for_cpu+0x20/0x20
[ 842.610170] [<ffffffff8104e0ab>] ? schedule_on_each_cpu+0xab/0xe0
[ 842.610177] [<ffffffff810b5c05>] ? ksm_scan_thread+0x7f5/0xc20
[ 842.610184] [<ffffffff81052a20>] ? wake_up_bit+0x40/0x40
[ 842.610190] [<ffffffff810b5410>] ?
try_to_merge_with_ksm_page+0x570/0x570
[ 842.610196] [<ffffffff810b5410>] ?
try_to_merge_with_ksm_page+0x570/0x570
[ 842.610203] [<ffffffff810525b6>] ? kthread+0x96/0xa0
[ 842.610210] [<ffffffff813d8294>] ? kernel_thread_helper+0x4/0x10
[ 842.610218] [<ffffffff81052520>] ? kthread_worker_fn+0x120/0x120
[ 842.610223] [<ffffffff813d8290>] ? gs_change+0xb/0xb
[ 842.610229] INFO: task fsnotify_mark:662 blocked for more than 120
seconds.
[ 842.610233] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[ 842.610237] fsnotify_mark D 0000000000000000 0 662 2
0x00000000
[ 842.610244] ffff88041c31d860 0000000000000046 0000000000000000
0000000000000000
[ 842.610251] ffffffff81593020 ffff88041c365fd8 0000000000004000
ffff88041c364010
[ 842.610257] ffff88041c365fd8 ffff88041c31d860 0000000000000000
0000000000000000
[ 842.610263] Call Trace:
[ 842.610270] [<ffffffff81035521>] ? load_balance+0x91/0x5e0
[ 842.610278] [<ffffffff813d5595>] ? schedule_timeout+0x1c5/0x230
[ 842.610285] [<ffffffff8102f323>] ? pick_next_task_fair+0x103/0x190
[ 842.610291] [<ffffffff813d4a6d>] ? schedule+0x28d/0x910
[ 842.610297] [<ffffffff813d4562>] ? wait_for_common+0xd2/0x180
[ 842.610303] [<ffffffff810339b0>] ? try_to_wake_up+0x2c0/0x2c0
[ 842.610310] [<ffffffff81074420>] ? synchronize_rcu_bh+0x50/0x50
[ 842.610316] [<ffffffff8107446a>] ? synchronize_sched+0x4a/0x50
[ 842.610322] [<ffffffff8104f940>] ? find_ge_pid+0x40/0x40
[ 842.610329] [<ffffffff8105742b>] ? __synchronize_srcu+0x5b/0xc0
[ 842.610338] [<ffffffff810f5f63>] ? fsnotify_mark_destroy+0x83/0x150
[ 842.610344] [<ffffffff81052a20>] ? wake_up_bit+0x40/0x40
[ 842.610352] [<ffffffff810f5ee0>] ?
fsnotify_set_mark_ignored_mask_locked+0x20/0x20
[ 842.610360] [<ffffffff810f5ee0>] ?
fsnotify_set_mark_ignored_mask_locked+0x20/0x20
[ 842.610367] [<ffffffff810525b6>] ? kthread+0x96/0xa0
[ 842.610373] [<ffffffff813d8294>] ? kernel_thread_helper+0x4/0x10
[ 842.610381] [<ffffffff81052520>] ? kthread_worker_fn+0x120/0x120
[ 842.610386] [<ffffffff813d8290>] ? gs_change+0xb/0xb
[ 842.610395] INFO: task jbd2/md0-8:2528 blocked for more than 120 seconds.
[ 842.610399] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[ 842.610403] jbd2/md0-8 D 0000000000000005 0 2528 2
0x00000000
[ 842.610410] ffff88041d91c570 0000000000000046 ffff88041baeea28
ffffea0000000000
[ 842.610417] ffff88041d91a5e0 ffff880419f9bfd8 0000000000004000
ffff880419f9a010
[ 842.610423] ffff880419f9bfd8 ffff88041d91c570 ffff88041baeea28
ffffffff810b8495
[ 842.610429] Call Trace:
[ 842.610435] [<ffffffff810b8495>] ? init_object+0x85/0xa0
[ 842.610442] [<ffffffff810b8916>] ? free_debug_processing+0x196/0x250
[ 842.610450] [<ffffffff8105adae>] ? ktime_get_ts+0x6e/0xf0
[ 842.610458] [<ffffffff810810f0>] ? __lock_page+0x70/0x70
[ 842.610464] [<ffffffff813d5174>] ? io_schedule+0x84/0xd0
[ 842.610473] [<ffffffff811d4403>] ?
radix_tree_gang_lookup_tag_slot+0x93/0xf0
[ 842.610480] [<ffffffff810810f9>] ? sleep_on_page+0x9/0x10
[ 842.610486] [<ffffffff813d57df>] ? __wait_on_bit+0x4f/0x80
[ 842.610492] [<ffffffff810812eb>] ? wait_on_page_bit+0x6b/0x80
[ 842.610499] [<ffffffff81052a50>] ? autoremove_wake_function+0x30/0x30
[ 842.610507] [<ffffffff8108a458>] ? pagevec_lookup_tag+0x18/0x20
[ 842.610512] [<ffffffff81081f2a>] ? filemap_fdatawait_range+0xfa/0x180
[ 842.610521] [<ffffffff811be09f>] ? submit_bio+0x6f/0xf0
[ 842.610530] [<ffffffff81176276>] ?
jbd2_journal_commit_transaction+0x796/0x1270
[ 842.610539] [<ffffffff81179ed1>] ? kjournald2+0xb1/0x1e0
[ 842.610546] [<ffffffff81052a20>] ? wake_up_bit+0x40/0x40
[ 842.610553] [<ffffffff81179e20>] ? commit_timeout+0x10/0x10
[ 842.610560] [<ffffffff81179e20>] ? commit_timeout+0x10/0x10
[ 842.610566] [<ffffffff810525b6>] ? kthread+0x96/0xa0
[ 842.610572] [<ffffffff813d8294>] ? kernel_thread_helper+0x4/0x10
[ 842.610580] [<ffffffff81052520>] ? kthread_worker_fn+0x120/0x120
[ 842.610585] [<ffffffff813d8290>] ? gs_change+0xb/0xb
[ 842.610594] INFO: task nfsd:4326 blocked for more than 120 seconds.
[ 842.610598] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[ 842.610602] nfsd D 0000000000000000 0 4326 2
0x00000000
[ 842.610608] ffff88041b604bc0 0000000000000046 0000000000000016
ffffffff00000000
[ 842.610614] ffff88041d8fa5e0 ffff88041c527fd8 0000000000004000
ffff88041c526010
[ 842.610621] ffff88041c527fd8 ffff88041b604bc0 000000000000009c
ffff880000000000
[ 842.610626] Call Trace:
[ 842.610634] [<ffffffff8101e845>] ? amd_flush_garts+0x105/0x140
[ 842.610640] [<ffffffff8101fa80>] ? gart_map_sg+0x480/0x480
[ 842.610646] [<ffffffff8101f5d3>] ? flush_gart+0x23/0x50
[ 842.610653] [<ffffffff81080f98>] ? find_get_page+0x18/0x90
[ 842.610660] [<ffffffff81174b95>] ? do_get_write_access+0x265/0x4a0
[ 842.610668] [<ffffffff81052a50>] ? autoremove_wake_function+0x30/0x30
[ 842.610675] [<ffffffff81174ef9>] ?
jbd2_journal_get_write_access+0x29/0x50
[ 842.610684] [<ffffffff8115f122>] ?
__ext4_journal_get_write_access+0x32/0x80
[ 842.610692] [<ffffffff81143908>] ? ext4_reserve_inode_write+0x78/0xa0
[ 842.610700] [<ffffffff81143970>] ? ext4_mark_inode_dirty+0x40/0x1e0
[ 842.610706] [<ffffffff81156c0b>] ? ext4_journal_start_sb+0x6b/0x160
[ 842.610715] [<ffffffff81322645>] ? dev_hard_start_xmit+0x305/0x5f0
[ 842.610723] [<ffffffff81352fd0>] ? ip_finish_output2+0x290/0x290
[ 842.610730] [<ffffffff81143c65>] ? ext4_dirty_inode+0x35/0x70
[ 842.610738] [<ffffffff810e4a08>] ? __mark_inode_dirty+0x38/0x210
[ 842.610745] [<ffffffff810d9317>] ? file_update_time+0xf7/0x180
[ 842.610751] [<ffffffff81082458>] ? __generic_file_aio_write+0x1f8/0x430
[ 842.610760] [<ffffffff81373ef9>] ? udp_sendmsg+0x3c9/0x7e0
[ 842.610767] [<ffffffff81314260>] ? sock_alloc_send_pskb+0x1d0/0x320
[ 842.610774] [<ffffffff81082703>] ? generic_file_aio_write+0x73/0xf0
[ 842.610781] [<ffffffff8113f74e>] ? ext4_file_write+0x6e/0x2b0
[ 842.610788] [<ffffffff810da8ac>] ? iget_locked+0x4c/0x140
[ 842.610794] [<ffffffff8119a590>] ? fh_compose+0x4c0/0x4c0
[ 842.610800] [<ffffffff8113f6e0>] ? ext4_llseek+0x110/0x110
[ 842.610808] [<ffffffff810c24e9>] ? do_sync_readv_writev+0xa9/0xf0
[ 842.610816] [<ffffffff810c231b>] ? rw_copy_check_uvector+0x7b/0x140
[ 842.610824] [<ffffffff810c2bcf>] ? do_readv_writev+0xdf/0x210
[ 842.610830] [<ffffffff810b8735>] ? alloc_debug_processing+0x145/0x190
[ 842.610836] [<ffffffff810b9bc5>] ? kmem_cache_alloc+0xa5/0xb0
[ 842.610843] [<ffffffff8113f463>] ? ext4_file_open+0x63/0x180
[ 842.610849] [<ffffffff8119c21d>] ? nfsd_vfs_write+0xed/0x3a0
[ 842.610856] [<ffffffff810c1127>] ? __dentry_open+0x1f7/0x2b0
[ 842.610862] [<ffffffff8119c892>] ? nfsd_open+0xf2/0x1b0
[ 842.610867] [<ffffffff8119cd34>] ? nfsd_write+0xf4/0x110
[ 842.610873] [<ffffffff81199930>] ? nfsd_proc_write+0xb0/0x120
[ 842.610880] [<ffffffff811971c5>] ? nfsd_dispatch+0xf5/0x230
[ 842.610886] [<ffffffff813b456f>] ? svc_process+0x4af/0x820
[ 842.610892] [<ffffffff810339b0>] ? try_to_wake_up+0x2c0/0x2c0
[ 842.610899] [<ffffffff811977a0>] ? nfsd_svc+0x1b0/0x1b0
[ 842.610906] [<ffffffff8119784d>] ? nfsd+0xad/0x150
[ 842.610912] [<ffffffff810525b6>] ? kthread+0x96/0xa0
[ 842.610918] [<ffffffff813d8294>] ? kernel_thread_helper+0x4/0x10
[ 842.610926] [<ffffffff81052520>] ? kthread_worker_fn+0x120/0x120
[ 842.610931] [<ffffffff813d8290>] ? gs_change+0xb/0xb
[ 854.241450]
=============================================================================
[ 854.241459] BUG kmalloc-1024: Object padding overwritten
[ 854.241463]
-----------------------------------------------------------------------------
[ 854.241466]
[ 854.241471] INFO: 0xffff8802c86059e6-0xffff8802c86059e7. First byte
0x0 instead of 0x5a
[ 854.241488] INFO: Allocated in __blockdev_direct_IO+0x16c/0xa90
age=518 cpu=5 pid=7094
[ 854.241500] INFO: Freed in __kfree_skb+0x11/0x90 age=519 cpu=5 pid=5247
[ 854.241507] INFO: Slab 0xffffea0009bd5000 objects=29 used=22
fp=0xffff8802c8606f50 flags=0x80000000000040c1
[ 854.241513] INFO: Object 0xffff8802c86055a0 @offset=21920
fp=0xffff8802c86066c0
[ 854.241516]
[ 854.241520] Bytes b4 0xffff8802c8605590: 00 00 00 00 00 00 00 00 5a
5a 5a 5a 5a 5a 5a 5a ........ZZZZZZZZ
[ 854.241536] Object 0xffff8802c86055a0: 00 00 00 00 00 00 00 00 e0
61 29 fa 03 88 ff ff ........àa)ú..ÿÿ
[ 854.241551] Object 0xffff8802c86055b0: 00 00 00 00 00 00 00 00 00
00 41 ce 04 00 00 00 ..........AÎ....
[ 854.241565] Object 0xffff8802c86055c0: 03 00 00 00 0c 00 00 00 00
00 00 00 01 00 00 00 ................
[ 854.241579] Object 0xffff8802c86055d0: 00 00 00 00 00 00 00 00 00
60 00 00 00 00 00 00 .........`......
[ 854.241593] Object 0xffff8802c86055e0: b7 2c 0a 00 00 00 00 00 00
00 00 00 00 00 00 00 ·,..............
[ 854.241607] Object 0xffff8802c86055f0: b7 2c 0a 00 00 00 00 00 00
00 00 00 00 00 00 00 ·,..............
[ 854.241621] Object 0xffff8802c8605600: 01 00 00 00 00 00 00 00 f0
77 14 81 ff ff ff ff ........ðw..ÿÿÿÿ
[ 854.241635] Object 0xffff8802c8605610: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 854.241649] Object 0xffff8802c8605620: 00 00 00 00 00 00 00 00 b7
54 3b 02 00 00 00 00 ........·T;.....
[ 854.241663] Object 0xffff8802c8605630: b7 54 3b 02 00 00 00 00 20
00 00 00 00 00 00 00 ·T;.............
[ 854.241677] Object 0xffff8802c8605640: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 854.241691] Object 0xffff8802c8605650: b6 54 3b 02 00 00 00 00 00
10 00 00 00 00 00 00 ¶T;.............
[ 854.241705] Object 0xffff8802c8605660: 00 00 00 00 00 00 00 00 00
80 8e 1a 04 88 ff ff ..............ÿÿ
[ 854.241719] Object 0xffff8802c8605670: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 854.241733] Object 0xffff8802c8605680: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 854.241747] Object 0xffff8802c8605690: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 854.241761] Object 0xffff8802c86056a0: 00 00 00 00 00 00 00 00 00
00 00 00 00 10 00 00 ................
[ 854.241775] Object 0xffff8802c86056b0: b6 54 3b 02 00 00 00 00 00
60 cb a2 00 00 00 00 ¶T;......`Ë¢....
[ 854.241789] Object 0xffff8802c86056c0: 06 06 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 854.241803] Object 0xffff8802c86056d0: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 854.241817] Object 0xffff8802c86056e0: 68 fd 1c c9 02 88 ff ff 00
00 00 00 00 00 00 00 hý.É..ÿÿ........
[ 854.241831] Object 0xffff8802c86056f0: 00 60 00 00 00 00 00 00 01
00 00 00 01 00 00 00 .`..............
[ 854.241845] Object 0xffff8802c8605700: 00 80 c9 dd 39 7f 00 00 01
00 00 00 01 00 00 00 ..ÉÝ9...........
[ 854.241860] Object 0xffff8802c8605710: 00 00 00 00 00 00 00 00 08
81 bb 0c 00 ea ff ff ..........»..êÿÿ
[ 854.241874] Object 0xffff8802c8605720: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.241888] Object 0xffff8802c8605730: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.241902] Object 0xffff8802c8605740: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.241916] Object 0xffff8802c8605750: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.241930] Object 0xffff8802c8605760: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.241944] Object 0xffff8802c8605770: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.241959] Object 0xffff8802c8605780: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.241973] Object 0xffff8802c8605790: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.241987] Object 0xffff8802c86057a0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.242001] Object 0xffff8802c86057b0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.242015] Object 0xffff8802c86057c0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.242029] Object 0xffff8802c86057d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.242043] Object 0xffff8802c86057e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.242058] Object 0xffff8802c86057f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.242072] Object 0xffff8802c8605800: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.242086] Object 0xffff8802c8605810: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.242100] Object 0xffff8802c8605820: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.242114] Object 0xffff8802c8605830: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.242128] Object 0xffff8802c8605840: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.242142] Object 0xffff8802c8605850: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.242157] Object 0xffff8802c8605860: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.242171] Object 0xffff8802c8605870: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.242185] Object 0xffff8802c8605880: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.242199] Object 0xffff8802c8605890: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.242213] Object 0xffff8802c86058a0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.242227] Object 0xffff8802c86058b0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.242241] Object 0xffff8802c86058c0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.242256] Object 0xffff8802c86058d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.242270] Object 0xffff8802c86058e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.242284] Object 0xffff8802c86058f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.242298] Object 0xffff8802c8605900: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.242312] Object 0xffff8802c8605910: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.242326] Object 0xffff8802c8605920: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.242340] Object 0xffff8802c8605930: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.242355] Object 0xffff8802c8605940: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.242369] Object 0xffff8802c8605950: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.242383] Object 0xffff8802c8605960: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.242397] Object 0xffff8802c8605970: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.242411] Object 0xffff8802c8605980: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.242425] Object 0xffff8802c8605990: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b a5 kkkkkkkkkkkkkkk¥
[ 854.242440] Redzone 0xffff8802c86059a0: cc cc cc cc cc cc cc cc
ÌÌÌÌÌÌÌÌ
[ 854.242453] Padding 0xffff8802c86059e0: 5a 5a 5a 5a 5a 5a 00 00
ZZZZZZ..
[ 854.242470] Pid: 7094, comm: qemu Not tainted 2.6.39 #2
[ 854.242474] Call Trace:
[ 854.242485] [<ffffffff810b7ccd>] ? check_bytes_and_report+0x10d/0x150
[ 854.242492] [<ffffffff810b7db9>] ? check_object+0xa9/0x260
[ 854.242500] [<ffffffff810f3126>] ? __blockdev_direct_IO+0xa16/0xa90
[ 854.242507] [<ffffffff810b88b3>] ? free_debug_processing+0x133/0x250
[ 854.242513] [<ffffffff810b8afb>] ? __slab_free+0x12b/0x140
[ 854.242520] [<ffffffff810f3126>] ? __blockdev_direct_IO+0xa16/0xa90
[ 854.242530] [<ffffffff81144a17>] ? ext4_ind_direct_IO+0xf7/0x410
[ 854.242536] [<ffffffff811477f0>] ? noalloc_get_block_write+0x30/0x30
[ 854.242543] [<ffffffff81082029>] ? __filemap_fdatawrite_range+0x49/0x50
[ 854.242550] [<ffffffff810830c3>] ? generic_file_aio_read+0x653/0x6d0
[ 854.242558] [<ffffffff810567a2>] ? hrtimer_cancel+0x12/0x20
[ 854.242565] [<ffffffff81062137>] ? futex_wait+0x197/0x240
[ 854.242572] [<ffffffff81082a70>] ? delete_from_page_cache+0x70/0x70
[ 854.242580] [<ffffffff810c24e9>] ? do_sync_readv_writev+0xa9/0xf0
[ 854.242587] [<ffffffff8106239f>] ? futex_wake+0x10f/0x120
[ 854.242594] [<ffffffff81063f7b>] ? do_futex+0x11b/0xa70
[ 854.242601] [<ffffffff810471cc>] ? T.680+0x13c/0x280
[ 854.242608] [<ffffffff810c231b>] ? rw_copy_check_uvector+0x7b/0x140
[ 854.242616] [<ffffffff810c2bcf>] ? do_readv_writev+0xdf/0x210
[ 854.242624] [<ffffffff810c3003>] ? sys_preadv+0xc3/0xd0
[ 854.242633] [<ffffffff813d753b>] ? system_call_fastpath+0x16/0x1b
[ 854.242640] FIX kmalloc-1024: Restoring
0xffff8802c86059e6-0xffff8802c86059e7=0x5a
[ 854.242643]
[ 860.203918]
=============================================================================
[ 860.203927] BUG kmalloc-1024: Object padding overwritten
[ 860.203931]
-----------------------------------------------------------------------------
[ 860.203934]
[ 860.203939] INFO: 0xffff88041ad15156-0xffff88041ad15157. First byte
0x0 instead of 0x5a
[ 860.203955] INFO: Allocated in __blockdev_direct_IO+0x16c/0xa90
age=1609 cpu=3 pid=7094
[ 860.203965] INFO: Freed in __blockdev_direct_IO+0xa16/0xa90 age=1609
cpu=3 pid=7094
[ 860.203972] INFO: Slab 0xffffea000e5ddb80 objects=29 used=25
fp=0xffff88041ad14d10 flags=0x80000000000040c1
[ 860.203977] INFO: Object 0xffff88041ad14d10 @offset=19728
fp=0xffff88041ad10000
[ 860.203981]
[ 860.203984] Bytes b4 0xffff88041ad14d00: 62 28 00 00 01 00 00 00 5a
5a 5a 5a 5a 5a 5a 5a b(......ZZZZZZZZ
[ 860.204000] Object 0xffff88041ad14d10: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204015] Object 0xffff88041ad14d20: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204030] Object 0xffff88041ad14d30: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204044] Object 0xffff88041ad14d40: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204058] Object 0xffff88041ad14d50: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204073] Object 0xffff88041ad14d60: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204087] Object 0xffff88041ad14d70: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204101] Object 0xffff88041ad14d80: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204116] Object 0xffff88041ad14d90: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204130] Object 0xffff88041ad14da0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204144] Object 0xffff88041ad14db0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204159] Object 0xffff88041ad14dc0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204173] Object 0xffff88041ad14dd0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204187] Object 0xffff88041ad14de0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204202] Object 0xffff88041ad14df0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204216] Object 0xffff88041ad14e00: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204230] Object 0xffff88041ad14e10: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204245] Object 0xffff88041ad14e20: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204259] Object 0xffff88041ad14e30: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204273] Object 0xffff88041ad14e40: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204288] Object 0xffff88041ad14e50: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204302] Object 0xffff88041ad14e60: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204316] Object 0xffff88041ad14e70: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204331] Object 0xffff88041ad14e80: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204345] Object 0xffff88041ad14e90: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204359] Object 0xffff88041ad14ea0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204374] Object 0xffff88041ad14eb0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204388] Object 0xffff88041ad14ec0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204402] Object 0xffff88041ad14ed0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204417] Object 0xffff88041ad14ee0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204431] Object 0xffff88041ad14ef0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204445] Object 0xffff88041ad14f00: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204460] Object 0xffff88041ad14f10: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204474] Object 0xffff88041ad14f20: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204488] Object 0xffff88041ad14f30: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204503] Object 0xffff88041ad14f40: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204517] Object 0xffff88041ad14f50: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204531] Object 0xffff88041ad14f60: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204546] Object 0xffff88041ad14f70: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204560] Object 0xffff88041ad14f80: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204574] Object 0xffff88041ad14f90: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204589] Object 0xffff88041ad14fa0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204603] Object 0xffff88041ad14fb0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204618] Object 0xffff88041ad14fc0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204632] Object 0xffff88041ad14fd0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204646] Object 0xffff88041ad14fe0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204661] Object 0xffff88041ad14ff0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204675] Object 0xffff88041ad15000: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204689] Object 0xffff88041ad15010: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204704] Object 0xffff88041ad15020: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204718] Object 0xffff88041ad15030: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204732] Object 0xffff88041ad15040: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204747] Object 0xffff88041ad15050: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204761] Object 0xffff88041ad15060: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204775] Object 0xffff88041ad15070: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204790] Object 0xffff88041ad15080: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204804] Object 0xffff88041ad15090: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204818] Object 0xffff88041ad150a0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204833] Object 0xffff88041ad150b0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204847] Object 0xffff88041ad150c0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204861] Object 0xffff88041ad150d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204876] Object 0xffff88041ad150e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204890] Object 0xffff88041ad150f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204904] Object 0xffff88041ad15100: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b a5 kkkkkkkkkkkkkkk¥
[ 860.204919] Redzone 0xffff88041ad15110: bb bb bb bb bb bb bb bb
»»»»»»»»
[ 860.204932] Padding 0xffff88041ad15150: 5a 5a 5a 5a 5a 5a 00 00
ZZZZZZ..
[ 860.204949] Pid: 7100, comm: qemu Not tainted 2.6.39 #2
[ 860.204953] Call Trace:
[ 860.204964] [<ffffffff810b7ccd>] ? check_bytes_and_report+0x10d/0x150
[ 860.204973] [<ffffffff810f287c>] ? __blockdev_direct_IO+0x16c/0xa90
[ 860.204979] [<ffffffff810b7db9>] ? check_object+0xa9/0x260
[ 860.204986] [<ffffffff810f287c>] ? __blockdev_direct_IO+0x16c/0xa90
[ 860.204993] [<ffffffff810b86f4>] ? alloc_debug_processing+0x104/0x190
[ 860.204999] [<ffffffff810b9ac2>] ? T.912+0x272/0x2d0
[ 860.205006] [<ffffffff810b9bc5>] ? kmem_cache_alloc+0xa5/0xb0
[ 860.205013] [<ffffffff810f287c>] ? __blockdev_direct_IO+0x16c/0xa90
[ 860.205022] [<ffffffff8108a475>] ? pagevec_lookup+0x15/0x20
[ 860.205029] [<ffffffff8108b5b7>] ?
invalidate_inode_pages2_range+0x87/0x300
[ 860.205039] [<ffffffff81144e41>] ? ext4_direct_IO+0x111/0x1f0
[ 860.205045] [<ffffffff811477a0>] ? _ext4_get_block+0x160/0x160
[ 860.205052] [<ffffffff81144f20>] ? ext4_direct_IO+0x1f0/0x1f0
[ 860.205058] [<ffffffff81082182>] ? generic_file_direct_write+0xd2/0x1b0
[ 860.205064] [<ffffffff8108252b>] ? __generic_file_aio_write+0x2cb/0x430
[ 860.205071] [<ffffffff81082703>] ? generic_file_aio_write+0x73/0xf0
[ 860.205079] [<ffffffff8113f74e>] ? ext4_file_write+0x6e/0x2b0
[ 860.205087] [<ffffffff810c25ef>] ? do_sync_write+0xbf/0x100
[ 860.205095] [<ffffffff810471cc>] ? T.680+0x13c/0x280
[ 860.205103] [<ffffffff810485a8>] ? kill_pid_info+0x38/0x60
[ 860.205110] [<ffffffff81048854>] ? sys_kill+0x94/0x1b0
[ 860.205117] [<ffffffff810c3198>] ? vfs_write+0xc8/0x190
[ 860.205124] [<ffffffff810c32fb>] ? sys_pwrite64+0x9b/0xb0
[ 860.205134] [<ffffffff813d753b>] ? system_call_fastpath+0x16/0x1b
[ 860.205140] FIX kmalloc-1024: Restoring
0xffff88041ad15156-0xffff88041ad15157=0x5a
[ 860.205143]
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 111+ messages in thread
* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
@ 2011-06-01 9:29 ` Brad Campbell
0 siblings, 0 replies; 111+ messages in thread
From: Brad Campbell @ 2011-06-01 9:29 UTC (permalink / raw)
To: Avi Kivity
Cc: Hugh Dickins, Andrea Arcangeli, Borislav Petkov, linux-kernel,
kvm, linux-mm
On 01/06/11 14:56, Avi Kivity wrote:
> On 06/01/2011 09:31 AM, Brad Campbell wrote:
>> On 01/06/11 12:52, Hugh Dickins wrote:
>>
>>>
>>> I guess Brad could try SLUB debugging, boot with slub_debug=P
>>> for poisoning perhaps; though it might upset alignments and
>>> drive the problem underground. Or see if the same happens
>>> with SLAB instead of SLUB.
>>
>> Not much use I'm afraid.
>> This is all I get in the log
>>
>> [ 3161.300073]
>> =============================================================================
>>
>> [ 3161.300147] BUG kmalloc-512: Freechain corrupt
>>
>> The qemu process is then frozen, unkillable but reported in state "R"
>>
>> 13881 ? R 3:27 /usr/bin/qemu -S -M pc-0.13 -enable-kvm -m 1024 -smp
>> 2,sockets=2,cores=1,threads=1 -nam
>>
>> The machine then progressively dies until it's frozen solid with no
>> further error messages.
>>
>> I stupidly forgot to do an alt-sysrq-t prior to doing an alt-sysrq-b,
>> but at least it responded to that.
>>
>> On the bright side I can reproduce it at will.
>
> Please try slub_debug=FZPU; that should point the finger (hopefully at
> somebody else).
>
Well the first attempt locked the machine solid. No network, no console..
I saw
"=========================================================================="
on the console.. nothing after that. Would not respond to sysrq-t or any
other sysrq combination other than -b, which rebooted the box.
No output on netconsole at all, I had to walk to the other building to
look at the monitor and reboot it.
The second attempt jammed netconsole again, but I managed to get this
from an ssh session I already had established. The machine died a slow
and horrible death, but remained interactive enough for me to reboot it with
echo b > /proc/sysrq-trigger
Nothing else worked.
[ 376.269051]
=============================================================================
[ 413.755328]
=============================================================================
[ 413.755337] BUG kmalloc-1024: Object padding overwritten
[ 413.755341]
-----------------------------------------------------------------------------
[ 413.755344]
[ 413.755349] INFO: 0xffff880417179566-0xffff880417179567. First byte
0x0 instead of 0x5a
[ 413.755363] INFO: Allocated in tcp_send_ack+0x26/0x120 age=5320 cpu=5
pid=0
[ 413.755374] INFO: Freed in __kfree_skb+0x11/0x90 age=5320 cpu=5 pid=0
[ 413.755380] INFO: Slab 0xffffea000e50d240 objects=29 used=5
fp=0xffff880417179120 flags=0x80000000000040c1
[ 413.755386] INFO: Object 0xffff880417179120 @offset=4384
fp=0xffff8804171799b0
[ 413.755389]
[ 413.755392] Bytes b4 0xffff880417179110: d9 2b 00 00 01 00 00 00 5a
5a 5a 5a 5a 5a 5a 5a A?+......ZZZZZZZZ
[ 413.755408] Object 0xffff880417179120: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755423] Object 0xffff880417179130: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755438] Object 0xffff880417179140: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755452] Object 0xffff880417179150: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755466] Object 0xffff880417179160: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755480] Object 0xffff880417179170: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755494] Object 0xffff880417179180: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755508] Object 0xffff880417179190: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755522] Object 0xffff8804171791a0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755536] Object 0xffff8804171791b0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755551] Object 0xffff8804171791c0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755565] Object 0xffff8804171791d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755579] Object 0xffff8804171791e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755593] Object 0xffff8804171791f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755607] Object 0xffff880417179200: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755621] Object 0xffff880417179210: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755635] Object 0xffff880417179220: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755650] Object 0xffff880417179230: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755664] Object 0xffff880417179240: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755678] Object 0xffff880417179250: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755692] Object 0xffff880417179260: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755706] Object 0xffff880417179270: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755720] Object 0xffff880417179280: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755734] Object 0xffff880417179290: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755749] Object 0xffff8804171792a0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755763] Object 0xffff8804171792b0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755777] Object 0xffff8804171792c0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755791] Object 0xffff8804171792d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755805] Object 0xffff8804171792e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755819] Object 0xffff8804171792f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755834] Object 0xffff880417179300: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755848] Object 0xffff880417179310: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755862] Object 0xffff880417179320: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755876] Object 0xffff880417179330: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755890] Object 0xffff880417179340: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755904] Object 0xffff880417179350: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755919] Object 0xffff880417179360: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755933] Object 0xffff880417179370: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755947] Object 0xffff880417179380: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755961] Object 0xffff880417179390: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755975] Object 0xffff8804171793a0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.755989] Object 0xffff8804171793b0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.756004] Object 0xffff8804171793c0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.756018] Object 0xffff8804171793d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.756032] Object 0xffff8804171793e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.756046] Object 0xffff8804171793f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.756060] Object 0xffff880417179400: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.756074] Object 0xffff880417179410: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.756089] Object 0xffff880417179420: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.756103] Object 0xffff880417179430: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.756117] Object 0xffff880417179440: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.756131] Object 0xffff880417179450: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.756145] Object 0xffff880417179460: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.756160] Object 0xffff880417179470: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.756174] Object 0xffff880417179480: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.756188] Object 0xffff880417179490: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.756202] Object 0xffff8804171794a0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.756216] Object 0xffff8804171794b0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.756230] Object 0xffff8804171794c0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.756245] Object 0xffff8804171794d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.756259] Object 0xffff8804171794e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.756273] Object 0xffff8804171794f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.756287] Object 0xffff880417179500: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 413.756301] Object 0xffff880417179510: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b a5 kkkkkkkkkkkkkkkAJPY
[ 413.756316] Redzone 0xffff880417179520: bb bb bb bb bb bb bb bb
A>>A>>A>>A>>A>>A>>A>>A>>
[ 413.756329] Padding 0xffff880417179560: 5a 5a 5a 5a 5a 5a 00 00
ZZZZZZ..
[ 413.756345] Pid: 5247, comm: qemu Not tainted 2.6.39 #2
[ 413.756349] Call Trace:
[ 413.756353] <IRQ> [<ffffffff810b7ccd>] ?
check_bytes_and_report+0x10d/0x150
[ 413.756372] [<ffffffff81318f1c>] ? pskb_expand_head+0x15c/0x250
[ 413.756379] [<ffffffff810b7db9>] ? check_object+0xa9/0x260
[ 413.756387] [<ffffffff81318f1c>] ? pskb_expand_head+0x15c/0x250
[ 413.756393] [<ffffffff810b86f4>] ? alloc_debug_processing+0x104/0x190
[ 413.756400] [<ffffffff810b9ac2>] ? T.912+0x272/0x2d0
[ 413.756409] [<ffffffff810ba59d>] ? __kmalloc+0x10d/0x160
[ 413.756416] [<ffffffff81318f1c>] ? pskb_expand_head+0x15c/0x250
[ 413.756424] [<ffffffff813a6c45>] ? nf_bridge_copy_header+0x145/0x160
[ 413.756431] [<ffffffff8139f78d>] ? br_dev_queue_push_xmit+0x6d/0x80
[ 413.756439] [<ffffffff813a55a0>] ? br_nf_post_routing+0x2a0/0x2f0
[ 413.756447] [<ffffffff81346bc4>] ? nf_iterate+0x84/0xb0
[ 413.756453] [<ffffffff8139f720>] ? br_flood_deliver+0x20/0x20
[ 413.756459] [<ffffffff81346c64>] ? nf_hook_slow+0x74/0x120
[ 413.756465] [<ffffffff8139f720>] ? br_flood_deliver+0x20/0x20
[ 413.756472] [<ffffffff8139f7da>] ? br_forward_finish+0x3a/0x60
[ 413.756479] [<ffffffff813a5758>] ? br_nf_forward_finish+0x168/0x170
[ 413.756487] [<ffffffff813a5c90>] ? br_nf_forward_ip+0x360/0x3a0
[ 413.756492] [<ffffffff81346bc4>] ? nf_iterate+0x84/0xb0
[ 413.756498] [<ffffffff8139f7a0>] ? br_dev_queue_push_xmit+0x80/0x80
[ 413.756504] [<ffffffff81346c64>] ? nf_hook_slow+0x74/0x120
[ 413.756510] [<ffffffff8139f7a0>] ? br_dev_queue_push_xmit+0x80/0x80
[ 413.756516] [<ffffffff8139f800>] ? br_forward_finish+0x60/0x60
[ 413.756522] [<ffffffff8139f800>] ? br_forward_finish+0x60/0x60
[ 413.756528] [<ffffffff8139f875>] ? __br_forward+0x75/0xc0
[ 413.756534] [<ffffffff8139f426>] ? deliver_clone+0x36/0x60
[ 413.756540] [<ffffffff8139f69d>] ? br_flood+0xbd/0x100
[ 413.756546] [<ffffffff813a05b0>] ? br_handle_local_finish+0x40/0x40
[ 413.756552] [<ffffffff813a080e>] ? br_handle_frame_finish+0x25e/0x280
[ 413.756560] [<ffffffff813a60f0>] ? br_nf_pre_routing_finish+0x1a0/0x330
[ 413.756568] [<ffffffff813a6958>] ? br_nf_pre_routing+0x6d8/0x800
[ 413.756577] [<ffffffff8102d46a>] ? enqueue_task+0x3a/0x90
[ 413.756582] [<ffffffff81346bc4>] ? nf_iterate+0x84/0xb0
[ 413.756589] [<ffffffff813a05b0>] ? br_handle_local_finish+0x40/0x40
[ 413.756594] [<ffffffff81346c64>] ? nf_hook_slow+0x74/0x120
[ 413.756600] [<ffffffff813a05b0>] ? br_handle_local_finish+0x40/0x40
[ 413.756607] [<ffffffff810339b0>] ? try_to_wake_up+0x2c0/0x2c0
[ 413.756613] [<ffffffff813a09d9>] ? br_handle_frame+0x1a9/0x280
[ 413.756620] [<ffffffff813a0830>] ? br_handle_frame_finish+0x280/0x280
[ 413.756627] [<ffffffff81320ef7>] ? __netif_receive_skb+0x157/0x5c0
[ 413.756634] [<ffffffff81321443>] ? process_backlog+0xe3/0x1d0
[ 413.756641] [<ffffffff81321da5>] ? net_rx_action+0xc5/0x1d0
[ 413.756650] [<ffffffff8103df11>] ? __do_softirq+0x91/0x120
[ 413.756657] [<ffffffff813d838c>] ? call_softirq+0x1c/0x30
[ 413.756660] <EOI> [<ffffffff81003cbd>] ? do_softirq+0x4d/0x80
[ 413.756673] [<ffffffff81321ece>] ? netif_rx_ni+0x1e/0x30
[ 413.756681] [<ffffffff812b3ae2>] ? tun_chr_aio_write+0x332/0x4e0
[ 413.756688] [<ffffffff812b37b0>] ? tun_sendmsg+0x4d0/0x4d0
[ 413.756697] [<ffffffff810c24e9>] ? do_sync_readv_writev+0xa9/0xf0
[ 413.756704] [<ffffffff81063f9c>] ? do_futex+0x13c/0xa70
[ 413.756711] [<ffffffff811d6730>] ? timerqueue_add+0x60/0xb0
[ 413.756719] [<ffffffff81056ab7>] ? __hrtimer_start_range_ns+0x1e7/0x410
[ 413.756726] [<ffffffff810c231b>] ? rw_copy_check_uvector+0x7b/0x140
[ 413.756734] [<ffffffff810c2bcf>] ? do_readv_writev+0xdf/0x210
[ 413.756742] [<ffffffff810c2e7e>] ? sys_writev+0x4e/0xc0
[ 413.756750] [<ffffffff813d753b>] ? system_call_fastpath+0x16/0x1b
[ 413.756756] FIX kmalloc-1024: Restoring
0xffff880417179566-0xffff880417179567=0x5a
[ 413.756760]
[ 556.640033]
=============================================================================
[ 556.640041] BUG kmalloc-512: Object padding overwritten
[ 556.640045]
-----------------------------------------------------------------------------
[ 556.640048]
[ 556.640053] INFO: 0xffff880403bf091e-0xffff880403bf091f. First byte
0x0 instead of 0x5a
[ 556.640069] INFO: Allocated in sock_alloc_send_pskb+0x1d0/0x320
age=21401 cpu=2 pid=5630
[ 556.640080] INFO: Freed in __kfree_skb+0x11/0x90 age=21386 cpu=2 pid=3753
[ 556.640086] INFO: Slab 0xffffea000e0d1c80 objects=28 used=10
fp=0xffff880403bf06d8 flags=0x80000000000040c1
[ 556.640092] INFO: Object 0xffff880403bf06d8 @offset=1752
fp=0xffff880403bf1488
[ 556.640096]
[ 556.640099] Bytes b4 0xffff880403bf06c8: 90 c4 ff ff 00 00 00 00 5a
5a 5a 5a 5a 5a 5a 5a .A?A?A?....ZZZZZZZZ
[ 556.640115] Object 0xffff880403bf06d8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 556.640130] Object 0xffff880403bf06e8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 556.640144] Object 0xffff880403bf06f8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 556.640158] Object 0xffff880403bf0708: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 556.640172] Object 0xffff880403bf0718: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 556.640187] Object 0xffff880403bf0728: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 556.640201] Object 0xffff880403bf0738: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 556.640215] Object 0xffff880403bf0748: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 556.640229] Object 0xffff880403bf0758: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 556.640243] Object 0xffff880403bf0768: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 556.640257] Object 0xffff880403bf0778: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 556.640271] Object 0xffff880403bf0788: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 556.640286] Object 0xffff880403bf0798: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 556.640300] Object 0xffff880403bf07a8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 556.640314] Object 0xffff880403bf07b8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 556.640328] Object 0xffff880403bf07c8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 556.640343] Object 0xffff880403bf07d8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 556.640357] Object 0xffff880403bf07e8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 556.640371] Object 0xffff880403bf07f8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 556.640385] Object 0xffff880403bf0808: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 556.640399] Object 0xffff880403bf0818: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 556.640413] Object 0xffff880403bf0828: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 556.640428] Object 0xffff880403bf0838: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 556.640442] Object 0xffff880403bf0848: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 556.640456] Object 0xffff880403bf0858: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 556.640471] Object 0xffff880403bf0868: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 556.640485] Object 0xffff880403bf0878: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 556.640499] Object 0xffff880403bf0888: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 556.640513] Object 0xffff880403bf0898: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 556.640527] Object 0xffff880403bf08a8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 556.640542] Object 0xffff880403bf08b8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 556.640556] Object 0xffff880403bf08c8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b a5 kkkkkkkkkkkkkkkAJPY
[ 556.640570] Redzone 0xffff880403bf08d8: bb bb bb bb bb bb bb bb
A>>A>>A>>A>>A>>A>>A>>A>>
[ 556.640583] Padding 0xffff880403bf0918: 5a 5a 5a 5a 5a 5a 00 00
ZZZZZZ..
[ 556.640599] Pid: 4809, comm: qemu Not tainted 2.6.39 #2
[ 556.640603] Call Trace:
[ 556.640607] <IRQ> [<ffffffff810b7ccd>] ?
check_bytes_and_report+0x10d/0x150
[ 556.640626] [<ffffffff8131aae7>] ? __netdev_alloc_skb+0x17/0x40
[ 556.640632] [<ffffffff810b7db9>] ? check_object+0xa9/0x260
[ 556.640640] [<ffffffff8131aae7>] ? __netdev_alloc_skb+0x17/0x40
[ 556.640647] [<ffffffff810b86f4>] ? alloc_debug_processing+0x104/0x190
[ 556.640654] [<ffffffff810b9ac2>] ? T.912+0x272/0x2d0
[ 556.640661] [<ffffffff810bb70d>] ? __kmalloc_track_caller+0x10d/0x160
[ 556.640668] [<ffffffff813198c2>] ? __alloc_skb+0x72/0x160
[ 556.640676] [<ffffffff8131aae7>] ? __netdev_alloc_skb+0x17/0x40
[ 556.640684] [<ffffffff813a7b93>] ?
br_ip4_multicast_alloc_query+0x23/0x1c0
[ 556.640690] [<ffffffff813a7e76>] ? br_multicast_send_query+0x76/0x130
[ 556.640698] [<ffffffff8104dba8>] ? wq_worker_waking_up+0x8/0x30
[ 556.640706] [<ffffffff8102ca81>] ? sched_slice+0x51/0x80
[ 556.640711] [<ffffffff813a7fb5>] ?
br_multicast_port_query_expired+0x45/0x70
[ 556.640719] [<ffffffff81043f77>] ? run_timer_softirq+0x137/0x270
[ 556.640725] [<ffffffff81033fe9>] ? scheduler_tick+0x289/0x2d0
[ 556.640731] [<ffffffff813a7f70>] ? br_multicast_query_expired+0x40/0x40
[ 556.640740] [<ffffffff8103df11>] ? __do_softirq+0x91/0x120
[ 556.640747] [<ffffffff813d838c>] ? call_softirq+0x1c/0x30
[ 556.640755] [<ffffffff81003cbd>] ? do_softirq+0x4d/0x80
[ 556.640762] [<ffffffff8103ddbe>] ? irq_exit+0x8e/0xb0
[ 556.640770] [<ffffffff81019d1a>] ? smp_apic_timer_interrupt+0x6a/0xa0
[ 556.640776] [<ffffffff813d7e53>] ? apic_timer_interrupt+0x13/0x20
[ 556.640780] <EOI> [<ffffffff8102965a>] ?
flush_tlb_others_ipi+0x11a/0x130
[ 556.640791] [<ffffffff81029647>] ? flush_tlb_others_ipi+0x107/0x130
[ 556.640799] [<ffffffff810aadab>] ? ptep_clear_flush+0xb/0x10
[ 556.640806] [<ffffffff8109a40b>] ? do_wp_page+0x34b/0x7a0
[ 556.640836] [<ffffffffa00a188a>] ? kvm_read_guest_page+0x5a/0x70 [kvm]
[ 556.640863] [<ffffffffa00b7db4>] ?
paging64_walk_addr_generic+0x264/0x4c0 [kvm]
[ 556.640871] [<ffffffff8109acad>] ? handle_pte_fault+0x44d/0x990
[ 556.640878] [<ffffffff8109b4e8>] ? follow_page+0x268/0x440
[ 556.640886] [<ffffffff8109bd4d>] ? __get_user_pages+0x12d/0x530
[ 556.640892] [<ffffffff81028dbb>] ? gup_pud_range+0x12b/0x1b0
[ 556.640914] [<ffffffffa00a2d2e>] ? get_user_page_nowait+0x2e/0x40 [kvm]
[ 556.640935] [<ffffffffa00a2e7b>] ? hva_to_pfn+0x13b/0x440 [kvm]
[ 556.640956] [<ffffffffa00a31e9>] ? __gfn_to_pfn+0x39/0xf0 [kvm]
[ 556.640979] [<ffffffffa00b825d>] ? try_async_pf+0x4d/0x190 [kvm]
[ 556.641002] [<ffffffffa00b92be>] ? tdp_page_fault+0x10e/0x200 [kvm]
[ 556.641026] [<ffffffffa00b968c>] ? kvm_mmu_page_fault+0x1c/0x80 [kvm]
[ 556.641050] [<ffffffffa00b0a7a>] ?
kvm_arch_vcpu_ioctl_run+0x3fa/0xcf0 [kvm]
[ 556.641058] [<ffffffff8106239f>] ? futex_wake+0x10f/0x120
[ 556.641065] [<ffffffff81063f7b>] ? do_futex+0x11b/0xa70
[ 556.641086] [<ffffffffa00a517f>] ? kvm_vcpu_ioctl+0x4df/0x5e0 [kvm]
[ 556.641097] [<ffffffff81046671>] ? __dequeue_signal+0xe1/0x170
[ 556.641104] [<ffffffff8104821c>] ? do_send_sig_info+0x6c/0x90
[ 556.641112] [<ffffffff81046aac>] ? dequeue_signal+0x3c/0x170
[ 556.641119] [<ffffffff8104551f>] ? copy_siginfo_to_user+0xef/0x1d0
[ 556.641125] [<ffffffff810d26db>] ? do_vfs_ioctl+0x9b/0x4f0
[ 556.641132] [<ffffffff8106494a>] ? sys_futex+0x7a/0x180
[ 556.641137] [<ffffffff810d2b79>] ? sys_ioctl+0x49/0x80
[ 556.641145] [<ffffffff813d753b>] ? system_call_fastpath+0x16/0x1b
[ 556.641152] FIX kmalloc-512: Restoring
0xffff880403bf091e-0xffff880403bf091f=0x5a
[ 556.641155]
[ 602.610062] INFO: task ksmd:552 blocked for more than 120 seconds.
[ 602.610068] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[ 602.610073] ksmd D 0000000000000000 0 552 2
0x00000000
[ 602.610083] ffff88041db68ca0 0000000000000046 000212d01fc11d00
ffff880400000000
[ 602.610090] ffffffff81593020 ffff88041c1a9fd8 0000000000004000
ffff88041c1a8010
[ 602.610097] ffff88041c1a9fd8 ffff88041db68ca0 0000000000000000
0000000200000002
[ 602.610104] Call Trace:
[ 602.610121] [<ffffffff813d5595>] ? schedule_timeout+0x1c5/0x230
[ 602.610132] [<ffffffff8102f69c>] ? enqueue_task_fair+0x14c/0x190
[ 602.610141] [<ffffffff8102db27>] ? task_rq_lock+0x47/0x90
[ 602.610148] [<ffffffff813d4562>] ? wait_for_common+0xd2/0x180
[ 602.610154] [<ffffffff810339b0>] ? try_to_wake_up+0x2c0/0x2c0
[ 602.610162] [<ffffffff8104cb94>] ? flush_work+0x24/0x30
[ 602.610167] [<ffffffff8104be10>] ? do_work_for_cpu+0x20/0x20
[ 602.610174] [<ffffffff8104e0ab>] ? schedule_on_each_cpu+0xab/0xe0
[ 602.610181] [<ffffffff810b5c05>] ? ksm_scan_thread+0x7f5/0xc20
[ 602.610189] [<ffffffff81052a20>] ? wake_up_bit+0x40/0x40
[ 602.610194] [<ffffffff810b5410>] ?
try_to_merge_with_ksm_page+0x570/0x570
[ 602.610200] [<ffffffff810b5410>] ?
try_to_merge_with_ksm_page+0x570/0x570
[ 602.610207] [<ffffffff810525b6>] ? kthread+0x96/0xa0
[ 602.610214] [<ffffffff813d8294>] ? kernel_thread_helper+0x4/0x10
[ 602.610221] [<ffffffff81052520>] ? kthread_worker_fn+0x120/0x120
[ 602.610227] [<ffffffff813d8290>] ? gs_change+0xb/0xb
[ 602.610233] INFO: task fsnotify_mark:662 blocked for more than 120
seconds.
[ 602.610237] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[ 602.610241] fsnotify_mark D 0000000000000000 0 662 2
0x00000000
[ 602.610248] ffff88041c31d860 0000000000000046 0000000000000000
0000000000000000
[ 602.610255] ffffffff81593020 ffff88041c365fd8 0000000000004000
ffff88041c364010
[ 602.610262] ffff88041c365fd8 ffff88041c31d860 0000000000000000
0000000000000000
[ 602.610268] Call Trace:
[ 602.610275] [<ffffffff81035521>] ? load_balance+0x91/0x5e0
[ 602.610282] [<ffffffff813d5595>] ? schedule_timeout+0x1c5/0x230
[ 602.610289] [<ffffffff8102f323>] ? pick_next_task_fair+0x103/0x190
[ 602.610295] [<ffffffff813d4a6d>] ? schedule+0x28d/0x910
[ 602.610302] [<ffffffff813d4562>] ? wait_for_common+0xd2/0x180
[ 602.610307] [<ffffffff810339b0>] ? try_to_wake_up+0x2c0/0x2c0
[ 602.610314] [<ffffffff81074420>] ? synchronize_rcu_bh+0x50/0x50
[ 602.610320] [<ffffffff8107446a>] ? synchronize_sched+0x4a/0x50
[ 602.610326] [<ffffffff8104f940>] ? find_ge_pid+0x40/0x40
[ 602.610333] [<ffffffff8105742b>] ? __synchronize_srcu+0x5b/0xc0
[ 602.610342] [<ffffffff810f5f63>] ? fsnotify_mark_destroy+0x83/0x150
[ 602.610348] [<ffffffff81052a20>] ? wake_up_bit+0x40/0x40
[ 602.610356] [<ffffffff810f5ee0>] ?
fsnotify_set_mark_ignored_mask_locked+0x20/0x20
[ 602.610364] [<ffffffff810f5ee0>] ?
fsnotify_set_mark_ignored_mask_locked+0x20/0x20
[ 602.610371] [<ffffffff810525b6>] ? kthread+0x96/0xa0
[ 602.610377] [<ffffffff813d8294>] ? kernel_thread_helper+0x4/0x10
[ 602.610385] [<ffffffff81052520>] ? kthread_worker_fn+0x120/0x120
[ 602.610390] [<ffffffff813d8290>] ? gs_change+0xb/0xb
[ 644.689408]
=============================================================================
[ 644.689416] BUG kmalloc-512: Object padding overwritten
[ 644.689420]
-----------------------------------------------------------------------------
[ 644.689423]
[ 644.689428] INFO: 0xffff88041411f476-0xffff88041411f477. First byte
0x0 instead of 0x5a
[ 644.689444] INFO: Allocated in load_elf_binary+0xa6c/0x1c00 age=18648
cpu=3 pid=5915
[ 644.689454] INFO: Freed in load_elf_binary+0xab8/0x1c00 age=18648
cpu=3 pid=5915
[ 644.689461] INFO: Slab 0xffffea000e463e20 objects=28 used=10
fp=0xffff88041411f230 flags=0x80000000000040c1
[ 644.689467] INFO: Object 0xffff88041411f230 @offset=12848
fp=0xffff88041411eda0
[ 644.689470]
[ 644.689474] Bytes b4 0xffff88041411f220: 22 6f 00 00 01 00 00 00 5a
5a 5a 5a 5a 5a 5a 5a "o......ZZZZZZZZ
[ 644.689490] Object 0xffff88041411f230: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 644.689505] Object 0xffff88041411f240: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 644.689519] Object 0xffff88041411f250: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 644.689533] Object 0xffff88041411f260: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 644.689547] Object 0xffff88041411f270: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 644.689562] Object 0xffff88041411f280: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 644.689576] Object 0xffff88041411f290: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 644.689590] Object 0xffff88041411f2a0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 644.689604] Object 0xffff88041411f2b0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 644.689618] Object 0xffff88041411f2c0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 644.689632] Object 0xffff88041411f2d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 644.689647] Object 0xffff88041411f2e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 644.689661] Object 0xffff88041411f2f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 644.689675] Object 0xffff88041411f300: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 644.689689] Object 0xffff88041411f310: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 644.689703] Object 0xffff88041411f320: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 644.689717] Object 0xffff88041411f330: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 644.689732] Object 0xffff88041411f340: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 644.689746] Object 0xffff88041411f350: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 644.689760] Object 0xffff88041411f360: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 644.689774] Object 0xffff88041411f370: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 644.689788] Object 0xffff88041411f380: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 644.689802] Object 0xffff88041411f390: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 644.689817] Object 0xffff88041411f3a0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 644.689831] Object 0xffff88041411f3b0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 644.689845] Object 0xffff88041411f3c0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 644.689859] Object 0xffff88041411f3d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 644.689873] Object 0xffff88041411f3e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 644.689888] Object 0xffff88041411f3f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 644.689902] Object 0xffff88041411f400: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 644.689916] Object 0xffff88041411f410: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 644.689930] Object 0xffff88041411f420: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b a5 kkkkkkkkkkkkkkkAJPY
[ 644.689945] Redzone 0xffff88041411f430: bb bb bb bb bb bb bb bb
A>>A>>A>>A>>A>>A>>A>>A>>
[ 644.689957] Padding 0xffff88041411f470: 5a 5a 5a 5a 5a 5a 00 00
ZZZZZZ..
[ 644.689974] Pid: 6597, comm: awk Not tainted 2.6.39 #2
[ 644.689978] Call Trace:
[ 644.689989] [<ffffffff810b7ccd>] ? check_bytes_and_report+0x10d/0x150
[ 644.689998] [<ffffffff8110773c>] ? load_elf_binary+0xa6c/0x1c00
[ 644.690005] [<ffffffff810b7db9>] ? check_object+0xa9/0x260
[ 644.690012] [<ffffffff8110773c>] ? load_elf_binary+0xa6c/0x1c00
[ 644.690019] [<ffffffff810b86f4>] ? alloc_debug_processing+0x104/0x190
[ 644.690026] [<ffffffff810b9ac2>] ? T.912+0x272/0x2d0
[ 644.690032] [<ffffffff810ba59d>] ? __kmalloc+0x10d/0x160
[ 644.690039] [<ffffffff8110773c>] ? load_elf_binary+0xa6c/0x1c00
[ 644.690047] [<ffffffff8109bd9d>] ? __get_user_pages+0x17d/0x530
[ 644.690055] [<ffffffff810c9556>] ? get_arg_page+0x56/0x100
[ 644.690062] [<ffffffff810c8070>] ? search_binary_handler+0x90/0x240
[ 644.690069] [<ffffffff810c9e9f>] ? do_execve+0x22f/0x2f0
[ 644.690076] [<ffffffff810094a6>] ? sys_execve+0x36/0x60
[ 644.690085] [<ffffffff813d78dc>] ? stub_execve+0x6c/0xc0
[ 644.690092] FIX kmalloc-512: Restoring
0xffff88041411f476-0xffff88041411f477=0x5a
[ 644.690095]
[ 704.979481]
=============================================================================
[ 704.979488] BUG kmalloc-512: Object padding overwritten
[ 704.979492]
-----------------------------------------------------------------------------
[ 704.979496]
[ 704.979501] INFO: 0xffff8804168c1fee-0xffff8804168c1fef. First byte
0x0 instead of 0x5a
[ 704.979517] INFO: Allocated in load_elf_binary+0xa6c/0x1c00 age=24184
cpu=0 pid=5999
[ 704.979527] INFO: Freed in load_elf_binary+0xab8/0x1c00 age=24184
cpu=0 pid=5999
[ 704.979534] INFO: Slab 0xffffea000e4eea00 objects=28 used=10
fp=0xffff8804168c1da8 flags=0x80000000000040c1
[ 704.979540] INFO: Object 0xffff8804168c1da8 @offset=7592
fp=0xffff8804168c2910
[ 704.979543]
[ 704.979547] Bytes b4 0xffff8804168c1d98: 93 98 ff ff 00 00 00 00 5a
5a 5a 5a 5a 5a 5a 5a ..A?A?....ZZZZZZZZ
[ 704.979563] Object 0xffff8804168c1da8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 704.979578] Object 0xffff8804168c1db8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 704.979592] Object 0xffff8804168c1dc8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 704.979606] Object 0xffff8804168c1dd8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 704.979620] Object 0xffff8804168c1de8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 704.979634] Object 0xffff8804168c1df8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 704.979648] Object 0xffff8804168c1e08: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 704.979662] Object 0xffff8804168c1e18: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 704.979676] Object 0xffff8804168c1e28: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 704.979690] Object 0xffff8804168c1e38: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 704.979704] Object 0xffff8804168c1e48: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 704.979719] Object 0xffff8804168c1e58: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 704.979733] Object 0xffff8804168c1e68: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 704.979747] Object 0xffff8804168c1e78: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 704.979761] Object 0xffff8804168c1e88: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 704.979775] Object 0xffff8804168c1e98: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 704.979789] Object 0xffff8804168c1ea8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 704.979803] Object 0xffff8804168c1eb8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 704.979817] Object 0xffff8804168c1ec8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 704.979831] Object 0xffff8804168c1ed8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 704.979845] Object 0xffff8804168c1ee8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 704.979859] Object 0xffff8804168c1ef8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 704.979873] Object 0xffff8804168c1f08: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 704.979888] Object 0xffff8804168c1f18: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 704.979902] Object 0xffff8804168c1f28: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 704.979916] Object 0xffff8804168c1f38: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 704.979930] Object 0xffff8804168c1f48: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 704.979944] Object 0xffff8804168c1f58: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 704.979958] Object 0xffff8804168c1f68: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 704.979972] Object 0xffff8804168c1f78: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 704.979986] Object 0xffff8804168c1f88: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 704.980000] Object 0xffff8804168c1f98: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b a5 kkkkkkkkkkkkkkkAJPY
[ 704.980015] Redzone 0xffff8804168c1fa8: bb bb bb bb bb bb bb bb
A>>A>>A>>A>>A>>A>>A>>A>>
[ 704.980028] Padding 0xffff8804168c1fe8: 5a 5a 5a 5a 5a 5a 00 00
ZZZZZZ..
[ 704.980044] Pid: 6812, comm: get-rrdtool-dat Not tainted 2.6.39 #2
[ 704.980048] Call Trace:
[ 704.980060] [<ffffffff810b7ccd>] ? check_bytes_and_report+0x10d/0x150
[ 704.980069] [<ffffffff81106e49>] ? load_elf_binary+0x179/0x1c00
[ 704.980075] [<ffffffff810b7db9>] ? check_object+0xa9/0x260
[ 704.980083] [<ffffffff81106e49>] ? load_elf_binary+0x179/0x1c00
[ 704.980090] [<ffffffff810b86f4>] ? alloc_debug_processing+0x104/0x190
[ 704.980096] [<ffffffff810b9ac2>] ? T.912+0x272/0x2d0
[ 704.980102] [<ffffffff810ba59d>] ? __kmalloc+0x10d/0x160
[ 704.980109] [<ffffffff81106e49>] ? load_elf_binary+0x179/0x1c00
[ 704.980117] [<ffffffff8109b782>] ? __pte_alloc+0x42/0x130
[ 704.980123] [<ffffffff8109b5a5>] ? follow_page+0x325/0x440
[ 704.980130] [<ffffffff8109b5a5>] ? follow_page+0x325/0x440
[ 704.980137] [<ffffffff8109bd9d>] ? __get_user_pages+0x17d/0x530
[ 704.980145] [<ffffffff810c9556>] ? get_arg_page+0x56/0x100
[ 704.980152] [<ffffffff810c8070>] ? search_binary_handler+0x90/0x240
[ 704.980158] [<ffffffff810c9e9f>] ? do_execve+0x22f/0x2f0
[ 704.980166] [<ffffffff810094a6>] ? sys_execve+0x36/0x60
[ 704.980175] [<ffffffff813d78dc>] ? stub_execve+0x6c/0xc0
[ 704.980182] FIX kmalloc-512: Restoring
0xffff8804168c1fee-0xffff8804168c1fef=0x5a
[ 704.980185]
[ 722.610058] INFO: task ksmd:552 blocked for more than 120 seconds.
[ 722.610063] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[ 722.610069] ksmd D 0000000000000000 0 552 2
0x00000000
[ 722.610078] ffff88041db68ca0 0000000000000046 000212d01fc11d00
ffff880400000000
[ 722.610085] ffffffff81593020 ffff88041c1a9fd8 0000000000004000
ffff88041c1a8010
[ 722.610092] ffff88041c1a9fd8 ffff88041db68ca0 0000000000000000
0000000200000002
[ 722.610099] Call Trace:
[ 722.610115] [<ffffffff813d5595>] ? schedule_timeout+0x1c5/0x230
[ 722.610127] [<ffffffff8102f69c>] ? enqueue_task_fair+0x14c/0x190
[ 722.610135] [<ffffffff8102db27>] ? task_rq_lock+0x47/0x90
[ 722.610142] [<ffffffff813d4562>] ? wait_for_common+0xd2/0x180
[ 722.610148] [<ffffffff810339b0>] ? try_to_wake_up+0x2c0/0x2c0
[ 722.610156] [<ffffffff8104cb94>] ? flush_work+0x24/0x30
[ 722.610162] [<ffffffff8104be10>] ? do_work_for_cpu+0x20/0x20
[ 722.610168] [<ffffffff8104e0ab>] ? schedule_on_each_cpu+0xab/0xe0
[ 722.610175] [<ffffffff810b5c05>] ? ksm_scan_thread+0x7f5/0xc20
[ 722.610182] [<ffffffff81052a20>] ? wake_up_bit+0x40/0x40
[ 722.610188] [<ffffffff810b5410>] ?
try_to_merge_with_ksm_page+0x570/0x570
[ 722.610194] [<ffffffff810b5410>] ?
try_to_merge_with_ksm_page+0x570/0x570
[ 722.610201] [<ffffffff810525b6>] ? kthread+0x96/0xa0
[ 722.610208] [<ffffffff813d8294>] ? kernel_thread_helper+0x4/0x10
[ 722.610215] [<ffffffff81052520>] ? kthread_worker_fn+0x120/0x120
[ 722.610221] [<ffffffff813d8290>] ? gs_change+0xb/0xb
[ 722.610226] INFO: task fsnotify_mark:662 blocked for more than 120
seconds.
[ 722.610230] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[ 722.610235] fsnotify_mark D 0000000000000000 0 662 2
0x00000000
[ 722.610241] ffff88041c31d860 0000000000000046 0000000000000000
0000000000000000
[ 722.610248] ffffffff81593020 ffff88041c365fd8 0000000000004000
ffff88041c364010
[ 722.610255] ffff88041c365fd8 ffff88041c31d860 0000000000000000
0000000000000000
[ 722.610261] Call Trace:
[ 722.610268] [<ffffffff81035521>] ? load_balance+0x91/0x5e0
[ 722.610275] [<ffffffff813d5595>] ? schedule_timeout+0x1c5/0x230
[ 722.610282] [<ffffffff8102f323>] ? pick_next_task_fair+0x103/0x190
[ 722.610289] [<ffffffff813d4a6d>] ? schedule+0x28d/0x910
[ 722.610295] [<ffffffff813d4562>] ? wait_for_common+0xd2/0x180
[ 722.610301] [<ffffffff810339b0>] ? try_to_wake_up+0x2c0/0x2c0
[ 722.610308] [<ffffffff81074420>] ? synchronize_rcu_bh+0x50/0x50
[ 722.610313] [<ffffffff8107446a>] ? synchronize_sched+0x4a/0x50
[ 722.610319] [<ffffffff8104f940>] ? find_ge_pid+0x40/0x40
[ 722.610326] [<ffffffff8105742b>] ? __synchronize_srcu+0x5b/0xc0
[ 722.610335] [<ffffffff810f5f63>] ? fsnotify_mark_destroy+0x83/0x150
[ 722.610342] [<ffffffff81052a20>] ? wake_up_bit+0x40/0x40
[ 722.610350] [<ffffffff810f5ee0>] ?
fsnotify_set_mark_ignored_mask_locked+0x20/0x20
[ 722.610358] [<ffffffff810f5ee0>] ?
fsnotify_set_mark_ignored_mask_locked+0x20/0x20
[ 722.610365] [<ffffffff810525b6>] ? kthread+0x96/0xa0
[ 722.610371] [<ffffffff813d8294>] ? kernel_thread_helper+0x4/0x10
[ 722.610378] [<ffffffff81052520>] ? kthread_worker_fn+0x120/0x120
[ 722.610384] [<ffffffff813d8290>] ? gs_change+0xb/0xb
[ 722.610392] INFO: task jbd2/md0-8:2528 blocked for more than 120 seconds.
[ 722.610396] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[ 722.610400] jbd2/md0-8 D 0000000000000005 0 2528 2
0x00000000
[ 722.610407] ffff88041d91c570 0000000000000046 ffff88041baeea28
ffffea0000000000
[ 722.610413] ffff88041d91a5e0 ffff880419f9bfd8 0000000000004000
ffff880419f9a010
[ 722.610420] ffff880419f9bfd8 ffff88041d91c570 ffff88041baeea28
ffffffff810b8495
[ 722.610426] Call Trace:
[ 722.610432] [<ffffffff810b8495>] ? init_object+0x85/0xa0
[ 722.610438] [<ffffffff810b8916>] ? free_debug_processing+0x196/0x250
[ 722.610447] [<ffffffff8105adae>] ? ktime_get_ts+0x6e/0xf0
[ 722.610455] [<ffffffff810810f0>] ? __lock_page+0x70/0x70
[ 722.610461] [<ffffffff813d5174>] ? io_schedule+0x84/0xd0
[ 722.610469] [<ffffffff811d4403>] ?
radix_tree_gang_lookup_tag_slot+0x93/0xf0
[ 722.610476] [<ffffffff810810f9>] ? sleep_on_page+0x9/0x10
[ 722.610482] [<ffffffff813d57df>] ? __wait_on_bit+0x4f/0x80
[ 722.610489] [<ffffffff810812eb>] ? wait_on_page_bit+0x6b/0x80
[ 722.610496] [<ffffffff81052a50>] ? autoremove_wake_function+0x30/0x30
[ 722.610504] [<ffffffff8108a458>] ? pagevec_lookup_tag+0x18/0x20
[ 722.610509] [<ffffffff81081f2a>] ? filemap_fdatawait_range+0xfa/0x180
[ 722.610518] [<ffffffff811be09f>] ? submit_bio+0x6f/0xf0
[ 722.610526] [<ffffffff81176276>] ?
jbd2_journal_commit_transaction+0x796/0x1270
[ 722.610536] [<ffffffff81179ed1>] ? kjournald2+0xb1/0x1e0
[ 722.610542] [<ffffffff81052a20>] ? wake_up_bit+0x40/0x40
[ 722.610549] [<ffffffff81179e20>] ? commit_timeout+0x10/0x10
[ 722.610556] [<ffffffff81179e20>] ? commit_timeout+0x10/0x10
[ 722.610563] [<ffffffff810525b6>] ? kthread+0x96/0xa0
[ 722.610569] [<ffffffff813d8294>] ? kernel_thread_helper+0x4/0x10
[ 722.610576] [<ffffffff81052520>] ? kthread_worker_fn+0x120/0x120
[ 722.610582] [<ffffffff813d8290>] ? gs_change+0xb/0xb
[ 722.610591] INFO: task nfsd:4326 blocked for more than 120 seconds.
[ 722.610594] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[ 722.610598] nfsd D 0000000000000000 0 4326 2
0x00000000
[ 722.610605] ffff88041b604bc0 0000000000000046 0000000000000016
ffffffff00000000
[ 722.610611] ffff88041d8fa5e0 ffff88041c527fd8 0000000000004000
ffff88041c526010
[ 722.610617] ffff88041c527fd8 ffff88041b604bc0 000000000000009c
ffff880000000000
[ 722.610623] Call Trace:
[ 722.610630] [<ffffffff8101e845>] ? amd_flush_garts+0x105/0x140
[ 722.610637] [<ffffffff8101fa80>] ? gart_map_sg+0x480/0x480
[ 722.610643] [<ffffffff8101f5d3>] ? flush_gart+0x23/0x50
[ 722.610650] [<ffffffff81080f98>] ? find_get_page+0x18/0x90
[ 722.610657] [<ffffffff81174b95>] ? do_get_write_access+0x265/0x4a0
[ 722.610665] [<ffffffff81052a50>] ? autoremove_wake_function+0x30/0x30
[ 722.610672] [<ffffffff81174ef9>] ?
jbd2_journal_get_write_access+0x29/0x50
[ 722.610680] [<ffffffff8115f122>] ?
__ext4_journal_get_write_access+0x32/0x80
[ 722.610689] [<ffffffff81143908>] ? ext4_reserve_inode_write+0x78/0xa0
[ 722.610696] [<ffffffff81143970>] ? ext4_mark_inode_dirty+0x40/0x1e0
[ 722.610703] [<ffffffff81156c0b>] ? ext4_journal_start_sb+0x6b/0x160
[ 722.610711] [<ffffffff81322645>] ? dev_hard_start_xmit+0x305/0x5f0
[ 722.610719] [<ffffffff81352fd0>] ? ip_finish_output2+0x290/0x290
[ 722.610726] [<ffffffff81143c65>] ? ext4_dirty_inode+0x35/0x70
[ 722.610733] [<ffffffff810e4a08>] ? __mark_inode_dirty+0x38/0x210
[ 722.610741] [<ffffffff810d9317>] ? file_update_time+0xf7/0x180
[ 722.610747] [<ffffffff81082458>] ? __generic_file_aio_write+0x1f8/0x430
[ 722.610755] [<ffffffff81373ef9>] ? udp_sendmsg+0x3c9/0x7e0
[ 722.610763] [<ffffffff81314260>] ? sock_alloc_send_pskb+0x1d0/0x320
[ 722.610769] [<ffffffff81082703>] ? generic_file_aio_write+0x73/0xf0
[ 722.610776] [<ffffffff8113f74e>] ? ext4_file_write+0x6e/0x2b0
[ 722.610783] [<ffffffff810da8ac>] ? iget_locked+0x4c/0x140
[ 722.610789] [<ffffffff8119a590>] ? fh_compose+0x4c0/0x4c0
[ 722.610795] [<ffffffff8113f6e0>] ? ext4_llseek+0x110/0x110
[ 722.610803] [<ffffffff810c24e9>] ? do_sync_readv_writev+0xa9/0xf0
[ 722.610811] [<ffffffff810c231b>] ? rw_copy_check_uvector+0x7b/0x140
[ 722.610819] [<ffffffff810c2bcf>] ? do_readv_writev+0xdf/0x210
[ 722.610825] [<ffffffff810b8735>] ? alloc_debug_processing+0x145/0x190
[ 722.610831] [<ffffffff810b9bc5>] ? kmem_cache_alloc+0xa5/0xb0
[ 722.610838] [<ffffffff8113f463>] ? ext4_file_open+0x63/0x180
[ 722.610844] [<ffffffff8119c21d>] ? nfsd_vfs_write+0xed/0x3a0
[ 722.610851] [<ffffffff810c1127>] ? __dentry_open+0x1f7/0x2b0
[ 722.610857] [<ffffffff8119c892>] ? nfsd_open+0xf2/0x1b0
[ 722.610862] [<ffffffff8119cd34>] ? nfsd_write+0xf4/0x110
[ 722.610868] [<ffffffff81199930>] ? nfsd_proc_write+0xb0/0x120
[ 722.610876] [<ffffffff811971c5>] ? nfsd_dispatch+0xf5/0x230
[ 722.610882] [<ffffffff813b456f>] ? svc_process+0x4af/0x820
[ 722.610887] [<ffffffff810339b0>] ? try_to_wake_up+0x2c0/0x2c0
[ 722.610894] [<ffffffff811977a0>] ? nfsd_svc+0x1b0/0x1b0
[ 722.610901] [<ffffffff8119784d>] ? nfsd+0xad/0x150
[ 722.610907] [<ffffffff810525b6>] ? kthread+0x96/0xa0
[ 722.610913] [<ffffffff813d8294>] ? kernel_thread_helper+0x4/0x10
[ 722.610921] [<ffffffff81052520>] ? kthread_worker_fn+0x120/0x120
[ 722.610926] [<ffffffff813d8290>] ? gs_change+0xb/0xb
[ 765.241265]
=============================================================================
[ 765.241267] BUG kmalloc-512: Object padding overwritten
[ 765.241268]
-----------------------------------------------------------------------------
[ 765.241269]
[ 765.241271] INFO: 0xffff8804168c1486-0xffff8804168c1487. First byte
0x0 instead of 0x5a
[ 765.241277] INFO: Allocated in bio_kmalloc+0x2d/0x70 age=73010 cpu=5
pid=2495
[ 765.241281] INFO: Freed in r10buf_pool_free+0x71/0x90 age=72999 cpu=5
pid=1728
[ 765.241283] INFO: Slab 0xffffea000e4eea00 objects=28 used=13
fp=0xffff8804168c1240 flags=0x80000000000040c1
[ 765.241284] INFO: Object 0xffff8804168c1240 @offset=4672
fp=0xffff8804168c0ff8
[ 765.241285]
[ 765.241286] Bytes b4 0xffff8804168c1230: 95 98 ff ff 00 00 00 00 5a
5a 5a 5a 5a 5a 5a 5a ..A?A?....ZZZZZZZZ
[ 765.241291] Object 0xffff8804168c1240: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 765.241294] Object 0xffff8804168c1250: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 765.241298] Object 0xffff8804168c1260: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 765.241301] Object 0xffff8804168c1270: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 765.241305] Object 0xffff8804168c1280: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 765.241308] Object 0xffff8804168c1290: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 765.241312] Object 0xffff8804168c12a0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 765.241315] Object 0xffff8804168c12b0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 765.241319] Object 0xffff8804168c12c0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 765.241322] Object 0xffff8804168c12d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 765.241326] Object 0xffff8804168c12e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 765.241329] Object 0xffff8804168c12f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 765.241332] Object 0xffff8804168c1300: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 765.241336] Object 0xffff8804168c1310: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 765.241339] Object 0xffff8804168c1320: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 765.241343] Object 0xffff8804168c1330: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 765.241346] Object 0xffff8804168c1340: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 765.241350] Object 0xffff8804168c1350: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 765.241353] Object 0xffff8804168c1360: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 765.241357] Object 0xffff8804168c1370: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 765.241360] Object 0xffff8804168c1380: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 765.241364] Object 0xffff8804168c1390: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 765.241367] Object 0xffff8804168c13a0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 765.241371] Object 0xffff8804168c13b0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 765.241374] Object 0xffff8804168c13c0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 765.241378] Object 0xffff8804168c13d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 765.241381] Object 0xffff8804168c13e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 765.241385] Object 0xffff8804168c13f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 765.241388] Object 0xffff8804168c1400: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 765.241392] Object 0xffff8804168c1410: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 765.241395] Object 0xffff8804168c1420: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 765.241399] Object 0xffff8804168c1430: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b a5 kkkkkkkkkkkkkkkAJPY
[ 765.241402] Redzone 0xffff8804168c1440: bb bb bb bb bb bb bb bb
A>>A>>A>>A>>A>>A>>A>>A>>
[ 765.241406] Padding 0xffff8804168c1480: 5a 5a 5a 5a 5a 5a 00 00
ZZZZZZ..
[ 765.241410] Pid: 7147, comm: apcaccess Not tainted 2.6.39 #2
[ 765.241411] Call Trace:
[ 765.241416] [<ffffffff810b7ccd>] ? check_bytes_and_report+0x10d/0x150
[ 765.241419] [<ffffffff8110773c>] ? load_elf_binary+0xa6c/0x1c00
[ 765.241421] [<ffffffff810b7db9>] ? check_object+0xa9/0x260
[ 765.241423] [<ffffffff8110773c>] ? load_elf_binary+0xa6c/0x1c00
[ 765.241424] [<ffffffff810b86f4>] ? alloc_debug_processing+0x104/0x190
[ 765.241426] [<ffffffff810b9ac2>] ? T.912+0x272/0x2d0
[ 765.241428] [<ffffffff810ba59d>] ? __kmalloc+0x10d/0x160
[ 765.241429] [<ffffffff8110773c>] ? load_elf_binary+0xa6c/0x1c00
[ 765.241432] [<ffffffff8109bd9d>] ? __get_user_pages+0x17d/0x530
[ 765.241434] [<ffffffff810c9556>] ? get_arg_page+0x56/0x100
[ 765.241437] [<ffffffff811da4ad>] ? strnlen_user+0x2d/0x80
[ 765.241439] [<ffffffff810c8070>] ? search_binary_handler+0x90/0x240
[ 765.241440] [<ffffffff810c9e9f>] ? do_execve+0x22f/0x2f0
[ 765.241443] [<ffffffff810094a6>] ? sys_execve+0x36/0x60
[ 765.241446] [<ffffffff813d78dc>] ? stub_execve+0x6c/0xc0
[ 765.241448] FIX kmalloc-512: Restoring
0xffff8804168c1486-0xffff8804168c1487=0x5a
[ 765.241449]
[ 789.536436]
=============================================================================
[ 789.536444] BUG kmalloc-2048: Object padding overwritten
[ 789.536448]
-----------------------------------------------------------------------------
[ 789.536451]
[ 789.536456] INFO: 0xffff8802c8b1a11e-0xffff8802c8b1a11f. First byte
0x0 instead of 0x5a
[ 789.536471] INFO: Allocated in sk_stream_alloc_skb+0x3a/0x110 age=63
cpu=3 pid=5169
[ 789.536482] INFO: Freed in __kfree_skb+0x11/0x90 age=63 cpu=5 pid=0
[ 789.536488] INFO: Slab 0xffffea0009be6d40 objects=15 used=4
fp=0xffff8802c8b198d8 flags=0x80000000000040c1
[ 789.536494] INFO: Object 0xffff8802c8b198d8 @offset=6360
fp=0xffff8802c8b1a968
[ 789.536497]
[ 789.536501] Bytes b4 0xffff8802c8b198c8: cc be 00 00 01 00 00 00 5a
5a 5a 5a 5a 5a 5a 5a A?A,......ZZZZZZZZ
[ 789.536517] Object 0xffff8802c8b198d8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.536532] Object 0xffff8802c8b198e8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.536546] Object 0xffff8802c8b198f8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.536560] Object 0xffff8802c8b19908: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.536575] Object 0xffff8802c8b19918: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.536589] Object 0xffff8802c8b19928: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.536603] Object 0xffff8802c8b19938: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.536617] Object 0xffff8802c8b19948: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.536631] Object 0xffff8802c8b19958: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.536645] Object 0xffff8802c8b19968: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.536660] Object 0xffff8802c8b19978: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.536674] Object 0xffff8802c8b19988: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.536688] Object 0xffff8802c8b19998: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.536702] Object 0xffff8802c8b199a8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.536716] Object 0xffff8802c8b199b8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.536730] Object 0xffff8802c8b199c8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.536745] Object 0xffff8802c8b199d8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.536759] Object 0xffff8802c8b199e8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.536773] Object 0xffff8802c8b199f8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.536787] Object 0xffff8802c8b19a08: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.536801] Object 0xffff8802c8b19a18: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.536816] Object 0xffff8802c8b19a28: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.536830] Object 0xffff8802c8b19a38: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.536844] Object 0xffff8802c8b19a48: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.536858] Object 0xffff8802c8b19a58: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.536872] Object 0xffff8802c8b19a68: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.536887] Object 0xffff8802c8b19a78: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.536901] Object 0xffff8802c8b19a88: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.536915] Object 0xffff8802c8b19a98: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.536929] Object 0xffff8802c8b19aa8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.536943] Object 0xffff8802c8b19ab8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.536958] Object 0xffff8802c8b19ac8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.536972] Object 0xffff8802c8b19ad8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.536986] Object 0xffff8802c8b19ae8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537000] Object 0xffff8802c8b19af8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537014] Object 0xffff8802c8b19b08: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537029] Object 0xffff8802c8b19b18: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537043] Object 0xffff8802c8b19b28: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537057] Object 0xffff8802c8b19b38: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537071] Object 0xffff8802c8b19b48: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537085] Object 0xffff8802c8b19b58: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537100] Object 0xffff8802c8b19b68: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537114] Object 0xffff8802c8b19b78: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537128] Object 0xffff8802c8b19b88: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537143] Object 0xffff8802c8b19b98: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537158] Object 0xffff8802c8b19ba8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537172] Object 0xffff8802c8b19bb8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537187] Object 0xffff8802c8b19bc8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537201] Object 0xffff8802c8b19bd8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537215] Object 0xffff8802c8b19be8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537229] Object 0xffff8802c8b19bf8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537244] Object 0xffff8802c8b19c08: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537258] Object 0xffff8802c8b19c18: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537272] Object 0xffff8802c8b19c28: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537286] Object 0xffff8802c8b19c38: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537301] Object 0xffff8802c8b19c48: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537315] Object 0xffff8802c8b19c58: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537329] Object 0xffff8802c8b19c68: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537343] Object 0xffff8802c8b19c78: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537357] Object 0xffff8802c8b19c88: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537372] Object 0xffff8802c8b19c98: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537386] Object 0xffff8802c8b19ca8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537400] Object 0xffff8802c8b19cb8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537414] Object 0xffff8802c8b19cc8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537428] Object 0xffff8802c8b19cd8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537443] Object 0xffff8802c8b19ce8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537457] Object 0xffff8802c8b19cf8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537471] Object 0xffff8802c8b19d08: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537485] Object 0xffff8802c8b19d18: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537500] Object 0xffff8802c8b19d28: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537514] Object 0xffff8802c8b19d38: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537528] Object 0xffff8802c8b19d48: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537542] Object 0xffff8802c8b19d58: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537557] Object 0xffff8802c8b19d68: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537571] Object 0xffff8802c8b19d78: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537585] Object 0xffff8802c8b19d88: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537599] Object 0xffff8802c8b19d98: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537613] Object 0xffff8802c8b19da8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537628] Object 0xffff8802c8b19db8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537642] Object 0xffff8802c8b19dc8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537656] Object 0xffff8802c8b19dd8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537670] Object 0xffff8802c8b19de8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537685] Object 0xffff8802c8b19df8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537699] Object 0xffff8802c8b19e08: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537713] Object 0xffff8802c8b19e18: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537727] Object 0xffff8802c8b19e28: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537742] Object 0xffff8802c8b19e38: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537756] Object 0xffff8802c8b19e48: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537770] Object 0xffff8802c8b19e58: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537784] Object 0xffff8802c8b19e68: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537799] Object 0xffff8802c8b19e78: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537813] Object 0xffff8802c8b19e88: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537827] Object 0xffff8802c8b19e98: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537841] Object 0xffff8802c8b19ea8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537856] Object 0xffff8802c8b19eb8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537870] Object 0xffff8802c8b19ec8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537884] Object 0xffff8802c8b19ed8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537898] Object 0xffff8802c8b19ee8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537912] Object 0xffff8802c8b19ef8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537927] Object 0xffff8802c8b19f08: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537941] Object 0xffff8802c8b19f18: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537955] Object 0xffff8802c8b19f28: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537969] Object 0xffff8802c8b19f38: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537984] Object 0xffff8802c8b19f48: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.537998] Object 0xffff8802c8b19f58: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.538012] Object 0xffff8802c8b19f68: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.538026] Object 0xffff8802c8b19f78: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.538041] Object 0xffff8802c8b19f88: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.538055] Object 0xffff8802c8b19f98: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.538069] Object 0xffff8802c8b19fa8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.538083] Object 0xffff8802c8b19fb8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.538097] Object 0xffff8802c8b19fc8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.538112] Object 0xffff8802c8b19fd8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.538126] Object 0xffff8802c8b19fe8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.538140] Object 0xffff8802c8b19ff8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.538154] Object 0xffff8802c8b1a008: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.538169] Object 0xffff8802c8b1a018: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.538183] Object 0xffff8802c8b1a028: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.538197] Object 0xffff8802c8b1a038: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.538211] Object 0xffff8802c8b1a048: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.538226] Object 0xffff8802c8b1a058: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.538240] Object 0xffff8802c8b1a068: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.538254] Object 0xffff8802c8b1a078: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.538268] Object 0xffff8802c8b1a088: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.538283] Object 0xffff8802c8b1a098: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.538297] Object 0xffff8802c8b1a0a8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.538311] Object 0xffff8802c8b1a0b8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 789.538325] Object 0xffff8802c8b1a0c8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b a5 kkkkkkkkkkkkkkkAJPY
[ 789.538340] Redzone 0xffff8802c8b1a0d8: bb bb bb bb bb bb bb bb
A>>A>>A>>A>>A>>A>>A>>A>>
[ 789.538353] Padding 0xffff8802c8b1a118: 5a 5a 5a 5a 5a 5a 00 00
ZZZZZZ..
[ 789.538369] Pid: 5247, comm: qemu Not tainted 2.6.39 #2
[ 789.538373] Call Trace:
[ 789.538385] [<ffffffff810b7ccd>] ? check_bytes_and_report+0x10d/0x150
[ 789.538394] [<ffffffff81314260>] ? sock_alloc_send_pskb+0x1d0/0x320
[ 789.538400] [<ffffffff810b7db9>] ? check_object+0xa9/0x260
[ 789.538407] [<ffffffff81314260>] ? sock_alloc_send_pskb+0x1d0/0x320
[ 789.538414] [<ffffffff810b86f4>] ? alloc_debug_processing+0x104/0x190
[ 789.538420] [<ffffffff810b9ac2>] ? T.912+0x272/0x2d0
[ 789.538427] [<ffffffff810bb70d>] ? __kmalloc_track_caller+0x10d/0x160
[ 789.538434] [<ffffffff813198c2>] ? __alloc_skb+0x72/0x160
[ 789.538441] [<ffffffff81314260>] ? sock_alloc_send_pskb+0x1d0/0x320
[ 789.538450] [<ffffffff8105af7b>] ? getnstimeofday+0x4b/0xd0
[ 789.538457] [<ffffffff81320d32>] ? netif_rx+0xb2/0x120
[ 789.538465] [<ffffffff812b393e>] ? tun_chr_aio_write+0x18e/0x4e0
[ 789.538472] [<ffffffff812b37b0>] ? tun_sendmsg+0x4d0/0x4d0
[ 789.538481] [<ffffffff810c24e9>] ? do_sync_readv_writev+0xa9/0xf0
[ 789.538489] [<ffffffff810c231b>] ? rw_copy_check_uvector+0x7b/0x140
[ 789.538496] [<ffffffff810c2bcf>] ? do_readv_writev+0xdf/0x210
[ 789.538504] [<ffffffff810c2e7e>] ? sys_writev+0x4e/0xc0
[ 789.538513] [<ffffffff813d753b>] ? system_call_fastpath+0x16/0x1b
[ 789.538519] FIX kmalloc-2048: Restoring
0xffff8802c8b1a11e-0xffff8802c8b1a11f=0x5a
[ 789.538522]
[ 825.601291]
=============================================================================
[ 825.601298] BUG kmalloc-512: Object padding overwritten
[ 825.601302]
-----------------------------------------------------------------------------
[ 825.601306]
[ 825.601311] INFO: 0xffff88041411efe6-0xffff88041411efe7. First byte
0x0 instead of 0x5a
[ 825.601327] INFO: Allocated in sock_alloc_send_pskb+0x1d0/0x320
age=36742 cpu=3 pid=4846
[ 825.601338] INFO: Freed in __kfree_skb+0x11/0x90 age=36742 cpu=3 pid=3781
[ 825.601344] INFO: Slab 0xffffea000e463e20 objects=28 used=11
fp=0xffff88041411eda0 flags=0x80000000000040c1
[ 825.601350] INFO: Object 0xffff88041411eda0 @offset=11680
fp=0xffff88041411cdb0
[ 825.601353]
[ 825.601357] Bytes b4 0xffff88041411ed90: e2 96 ff ff 00 00 00 00 5a
5a 5a 5a 5a 5a 5a 5a Ac.A?A?....ZZZZZZZZ
[ 825.601373] Object 0xffff88041411eda0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 825.601388] Object 0xffff88041411edb0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 825.601402] Object 0xffff88041411edc0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 825.601416] Object 0xffff88041411edd0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 825.601430] Object 0xffff88041411ede0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 825.601444] Object 0xffff88041411edf0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 825.601459] Object 0xffff88041411ee00: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 825.601473] Object 0xffff88041411ee10: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 825.601487] Object 0xffff88041411ee20: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 825.601501] Object 0xffff88041411ee30: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 825.601515] Object 0xffff88041411ee40: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 825.601529] Object 0xffff88041411ee50: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 825.601543] Object 0xffff88041411ee60: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 825.601557] Object 0xffff88041411ee70: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 825.601571] Object 0xffff88041411ee80: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 825.601586] Object 0xffff88041411ee90: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 825.601600] Object 0xffff88041411eea0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 825.601614] Object 0xffff88041411eeb0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 825.601628] Object 0xffff88041411eec0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 825.601642] Object 0xffff88041411eed0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 825.601656] Object 0xffff88041411eee0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 825.601670] Object 0xffff88041411eef0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 825.601684] Object 0xffff88041411ef00: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 825.601699] Object 0xffff88041411ef10: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 825.601713] Object 0xffff88041411ef20: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 825.601727] Object 0xffff88041411ef30: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 825.601741] Object 0xffff88041411ef40: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 825.601755] Object 0xffff88041411ef50: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 825.601769] Object 0xffff88041411ef60: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 825.601783] Object 0xffff88041411ef70: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 825.601797] Object 0xffff88041411ef80: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 825.601812] Object 0xffff88041411ef90: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b a5 kkkkkkkkkkkkkkkAJPY
[ 825.601826] Redzone 0xffff88041411efa0: bb bb bb bb bb bb bb bb
A>>A>>A>>A>>A>>A>>A>>A>>
[ 825.601839] Padding 0xffff88041411efe0: 5a 5a 5a 5a 5a 5a 00 00
ZZZZZZ..
[ 825.601855] Pid: 7353, comm: apcaccess Not tainted 2.6.39 #2
[ 825.601859] Call Trace:
[ 825.601871] [<ffffffff810b7ccd>] ? check_bytes_and_report+0x10d/0x150
[ 825.601881] [<ffffffff8110773c>] ? load_elf_binary+0xa6c/0x1c00
[ 825.601887] [<ffffffff810b7db9>] ? check_object+0xa9/0x260
[ 825.601895] [<ffffffff8110773c>] ? load_elf_binary+0xa6c/0x1c00
[ 825.601902] [<ffffffff810b86f4>] ? alloc_debug_processing+0x104/0x190
[ 825.601908] [<ffffffff810b9ac2>] ? T.912+0x272/0x2d0
[ 825.601914] [<ffffffff810ba59d>] ? __kmalloc+0x10d/0x160
[ 825.601922] [<ffffffff8110773c>] ? load_elf_binary+0xa6c/0x1c00
[ 825.601930] [<ffffffff8109bd9d>] ? __get_user_pages+0x17d/0x530
[ 825.601938] [<ffffffff810c9556>] ? get_arg_page+0x56/0x100
[ 825.601945] [<ffffffff810c8070>] ? search_binary_handler+0x90/0x240
[ 825.601951] [<ffffffff810c9e9f>] ? do_execve+0x22f/0x2f0
[ 825.601959] [<ffffffff810094a6>] ? sys_execve+0x36/0x60
[ 825.601968] [<ffffffff813d78dc>] ? stub_execve+0x6c/0xc0
[ 825.601975] FIX kmalloc-512: Restoring
0xffff88041411efe6-0xffff88041411efe7=0x5a
[ 825.601978]
[ 842.610057] INFO: task ksmd:552 blocked for more than 120 seconds.
[ 842.610063] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[ 842.610069] ksmd D 0000000000000000 0 552 2
0x00000000
[ 842.610078] ffff88041db68ca0 0000000000000046 000212d01fc11d00
ffff880400000000
[ 842.610086] ffffffff81593020 ffff88041c1a9fd8 0000000000004000
ffff88041c1a8010
[ 842.610093] ffff88041c1a9fd8 ffff88041db68ca0 0000000000000000
0000000200000002
[ 842.610100] Call Trace:
[ 842.610116] [<ffffffff813d5595>] ? schedule_timeout+0x1c5/0x230
[ 842.610128] [<ffffffff8102f69c>] ? enqueue_task_fair+0x14c/0x190
[ 842.610137] [<ffffffff8102db27>] ? task_rq_lock+0x47/0x90
[ 842.610143] [<ffffffff813d4562>] ? wait_for_common+0xd2/0x180
[ 842.610150] [<ffffffff810339b0>] ? try_to_wake_up+0x2c0/0x2c0
[ 842.610157] [<ffffffff8104cb94>] ? flush_work+0x24/0x30
[ 842.610163] [<ffffffff8104be10>] ? do_work_for_cpu+0x20/0x20
[ 842.610170] [<ffffffff8104e0ab>] ? schedule_on_each_cpu+0xab/0xe0
[ 842.610177] [<ffffffff810b5c05>] ? ksm_scan_thread+0x7f5/0xc20
[ 842.610184] [<ffffffff81052a20>] ? wake_up_bit+0x40/0x40
[ 842.610190] [<ffffffff810b5410>] ?
try_to_merge_with_ksm_page+0x570/0x570
[ 842.610196] [<ffffffff810b5410>] ?
try_to_merge_with_ksm_page+0x570/0x570
[ 842.610203] [<ffffffff810525b6>] ? kthread+0x96/0xa0
[ 842.610210] [<ffffffff813d8294>] ? kernel_thread_helper+0x4/0x10
[ 842.610218] [<ffffffff81052520>] ? kthread_worker_fn+0x120/0x120
[ 842.610223] [<ffffffff813d8290>] ? gs_change+0xb/0xb
[ 842.610229] INFO: task fsnotify_mark:662 blocked for more than 120
seconds.
[ 842.610233] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[ 842.610237] fsnotify_mark D 0000000000000000 0 662 2
0x00000000
[ 842.610244] ffff88041c31d860 0000000000000046 0000000000000000
0000000000000000
[ 842.610251] ffffffff81593020 ffff88041c365fd8 0000000000004000
ffff88041c364010
[ 842.610257] ffff88041c365fd8 ffff88041c31d860 0000000000000000
0000000000000000
[ 842.610263] Call Trace:
[ 842.610270] [<ffffffff81035521>] ? load_balance+0x91/0x5e0
[ 842.610278] [<ffffffff813d5595>] ? schedule_timeout+0x1c5/0x230
[ 842.610285] [<ffffffff8102f323>] ? pick_next_task_fair+0x103/0x190
[ 842.610291] [<ffffffff813d4a6d>] ? schedule+0x28d/0x910
[ 842.610297] [<ffffffff813d4562>] ? wait_for_common+0xd2/0x180
[ 842.610303] [<ffffffff810339b0>] ? try_to_wake_up+0x2c0/0x2c0
[ 842.610310] [<ffffffff81074420>] ? synchronize_rcu_bh+0x50/0x50
[ 842.610316] [<ffffffff8107446a>] ? synchronize_sched+0x4a/0x50
[ 842.610322] [<ffffffff8104f940>] ? find_ge_pid+0x40/0x40
[ 842.610329] [<ffffffff8105742b>] ? __synchronize_srcu+0x5b/0xc0
[ 842.610338] [<ffffffff810f5f63>] ? fsnotify_mark_destroy+0x83/0x150
[ 842.610344] [<ffffffff81052a20>] ? wake_up_bit+0x40/0x40
[ 842.610352] [<ffffffff810f5ee0>] ?
fsnotify_set_mark_ignored_mask_locked+0x20/0x20
[ 842.610360] [<ffffffff810f5ee0>] ?
fsnotify_set_mark_ignored_mask_locked+0x20/0x20
[ 842.610367] [<ffffffff810525b6>] ? kthread+0x96/0xa0
[ 842.610373] [<ffffffff813d8294>] ? kernel_thread_helper+0x4/0x10
[ 842.610381] [<ffffffff81052520>] ? kthread_worker_fn+0x120/0x120
[ 842.610386] [<ffffffff813d8290>] ? gs_change+0xb/0xb
[ 842.610395] INFO: task jbd2/md0-8:2528 blocked for more than 120 seconds.
[ 842.610399] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[ 842.610403] jbd2/md0-8 D 0000000000000005 0 2528 2
0x00000000
[ 842.610410] ffff88041d91c570 0000000000000046 ffff88041baeea28
ffffea0000000000
[ 842.610417] ffff88041d91a5e0 ffff880419f9bfd8 0000000000004000
ffff880419f9a010
[ 842.610423] ffff880419f9bfd8 ffff88041d91c570 ffff88041baeea28
ffffffff810b8495
[ 842.610429] Call Trace:
[ 842.610435] [<ffffffff810b8495>] ? init_object+0x85/0xa0
[ 842.610442] [<ffffffff810b8916>] ? free_debug_processing+0x196/0x250
[ 842.610450] [<ffffffff8105adae>] ? ktime_get_ts+0x6e/0xf0
[ 842.610458] [<ffffffff810810f0>] ? __lock_page+0x70/0x70
[ 842.610464] [<ffffffff813d5174>] ? io_schedule+0x84/0xd0
[ 842.610473] [<ffffffff811d4403>] ?
radix_tree_gang_lookup_tag_slot+0x93/0xf0
[ 842.610480] [<ffffffff810810f9>] ? sleep_on_page+0x9/0x10
[ 842.610486] [<ffffffff813d57df>] ? __wait_on_bit+0x4f/0x80
[ 842.610492] [<ffffffff810812eb>] ? wait_on_page_bit+0x6b/0x80
[ 842.610499] [<ffffffff81052a50>] ? autoremove_wake_function+0x30/0x30
[ 842.610507] [<ffffffff8108a458>] ? pagevec_lookup_tag+0x18/0x20
[ 842.610512] [<ffffffff81081f2a>] ? filemap_fdatawait_range+0xfa/0x180
[ 842.610521] [<ffffffff811be09f>] ? submit_bio+0x6f/0xf0
[ 842.610530] [<ffffffff81176276>] ?
jbd2_journal_commit_transaction+0x796/0x1270
[ 842.610539] [<ffffffff81179ed1>] ? kjournald2+0xb1/0x1e0
[ 842.610546] [<ffffffff81052a20>] ? wake_up_bit+0x40/0x40
[ 842.610553] [<ffffffff81179e20>] ? commit_timeout+0x10/0x10
[ 842.610560] [<ffffffff81179e20>] ? commit_timeout+0x10/0x10
[ 842.610566] [<ffffffff810525b6>] ? kthread+0x96/0xa0
[ 842.610572] [<ffffffff813d8294>] ? kernel_thread_helper+0x4/0x10
[ 842.610580] [<ffffffff81052520>] ? kthread_worker_fn+0x120/0x120
[ 842.610585] [<ffffffff813d8290>] ? gs_change+0xb/0xb
[ 842.610594] INFO: task nfsd:4326 blocked for more than 120 seconds.
[ 842.610598] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[ 842.610602] nfsd D 0000000000000000 0 4326 2
0x00000000
[ 842.610608] ffff88041b604bc0 0000000000000046 0000000000000016
ffffffff00000000
[ 842.610614] ffff88041d8fa5e0 ffff88041c527fd8 0000000000004000
ffff88041c526010
[ 842.610621] ffff88041c527fd8 ffff88041b604bc0 000000000000009c
ffff880000000000
[ 842.610626] Call Trace:
[ 842.610634] [<ffffffff8101e845>] ? amd_flush_garts+0x105/0x140
[ 842.610640] [<ffffffff8101fa80>] ? gart_map_sg+0x480/0x480
[ 842.610646] [<ffffffff8101f5d3>] ? flush_gart+0x23/0x50
[ 842.610653] [<ffffffff81080f98>] ? find_get_page+0x18/0x90
[ 842.610660] [<ffffffff81174b95>] ? do_get_write_access+0x265/0x4a0
[ 842.610668] [<ffffffff81052a50>] ? autoremove_wake_function+0x30/0x30
[ 842.610675] [<ffffffff81174ef9>] ?
jbd2_journal_get_write_access+0x29/0x50
[ 842.610684] [<ffffffff8115f122>] ?
__ext4_journal_get_write_access+0x32/0x80
[ 842.610692] [<ffffffff81143908>] ? ext4_reserve_inode_write+0x78/0xa0
[ 842.610700] [<ffffffff81143970>] ? ext4_mark_inode_dirty+0x40/0x1e0
[ 842.610706] [<ffffffff81156c0b>] ? ext4_journal_start_sb+0x6b/0x160
[ 842.610715] [<ffffffff81322645>] ? dev_hard_start_xmit+0x305/0x5f0
[ 842.610723] [<ffffffff81352fd0>] ? ip_finish_output2+0x290/0x290
[ 842.610730] [<ffffffff81143c65>] ? ext4_dirty_inode+0x35/0x70
[ 842.610738] [<ffffffff810e4a08>] ? __mark_inode_dirty+0x38/0x210
[ 842.610745] [<ffffffff810d9317>] ? file_update_time+0xf7/0x180
[ 842.610751] [<ffffffff81082458>] ? __generic_file_aio_write+0x1f8/0x430
[ 842.610760] [<ffffffff81373ef9>] ? udp_sendmsg+0x3c9/0x7e0
[ 842.610767] [<ffffffff81314260>] ? sock_alloc_send_pskb+0x1d0/0x320
[ 842.610774] [<ffffffff81082703>] ? generic_file_aio_write+0x73/0xf0
[ 842.610781] [<ffffffff8113f74e>] ? ext4_file_write+0x6e/0x2b0
[ 842.610788] [<ffffffff810da8ac>] ? iget_locked+0x4c/0x140
[ 842.610794] [<ffffffff8119a590>] ? fh_compose+0x4c0/0x4c0
[ 842.610800] [<ffffffff8113f6e0>] ? ext4_llseek+0x110/0x110
[ 842.610808] [<ffffffff810c24e9>] ? do_sync_readv_writev+0xa9/0xf0
[ 842.610816] [<ffffffff810c231b>] ? rw_copy_check_uvector+0x7b/0x140
[ 842.610824] [<ffffffff810c2bcf>] ? do_readv_writev+0xdf/0x210
[ 842.610830] [<ffffffff810b8735>] ? alloc_debug_processing+0x145/0x190
[ 842.610836] [<ffffffff810b9bc5>] ? kmem_cache_alloc+0xa5/0xb0
[ 842.610843] [<ffffffff8113f463>] ? ext4_file_open+0x63/0x180
[ 842.610849] [<ffffffff8119c21d>] ? nfsd_vfs_write+0xed/0x3a0
[ 842.610856] [<ffffffff810c1127>] ? __dentry_open+0x1f7/0x2b0
[ 842.610862] [<ffffffff8119c892>] ? nfsd_open+0xf2/0x1b0
[ 842.610867] [<ffffffff8119cd34>] ? nfsd_write+0xf4/0x110
[ 842.610873] [<ffffffff81199930>] ? nfsd_proc_write+0xb0/0x120
[ 842.610880] [<ffffffff811971c5>] ? nfsd_dispatch+0xf5/0x230
[ 842.610886] [<ffffffff813b456f>] ? svc_process+0x4af/0x820
[ 842.610892] [<ffffffff810339b0>] ? try_to_wake_up+0x2c0/0x2c0
[ 842.610899] [<ffffffff811977a0>] ? nfsd_svc+0x1b0/0x1b0
[ 842.610906] [<ffffffff8119784d>] ? nfsd+0xad/0x150
[ 842.610912] [<ffffffff810525b6>] ? kthread+0x96/0xa0
[ 842.610918] [<ffffffff813d8294>] ? kernel_thread_helper+0x4/0x10
[ 842.610926] [<ffffffff81052520>] ? kthread_worker_fn+0x120/0x120
[ 842.610931] [<ffffffff813d8290>] ? gs_change+0xb/0xb
[ 854.241450]
=============================================================================
[ 854.241459] BUG kmalloc-1024: Object padding overwritten
[ 854.241463]
-----------------------------------------------------------------------------
[ 854.241466]
[ 854.241471] INFO: 0xffff8802c86059e6-0xffff8802c86059e7. First byte
0x0 instead of 0x5a
[ 854.241488] INFO: Allocated in __blockdev_direct_IO+0x16c/0xa90
age=518 cpu=5 pid=7094
[ 854.241500] INFO: Freed in __kfree_skb+0x11/0x90 age=519 cpu=5 pid=5247
[ 854.241507] INFO: Slab 0xffffea0009bd5000 objects=29 used=22
fp=0xffff8802c8606f50 flags=0x80000000000040c1
[ 854.241513] INFO: Object 0xffff8802c86055a0 @offset=21920
fp=0xffff8802c86066c0
[ 854.241516]
[ 854.241520] Bytes b4 0xffff8802c8605590: 00 00 00 00 00 00 00 00 5a
5a 5a 5a 5a 5a 5a 5a ........ZZZZZZZZ
[ 854.241536] Object 0xffff8802c86055a0: 00 00 00 00 00 00 00 00 e0
61 29 fa 03 88 ff ff ........A a)Ao..A?A?
[ 854.241551] Object 0xffff8802c86055b0: 00 00 00 00 00 00 00 00 00
00 41 ce 04 00 00 00 ..........AA?....
[ 854.241565] Object 0xffff8802c86055c0: 03 00 00 00 0c 00 00 00 00
00 00 00 01 00 00 00 ................
[ 854.241579] Object 0xffff8802c86055d0: 00 00 00 00 00 00 00 00 00
60 00 00 00 00 00 00 .........`......
[ 854.241593] Object 0xffff8802c86055e0: b7 2c 0a 00 00 00 00 00 00
00 00 00 00 00 00 00 A.,..............
[ 854.241607] Object 0xffff8802c86055f0: b7 2c 0a 00 00 00 00 00 00
00 00 00 00 00 00 00 A.,..............
[ 854.241621] Object 0xffff8802c8605600: 01 00 00 00 00 00 00 00 f0
77 14 81 ff ff ff ff ........A?w..A?A?A?A?
[ 854.241635] Object 0xffff8802c8605610: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 854.241649] Object 0xffff8802c8605620: 00 00 00 00 00 00 00 00 b7
54 3b 02 00 00 00 00 ........A.T;.....
[ 854.241663] Object 0xffff8802c8605630: b7 54 3b 02 00 00 00 00 20
00 00 00 00 00 00 00 A.T;.............
[ 854.241677] Object 0xffff8802c8605640: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 854.241691] Object 0xffff8802c8605650: b6 54 3b 02 00 00 00 00 00
10 00 00 00 00 00 00 A?T;.............
[ 854.241705] Object 0xffff8802c8605660: 00 00 00 00 00 00 00 00 00
80 8e 1a 04 88 ff ff ..............A?A?
[ 854.241719] Object 0xffff8802c8605670: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 854.241733] Object 0xffff8802c8605680: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 854.241747] Object 0xffff8802c8605690: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 854.241761] Object 0xffff8802c86056a0: 00 00 00 00 00 00 00 00 00
00 00 00 00 10 00 00 ................
[ 854.241775] Object 0xffff8802c86056b0: b6 54 3b 02 00 00 00 00 00
60 cb a2 00 00 00 00 A?T;......`A?Ac....
[ 854.241789] Object 0xffff8802c86056c0: 06 06 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 854.241803] Object 0xffff8802c86056d0: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 854.241817] Object 0xffff8802c86056e0: 68 fd 1c c9 02 88 ff ff 00
00 00 00 00 00 00 00 hA 1/2 .A?..A?A?........
[ 854.241831] Object 0xffff8802c86056f0: 00 60 00 00 00 00 00 00 01
00 00 00 01 00 00 00 .`..............
[ 854.241845] Object 0xffff8802c8605700: 00 80 c9 dd 39 7f 00 00 01
00 00 00 01 00 00 00 ..A?A?9...........
[ 854.241860] Object 0xffff8802c8605710: 00 00 00 00 00 00 00 00 08
81 bb 0c 00 ea ff ff ..........A>>..AaA?A?
[ 854.241874] Object 0xffff8802c8605720: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.241888] Object 0xffff8802c8605730: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.241902] Object 0xffff8802c8605740: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.241916] Object 0xffff8802c8605750: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.241930] Object 0xffff8802c8605760: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.241944] Object 0xffff8802c8605770: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.241959] Object 0xffff8802c8605780: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.241973] Object 0xffff8802c8605790: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.241987] Object 0xffff8802c86057a0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.242001] Object 0xffff8802c86057b0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.242015] Object 0xffff8802c86057c0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.242029] Object 0xffff8802c86057d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.242043] Object 0xffff8802c86057e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.242058] Object 0xffff8802c86057f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.242072] Object 0xffff8802c8605800: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.242086] Object 0xffff8802c8605810: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.242100] Object 0xffff8802c8605820: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.242114] Object 0xffff8802c8605830: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.242128] Object 0xffff8802c8605840: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.242142] Object 0xffff8802c8605850: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.242157] Object 0xffff8802c8605860: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.242171] Object 0xffff8802c8605870: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.242185] Object 0xffff8802c8605880: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.242199] Object 0xffff8802c8605890: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.242213] Object 0xffff8802c86058a0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.242227] Object 0xffff8802c86058b0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.242241] Object 0xffff8802c86058c0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.242256] Object 0xffff8802c86058d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.242270] Object 0xffff8802c86058e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.242284] Object 0xffff8802c86058f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.242298] Object 0xffff8802c8605900: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.242312] Object 0xffff8802c8605910: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.242326] Object 0xffff8802c8605920: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.242340] Object 0xffff8802c8605930: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.242355] Object 0xffff8802c8605940: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.242369] Object 0xffff8802c8605950: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.242383] Object 0xffff8802c8605960: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.242397] Object 0xffff8802c8605970: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.242411] Object 0xffff8802c8605980: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 854.242425] Object 0xffff8802c8605990: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b a5 kkkkkkkkkkkkkkkAJPY
[ 854.242440] Redzone 0xffff8802c86059a0: cc cc cc cc cc cc cc cc
A?A?A?A?A?A?A?A?
[ 854.242453] Padding 0xffff8802c86059e0: 5a 5a 5a 5a 5a 5a 00 00
ZZZZZZ..
[ 854.242470] Pid: 7094, comm: qemu Not tainted 2.6.39 #2
[ 854.242474] Call Trace:
[ 854.242485] [<ffffffff810b7ccd>] ? check_bytes_and_report+0x10d/0x150
[ 854.242492] [<ffffffff810b7db9>] ? check_object+0xa9/0x260
[ 854.242500] [<ffffffff810f3126>] ? __blockdev_direct_IO+0xa16/0xa90
[ 854.242507] [<ffffffff810b88b3>] ? free_debug_processing+0x133/0x250
[ 854.242513] [<ffffffff810b8afb>] ? __slab_free+0x12b/0x140
[ 854.242520] [<ffffffff810f3126>] ? __blockdev_direct_IO+0xa16/0xa90
[ 854.242530] [<ffffffff81144a17>] ? ext4_ind_direct_IO+0xf7/0x410
[ 854.242536] [<ffffffff811477f0>] ? noalloc_get_block_write+0x30/0x30
[ 854.242543] [<ffffffff81082029>] ? __filemap_fdatawrite_range+0x49/0x50
[ 854.242550] [<ffffffff810830c3>] ? generic_file_aio_read+0x653/0x6d0
[ 854.242558] [<ffffffff810567a2>] ? hrtimer_cancel+0x12/0x20
[ 854.242565] [<ffffffff81062137>] ? futex_wait+0x197/0x240
[ 854.242572] [<ffffffff81082a70>] ? delete_from_page_cache+0x70/0x70
[ 854.242580] [<ffffffff810c24e9>] ? do_sync_readv_writev+0xa9/0xf0
[ 854.242587] [<ffffffff8106239f>] ? futex_wake+0x10f/0x120
[ 854.242594] [<ffffffff81063f7b>] ? do_futex+0x11b/0xa70
[ 854.242601] [<ffffffff810471cc>] ? T.680+0x13c/0x280
[ 854.242608] [<ffffffff810c231b>] ? rw_copy_check_uvector+0x7b/0x140
[ 854.242616] [<ffffffff810c2bcf>] ? do_readv_writev+0xdf/0x210
[ 854.242624] [<ffffffff810c3003>] ? sys_preadv+0xc3/0xd0
[ 854.242633] [<ffffffff813d753b>] ? system_call_fastpath+0x16/0x1b
[ 854.242640] FIX kmalloc-1024: Restoring
0xffff8802c86059e6-0xffff8802c86059e7=0x5a
[ 854.242643]
[ 860.203918]
=============================================================================
[ 860.203927] BUG kmalloc-1024: Object padding overwritten
[ 860.203931]
-----------------------------------------------------------------------------
[ 860.203934]
[ 860.203939] INFO: 0xffff88041ad15156-0xffff88041ad15157. First byte
0x0 instead of 0x5a
[ 860.203955] INFO: Allocated in __blockdev_direct_IO+0x16c/0xa90
age=1609 cpu=3 pid=7094
[ 860.203965] INFO: Freed in __blockdev_direct_IO+0xa16/0xa90 age=1609
cpu=3 pid=7094
[ 860.203972] INFO: Slab 0xffffea000e5ddb80 objects=29 used=25
fp=0xffff88041ad14d10 flags=0x80000000000040c1
[ 860.203977] INFO: Object 0xffff88041ad14d10 @offset=19728
fp=0xffff88041ad10000
[ 860.203981]
[ 860.203984] Bytes b4 0xffff88041ad14d00: 62 28 00 00 01 00 00 00 5a
5a 5a 5a 5a 5a 5a 5a b(......ZZZZZZZZ
[ 860.204000] Object 0xffff88041ad14d10: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204015] Object 0xffff88041ad14d20: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204030] Object 0xffff88041ad14d30: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204044] Object 0xffff88041ad14d40: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204058] Object 0xffff88041ad14d50: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204073] Object 0xffff88041ad14d60: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204087] Object 0xffff88041ad14d70: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204101] Object 0xffff88041ad14d80: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204116] Object 0xffff88041ad14d90: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204130] Object 0xffff88041ad14da0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204144] Object 0xffff88041ad14db0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204159] Object 0xffff88041ad14dc0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204173] Object 0xffff88041ad14dd0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204187] Object 0xffff88041ad14de0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204202] Object 0xffff88041ad14df0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204216] Object 0xffff88041ad14e00: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204230] Object 0xffff88041ad14e10: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204245] Object 0xffff88041ad14e20: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204259] Object 0xffff88041ad14e30: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204273] Object 0xffff88041ad14e40: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204288] Object 0xffff88041ad14e50: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204302] Object 0xffff88041ad14e60: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204316] Object 0xffff88041ad14e70: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204331] Object 0xffff88041ad14e80: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204345] Object 0xffff88041ad14e90: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204359] Object 0xffff88041ad14ea0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204374] Object 0xffff88041ad14eb0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204388] Object 0xffff88041ad14ec0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204402] Object 0xffff88041ad14ed0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204417] Object 0xffff88041ad14ee0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204431] Object 0xffff88041ad14ef0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204445] Object 0xffff88041ad14f00: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204460] Object 0xffff88041ad14f10: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204474] Object 0xffff88041ad14f20: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204488] Object 0xffff88041ad14f30: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204503] Object 0xffff88041ad14f40: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204517] Object 0xffff88041ad14f50: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204531] Object 0xffff88041ad14f60: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204546] Object 0xffff88041ad14f70: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204560] Object 0xffff88041ad14f80: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204574] Object 0xffff88041ad14f90: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204589] Object 0xffff88041ad14fa0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204603] Object 0xffff88041ad14fb0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204618] Object 0xffff88041ad14fc0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204632] Object 0xffff88041ad14fd0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204646] Object 0xffff88041ad14fe0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204661] Object 0xffff88041ad14ff0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204675] Object 0xffff88041ad15000: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204689] Object 0xffff88041ad15010: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204704] Object 0xffff88041ad15020: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204718] Object 0xffff88041ad15030: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204732] Object 0xffff88041ad15040: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204747] Object 0xffff88041ad15050: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204761] Object 0xffff88041ad15060: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204775] Object 0xffff88041ad15070: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204790] Object 0xffff88041ad15080: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204804] Object 0xffff88041ad15090: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204818] Object 0xffff88041ad150a0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204833] Object 0xffff88041ad150b0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204847] Object 0xffff88041ad150c0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204861] Object 0xffff88041ad150d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204876] Object 0xffff88041ad150e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204890] Object 0xffff88041ad150f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 860.204904] Object 0xffff88041ad15100: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b a5 kkkkkkkkkkkkkkkAJPY
[ 860.204919] Redzone 0xffff88041ad15110: bb bb bb bb bb bb bb bb
A>>A>>A>>A>>A>>A>>A>>A>>
[ 860.204932] Padding 0xffff88041ad15150: 5a 5a 5a 5a 5a 5a 00 00
ZZZZZZ..
[ 860.204949] Pid: 7100, comm: qemu Not tainted 2.6.39 #2
[ 860.204953] Call Trace:
[ 860.204964] [<ffffffff810b7ccd>] ? check_bytes_and_report+0x10d/0x150
[ 860.204973] [<ffffffff810f287c>] ? __blockdev_direct_IO+0x16c/0xa90
[ 860.204979] [<ffffffff810b7db9>] ? check_object+0xa9/0x260
[ 860.204986] [<ffffffff810f287c>] ? __blockdev_direct_IO+0x16c/0xa90
[ 860.204993] [<ffffffff810b86f4>] ? alloc_debug_processing+0x104/0x190
[ 860.204999] [<ffffffff810b9ac2>] ? T.912+0x272/0x2d0
[ 860.205006] [<ffffffff810b9bc5>] ? kmem_cache_alloc+0xa5/0xb0
[ 860.205013] [<ffffffff810f287c>] ? __blockdev_direct_IO+0x16c/0xa90
[ 860.205022] [<ffffffff8108a475>] ? pagevec_lookup+0x15/0x20
[ 860.205029] [<ffffffff8108b5b7>] ?
invalidate_inode_pages2_range+0x87/0x300
[ 860.205039] [<ffffffff81144e41>] ? ext4_direct_IO+0x111/0x1f0
[ 860.205045] [<ffffffff811477a0>] ? _ext4_get_block+0x160/0x160
[ 860.205052] [<ffffffff81144f20>] ? ext4_direct_IO+0x1f0/0x1f0
[ 860.205058] [<ffffffff81082182>] ? generic_file_direct_write+0xd2/0x1b0
[ 860.205064] [<ffffffff8108252b>] ? __generic_file_aio_write+0x2cb/0x430
[ 860.205071] [<ffffffff81082703>] ? generic_file_aio_write+0x73/0xf0
[ 860.205079] [<ffffffff8113f74e>] ? ext4_file_write+0x6e/0x2b0
[ 860.205087] [<ffffffff810c25ef>] ? do_sync_write+0xbf/0x100
[ 860.205095] [<ffffffff810471cc>] ? T.680+0x13c/0x280
[ 860.205103] [<ffffffff810485a8>] ? kill_pid_info+0x38/0x60
[ 860.205110] [<ffffffff81048854>] ? sys_kill+0x94/0x1b0
[ 860.205117] [<ffffffff810c3198>] ? vfs_write+0xc8/0x190
[ 860.205124] [<ffffffff810c32fb>] ? sys_pwrite64+0x9b/0xb0
[ 860.205134] [<ffffffff813d753b>] ? system_call_fastpath+0x16/0x1b
[ 860.205140] FIX kmalloc-1024: Restoring
0xffff88041ad15156-0xffff88041ad15157=0x5a
[ 860.205143]
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 111+ messages in thread
* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
2011-06-01 9:29 ` Brad Campbell
@ 2011-06-01 9:40 ` Avi Kivity
-1 siblings, 0 replies; 111+ messages in thread
From: Avi Kivity @ 2011-06-01 9:40 UTC (permalink / raw)
To: Brad Campbell
Cc: Hugh Dickins, Andrea Arcangeli, Borislav Petkov, linux-kernel,
kvm, linux-mm
On 06/01/2011 12:29 PM, Brad Campbell wrote:
> On 01/06/11 14:56, Avi Kivity wrote:
>> On 06/01/2011 09:31 AM, Brad Campbell wrote:
>>> On 01/06/11 12:52, Hugh Dickins wrote:
>>>
>>>>
>>>> I guess Brad could try SLUB debugging, boot with slub_debug=P
>>>> for poisoning perhaps; though it might upset alignments and
>>>> drive the problem underground. Or see if the same happens
>>>> with SLAB instead of SLUB.
>>>
>>> Not much use I'm afraid.
>>> This is all I get in the log
>>>
>>> [ 3161.300073]
>>> =============================================================================
>>>
>>>
>>> [ 3161.300147] BUG kmalloc-512: Freechain corrupt
>>>
>>> The qemu process is then frozen, unkillable but reported in state "R"
>>>
>>> 13881 ? R 3:27 /usr/bin/qemu -S -M pc-0.13 -enable-kvm -m 1024 -smp
>>> 2,sockets=2,cores=1,threads=1 -nam
>>>
>>> The machine then progressively dies until it's frozen solid with no
>>> further error messages.
>>>
>>> I stupidly forgot to do an alt-sysrq-t prior to doing an alt-sysrq-b,
>>> but at least it responded to that.
>>>
>>> On the bright side I can reproduce it at will.
>>
>> Please try slub_debug=FZPU; that should point the finger (hopefully at
>> somebody else).
>>
>
> Well the first attempt locked the machine solid. No network, no console..
>
> I saw
> "=========================================================================="
>
> on the console.. nothing after that. Would not respond to sysrq-t or
> any other sysrq combination other than -b, which rebooted the box.
>
>
> No output on netconsole at all, I had to walk to the other building to
> look at the monitor and reboot it.
>
> The second attempt jammed netconsole again, but I managed to get this
> from an ssh session I already had established. The machine died a slow
> and horrible death, but remained interactive enough for me to reboot
> it with
>
> echo b > /proc/sysrq-trigger
>
> Nothing else worked.
>
>
> [ 413.756416] [<ffffffff81318f1c>] ? pskb_expand_head+0x15c/0x250
> [ 413.756424] [<ffffffff813a6c45>] ? nf_bridge_copy_header+0x145/0x160
> [ 413.756431] [<ffffffff8139f78d>] ? br_dev_queue_push_xmit+0x6d/0x80
> [ 413.756439] [<ffffffff813a55a0>] ? br_nf_post_routing+0x2a0/0x2f0
> [ 413.756447] [<ffffffff81346bc4>] ? nf_iterate+0x84/0xb0
> [ 413.756453] [<ffffffff8139f720>] ? br_flood_deliver+0x20/0x20
> [ 413.756459] [<ffffffff81346c64>] ? nf_hook_slow+0x74/0x120
> [ 413.756465] [<ffffffff8139f720>] ? br_flood_deliver+0x20/0x20
> [ 413.756472] [<ffffffff8139f7da>] ? br_forward_finish+0x3a/0x60
> [ 413.756479] [<ffffffff813a5758>] ? br_nf_forward_finish+0x168/0x170
> [ 413.756487] [<ffffffff813a5c90>] ? br_nf_forward_ip+0x360/0x3a0
> [ 413.756492] [<ffffffff81346bc4>] ? nf_iterate+0x84/0xb0
> [ 413.756498] [<ffffffff8139f7a0>] ? br_dev_queue_push_xmit+0x80/0x80
> [ 413.756504] [<ffffffff81346c64>] ? nf_hook_slow+0x74/0x120
> [ 413.756510] [<ffffffff8139f7a0>] ? br_dev_queue_push_xmit+0x80/0x80
> [ 413.756516] [<ffffffff8139f800>] ? br_forward_finish+0x60/0x60
> [ 413.756522] [<ffffffff8139f800>] ? br_forward_finish+0x60/0x60
> [ 413.756528] [<ffffffff8139f875>] ? __br_forward+0x75/0xc0
> [ 413.756534] [<ffffffff8139f426>] ? deliver_clone+0x36/0x60
> [ 413.756540] [<ffffffff8139f69d>] ? br_flood+0xbd/0x100
> [ 413.756546] [<ffffffff813a05b0>] ? br_handle_local_finish+0x40/0x40
> [ 413.756552] [<ffffffff813a080e>] ? br_handle_frame_finish+0x25e/0x280
> [ 413.756560] [<ffffffff813a60f0>] ?
> br_nf_pre_routing_finish+0x1a0/0x330
> [ 413.756568] [<ffffffff813a6958>] ? br_nf_pre_routing+0x6d8/0x800
> [ 413.756577] [<ffffffff8102d46a>] ? enqueue_task+0x3a/0x90
> [ 413.756582] [<ffffffff81346bc4>] ? nf_iterate+0x84/0xb0
> [ 413.756589] [<ffffffff813a05b0>] ? br_handle_local_finish+0x40/0x40
> [ 413.756594] [<ffffffff81346c64>] ? nf_hook_slow+0x74/0x120
> [ 413.756600] [<ffffffff813a05b0>] ? br_handle_local_finish+0x40/0x40
> [ 413.756607] [<ffffffff810339b0>] ? try_to_wake_up+0x2c0/0x2c0
> [ 413.756613] [<ffffffff813a09d9>] ? br_handle_frame+0x1a9/0x280
> [ 413.756620] [<ffffffff813a0830>] ? br_handle_frame_finish+0x280/0x280
> [ 413.756627] [<ffffffff81320ef7>] ? __netif_receive_skb+0x157/0x5c0
> [ 413.756634] [<ffffffff81321443>] ? process_backlog+0xe3/0x1d0
> [ 413.756641] [<ffffffff81321da5>] ? net_rx_action+0xc5/0x1d0
> [ 413.756650] [<ffffffff8103df11>] ? __do_softirq+0x91/0x120
> [ 413.756657] [<ffffffff813d838c>] ? call_softirq+0x1c/0x30
> [ 413.756660] <EOI> [<ffffffff81003cbd>] ? do_softirq+0x4d/0x80
> [ 413.756673] [<ffffffff81321ece>] ? netif_rx_ni+0x1e/0x30
> [ 413.756681] [<ffffffff812b3ae2>] ? tun_chr_aio_write+0x332/0x4e0
> [ 413.756688] [<ffffffff812b37b0>] ? tun_sendmsg+0x4d0/0x4d0
> [ 413.756697] [<ffffffff810c24e9>] ? do_sync_readv_writev+0xa9/0xf0
> [ 413.756704] [<ffffffff81063f9c>] ? do_futex+0x13c/0xa70
> [ 413.756711] [<ffffffff811d6730>] ? timerqueue_add+0x60/0xb0
> [ 413.756719] [<ffffffff81056ab7>] ?
> __hrtimer_start_range_ns+0x1e7/0x410
> [ 413.756726] [<ffffffff810c231b>] ? rw_copy_check_uvector+0x7b/0x140
> [ 413.756734] [<ffffffff810c2bcf>] ? do_readv_writev+0xdf/0x210
> [ 413.756742] [<ffffffff810c2e7e>] ? sys_writev+0x4e/0xc0
> [ 413.756750] [<ffffffff813d753b>] ? system_call_fastpath+0x16/0x1b
> [ 413.756756] FIX kmalloc-1024: Restoring
> 0xffff880417179566-0xffff880417179567=0x5a
bridge and netfilter, IIRC this was also the problem last time.
Do you have any ebtables loaded?
Can you try building a kernel without ebtables? Without netfilter at all?
Please run all tests with slub_debug=FZPU.
--
error compiling committee.c: too many arguments to function
^ permalink raw reply [flat|nested] 111+ messages in thread
* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
@ 2011-06-01 9:40 ` Avi Kivity
0 siblings, 0 replies; 111+ messages in thread
From: Avi Kivity @ 2011-06-01 9:40 UTC (permalink / raw)
To: Brad Campbell
Cc: Hugh Dickins, Andrea Arcangeli, Borislav Petkov, linux-kernel,
kvm, linux-mm
On 06/01/2011 12:29 PM, Brad Campbell wrote:
> On 01/06/11 14:56, Avi Kivity wrote:
>> On 06/01/2011 09:31 AM, Brad Campbell wrote:
>>> On 01/06/11 12:52, Hugh Dickins wrote:
>>>
>>>>
>>>> I guess Brad could try SLUB debugging, boot with slub_debug=P
>>>> for poisoning perhaps; though it might upset alignments and
>>>> drive the problem underground. Or see if the same happens
>>>> with SLAB instead of SLUB.
>>>
>>> Not much use I'm afraid.
>>> This is all I get in the log
>>>
>>> [ 3161.300073]
>>> =============================================================================
>>>
>>>
>>> [ 3161.300147] BUG kmalloc-512: Freechain corrupt
>>>
>>> The qemu process is then frozen, unkillable but reported in state "R"
>>>
>>> 13881 ? R 3:27 /usr/bin/qemu -S -M pc-0.13 -enable-kvm -m 1024 -smp
>>> 2,sockets=2,cores=1,threads=1 -nam
>>>
>>> The machine then progressively dies until it's frozen solid with no
>>> further error messages.
>>>
>>> I stupidly forgot to do an alt-sysrq-t prior to doing an alt-sysrq-b,
>>> but at least it responded to that.
>>>
>>> On the bright side I can reproduce it at will.
>>
>> Please try slub_debug=FZPU; that should point the finger (hopefully at
>> somebody else).
>>
>
> Well the first attempt locked the machine solid. No network, no console..
>
> I saw
> "=========================================================================="
>
> on the console.. nothing after that. Would not respond to sysrq-t or
> any other sysrq combination other than -b, which rebooted the box.
>
>
> No output on netconsole at all, I had to walk to the other building to
> look at the monitor and reboot it.
>
> The second attempt jammed netconsole again, but I managed to get this
> from an ssh session I already had established. The machine died a slow
> and horrible death, but remained interactive enough for me to reboot
> it with
>
> echo b > /proc/sysrq-trigger
>
> Nothing else worked.
>
>
> [ 413.756416] [<ffffffff81318f1c>] ? pskb_expand_head+0x15c/0x250
> [ 413.756424] [<ffffffff813a6c45>] ? nf_bridge_copy_header+0x145/0x160
> [ 413.756431] [<ffffffff8139f78d>] ? br_dev_queue_push_xmit+0x6d/0x80
> [ 413.756439] [<ffffffff813a55a0>] ? br_nf_post_routing+0x2a0/0x2f0
> [ 413.756447] [<ffffffff81346bc4>] ? nf_iterate+0x84/0xb0
> [ 413.756453] [<ffffffff8139f720>] ? br_flood_deliver+0x20/0x20
> [ 413.756459] [<ffffffff81346c64>] ? nf_hook_slow+0x74/0x120
> [ 413.756465] [<ffffffff8139f720>] ? br_flood_deliver+0x20/0x20
> [ 413.756472] [<ffffffff8139f7da>] ? br_forward_finish+0x3a/0x60
> [ 413.756479] [<ffffffff813a5758>] ? br_nf_forward_finish+0x168/0x170
> [ 413.756487] [<ffffffff813a5c90>] ? br_nf_forward_ip+0x360/0x3a0
> [ 413.756492] [<ffffffff81346bc4>] ? nf_iterate+0x84/0xb0
> [ 413.756498] [<ffffffff8139f7a0>] ? br_dev_queue_push_xmit+0x80/0x80
> [ 413.756504] [<ffffffff81346c64>] ? nf_hook_slow+0x74/0x120
> [ 413.756510] [<ffffffff8139f7a0>] ? br_dev_queue_push_xmit+0x80/0x80
> [ 413.756516] [<ffffffff8139f800>] ? br_forward_finish+0x60/0x60
> [ 413.756522] [<ffffffff8139f800>] ? br_forward_finish+0x60/0x60
> [ 413.756528] [<ffffffff8139f875>] ? __br_forward+0x75/0xc0
> [ 413.756534] [<ffffffff8139f426>] ? deliver_clone+0x36/0x60
> [ 413.756540] [<ffffffff8139f69d>] ? br_flood+0xbd/0x100
> [ 413.756546] [<ffffffff813a05b0>] ? br_handle_local_finish+0x40/0x40
> [ 413.756552] [<ffffffff813a080e>] ? br_handle_frame_finish+0x25e/0x280
> [ 413.756560] [<ffffffff813a60f0>] ?
> br_nf_pre_routing_finish+0x1a0/0x330
> [ 413.756568] [<ffffffff813a6958>] ? br_nf_pre_routing+0x6d8/0x800
> [ 413.756577] [<ffffffff8102d46a>] ? enqueue_task+0x3a/0x90
> [ 413.756582] [<ffffffff81346bc4>] ? nf_iterate+0x84/0xb0
> [ 413.756589] [<ffffffff813a05b0>] ? br_handle_local_finish+0x40/0x40
> [ 413.756594] [<ffffffff81346c64>] ? nf_hook_slow+0x74/0x120
> [ 413.756600] [<ffffffff813a05b0>] ? br_handle_local_finish+0x40/0x40
> [ 413.756607] [<ffffffff810339b0>] ? try_to_wake_up+0x2c0/0x2c0
> [ 413.756613] [<ffffffff813a09d9>] ? br_handle_frame+0x1a9/0x280
> [ 413.756620] [<ffffffff813a0830>] ? br_handle_frame_finish+0x280/0x280
> [ 413.756627] [<ffffffff81320ef7>] ? __netif_receive_skb+0x157/0x5c0
> [ 413.756634] [<ffffffff81321443>] ? process_backlog+0xe3/0x1d0
> [ 413.756641] [<ffffffff81321da5>] ? net_rx_action+0xc5/0x1d0
> [ 413.756650] [<ffffffff8103df11>] ? __do_softirq+0x91/0x120
> [ 413.756657] [<ffffffff813d838c>] ? call_softirq+0x1c/0x30
> [ 413.756660] <EOI> [<ffffffff81003cbd>] ? do_softirq+0x4d/0x80
> [ 413.756673] [<ffffffff81321ece>] ? netif_rx_ni+0x1e/0x30
> [ 413.756681] [<ffffffff812b3ae2>] ? tun_chr_aio_write+0x332/0x4e0
> [ 413.756688] [<ffffffff812b37b0>] ? tun_sendmsg+0x4d0/0x4d0
> [ 413.756697] [<ffffffff810c24e9>] ? do_sync_readv_writev+0xa9/0xf0
> [ 413.756704] [<ffffffff81063f9c>] ? do_futex+0x13c/0xa70
> [ 413.756711] [<ffffffff811d6730>] ? timerqueue_add+0x60/0xb0
> [ 413.756719] [<ffffffff81056ab7>] ?
> __hrtimer_start_range_ns+0x1e7/0x410
> [ 413.756726] [<ffffffff810c231b>] ? rw_copy_check_uvector+0x7b/0x140
> [ 413.756734] [<ffffffff810c2bcf>] ? do_readv_writev+0xdf/0x210
> [ 413.756742] [<ffffffff810c2e7e>] ? sys_writev+0x4e/0xc0
> [ 413.756750] [<ffffffff813d753b>] ? system_call_fastpath+0x16/0x1b
> [ 413.756756] FIX kmalloc-1024: Restoring
> 0xffff880417179566-0xffff880417179567=0x5a
bridge and netfilter, IIRC this was also the problem last time.
Do you have any ebtables loaded?
Can you try building a kernel without ebtables? Without netfilter at all?
Please run all tests with slub_debug=FZPU.
--
error compiling committee.c: too many arguments to function
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 111+ messages in thread
* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
2011-06-01 9:40 ` Avi Kivity
@ 2011-06-01 9:41 ` Avi Kivity
-1 siblings, 0 replies; 111+ messages in thread
From: Avi Kivity @ 2011-06-01 9:41 UTC (permalink / raw)
To: Brad Campbell
Cc: Hugh Dickins, Andrea Arcangeli, Borislav Petkov, linux-kernel,
kvm, linux-mm
On 06/01/2011 12:40 PM, Avi Kivity wrote:
>
> bridge and netfilter, IIRC this was also the problem last time.
>
> Do you have any ebtables loaded?
>
> Can you try building a kernel without ebtables? Without netfilter at
> all?
>
> Please run all tests with slub_debug=FZPU.
>
--
error compiling committee.c: too many arguments to function
^ permalink raw reply [flat|nested] 111+ messages in thread
* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
@ 2011-06-01 9:41 ` Avi Kivity
0 siblings, 0 replies; 111+ messages in thread
From: Avi Kivity @ 2011-06-01 9:41 UTC (permalink / raw)
To: Brad Campbell
Cc: Hugh Dickins, Andrea Arcangeli, Borislav Petkov, linux-kernel,
kvm, linux-mm
On 06/01/2011 12:40 PM, Avi Kivity wrote:
>
> bridge and netfilter, IIRC this was also the problem last time.
>
> Do you have any ebtables loaded?
>
> Can you try building a kernel without ebtables? Without netfilter at
> all?
>
> Please run all tests with slub_debug=FZPU.
>
--
error compiling committee.c: too many arguments to function
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 111+ messages in thread
* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
2011-06-01 9:41 ` Avi Kivity
@ 2011-06-01 10:53 ` Brad Campbell
-1 siblings, 0 replies; 111+ messages in thread
From: Brad Campbell @ 2011-06-01 10:53 UTC (permalink / raw)
To: Avi Kivity
Cc: Hugh Dickins, Andrea Arcangeli, Borislav Petkov, linux-kernel,
kvm, linux-mm
On 01/06/11 17:41, Avi Kivity wrote:
> On 06/01/2011 12:40 PM, Avi Kivity wrote:
>>
>> bridge and netfilter, IIRC this was also the problem last time.
>>
>> Do you have any ebtables loaded?
Never heard of them, but making a cursory check just in case..
brad@srv:/raid10/src/linux-2.6.39$ grep EBTABLE .config
# CONFIG_BRIDGE_NF_EBTABLES is not set
>> Can you try building a kernel without ebtables? Without netfilter at all?
Well, without netfilter I can't get it to crash. The problem is without
netfilter I can't actually use it the way I use it to get it to crash.
I rebooted into a netfilter kernel, and did all the steps I'd used on
the no-netfilter kernel and it ticked along happily.
So the result of the experiment is inconclusive. Having said that, the
backtraces certainly smell networky.
To get it to crash, I have to start IE in the VM and https to the public
address of the machine, which is then redirected by netfilter back into
another of the VM's.
I can https directly to the other VM's address, but that does not cause
it to crash, however without netfilter loaded I can't bounce off the
public IP. It's all rather confusing really.
What next Sherlock?
^ permalink raw reply [flat|nested] 111+ messages in thread
* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
@ 2011-06-01 10:53 ` Brad Campbell
0 siblings, 0 replies; 111+ messages in thread
From: Brad Campbell @ 2011-06-01 10:53 UTC (permalink / raw)
To: Avi Kivity
Cc: Hugh Dickins, Andrea Arcangeli, Borislav Petkov, linux-kernel,
kvm, linux-mm
On 01/06/11 17:41, Avi Kivity wrote:
> On 06/01/2011 12:40 PM, Avi Kivity wrote:
>>
>> bridge and netfilter, IIRC this was also the problem last time.
>>
>> Do you have any ebtables loaded?
Never heard of them, but making a cursory check just in case..
brad@srv:/raid10/src/linux-2.6.39$ grep EBTABLE .config
# CONFIG_BRIDGE_NF_EBTABLES is not set
>> Can you try building a kernel without ebtables? Without netfilter at all?
Well, without netfilter I can't get it to crash. The problem is without
netfilter I can't actually use it the way I use it to get it to crash.
I rebooted into a netfilter kernel, and did all the steps I'd used on
the no-netfilter kernel and it ticked along happily.
So the result of the experiment is inconclusive. Having said that, the
backtraces certainly smell networky.
To get it to crash, I have to start IE in the VM and https to the public
address of the machine, which is then redirected by netfilter back into
another of the VM's.
I can https directly to the other VM's address, but that does not cause
it to crash, however without netfilter loaded I can't bounce off the
public IP. It's all rather confusing really.
What next Sherlock?
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 111+ messages in thread
* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
2011-06-01 10:53 ` Brad Campbell
@ 2011-06-01 11:09 ` Avi Kivity
-1 siblings, 0 replies; 111+ messages in thread
From: Avi Kivity @ 2011-06-01 11:09 UTC (permalink / raw)
To: Brad Campbell
Cc: Hugh Dickins, Andrea Arcangeli, Borislav Petkov, linux-kernel,
kvm, linux-mm, netdev
On 06/01/2011 01:53 PM, Brad Campbell wrote:
> On 01/06/11 17:41, Avi Kivity wrote:
>> On 06/01/2011 12:40 PM, Avi Kivity wrote:
>>>
>>> bridge and netfilter, IIRC this was also the problem last time.
>>>
>>> Do you have any ebtables loaded?
>
> Never heard of them, but making a cursory check just in case..
>
> brad@srv:/raid10/src/linux-2.6.39$ grep EBTABLE .config
> # CONFIG_BRIDGE_NF_EBTABLES is not set
>
>>> Can you try building a kernel without ebtables? Without netfilter at
>>> all?
>
> Well, without netfilter I can't get it to crash. The problem is
> without netfilter I can't actually use it the way I use it to get it
> to crash.
>
> I rebooted into a netfilter kernel, and did all the steps I'd used on
> the no-netfilter kernel and it ticked along happily.
>
> So the result of the experiment is inconclusive. Having said that, the
> backtraces certainly smell networky.
>
> To get it to crash, I have to start IE in the VM and https to the
> public address of the machine, which is then redirected by netfilter
> back into another of the VM's.
>
> I can https directly to the other VM's address, but that does not
> cause it to crash, however without netfilter loaded I can't bounce off
> the public IP. It's all rather confusing really.
>
> What next Sherlock?
>
Maybe the Sherlocks at netdev@ can tell.
--
error compiling committee.c: too many arguments to function
^ permalink raw reply [flat|nested] 111+ messages in thread
* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
@ 2011-06-01 11:09 ` Avi Kivity
0 siblings, 0 replies; 111+ messages in thread
From: Avi Kivity @ 2011-06-01 11:09 UTC (permalink / raw)
To: Brad Campbell
Cc: Hugh Dickins, Andrea Arcangeli, Borislav Petkov, linux-kernel,
kvm, linux-mm, netdev
On 06/01/2011 01:53 PM, Brad Campbell wrote:
> On 01/06/11 17:41, Avi Kivity wrote:
>> On 06/01/2011 12:40 PM, Avi Kivity wrote:
>>>
>>> bridge and netfilter, IIRC this was also the problem last time.
>>>
>>> Do you have any ebtables loaded?
>
> Never heard of them, but making a cursory check just in case..
>
> brad@srv:/raid10/src/linux-2.6.39$ grep EBTABLE .config
> # CONFIG_BRIDGE_NF_EBTABLES is not set
>
>>> Can you try building a kernel without ebtables? Without netfilter at
>>> all?
>
> Well, without netfilter I can't get it to crash. The problem is
> without netfilter I can't actually use it the way I use it to get it
> to crash.
>
> I rebooted into a netfilter kernel, and did all the steps I'd used on
> the no-netfilter kernel and it ticked along happily.
>
> So the result of the experiment is inconclusive. Having said that, the
> backtraces certainly smell networky.
>
> To get it to crash, I have to start IE in the VM and https to the
> public address of the machine, which is then redirected by netfilter
> back into another of the VM's.
>
> I can https directly to the other VM's address, but that does not
> cause it to crash, however without netfilter loaded I can't bounce off
> the public IP. It's all rather confusing really.
>
> What next Sherlock?
>
Maybe the Sherlocks at netdev@ can tell.
--
error compiling committee.c: too many arguments to function
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 111+ messages in thread
* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
2011-06-01 10:53 ` Brad Campbell
@ 2011-06-01 11:18 ` CaT
-1 siblings, 0 replies; 111+ messages in thread
From: CaT @ 2011-06-01 11:18 UTC (permalink / raw)
To: Brad Campbell
Cc: Avi Kivity, Hugh Dickins, Andrea Arcangeli, Borislav Petkov,
linux-kernel, kvm, linux-mm, netdev
On Wed, Jun 01, 2011 at 06:53:31PM +0800, Brad Campbell wrote:
> I rebooted into a netfilter kernel, and did all the steps I'd used
> on the no-netfilter kernel and it ticked along happily.
>
> So the result of the experiment is inconclusive. Having said that,
> the backtraces certainly smell networky.
>
> To get it to crash, I have to start IE in the VM and https to the
> public address of the machine, which is then redirected by netfilter
> back into another of the VM's.
>
> I can https directly to the other VM's address, but that does not
> cause it to crash, however without netfilter loaded I can't bounce
> off the public IP. It's all rather confusing really.
>
> What next Sherlock?
I think you're hitting something I've seen. Can you try rewriting
your firewall rules so that it does not reference any bridge
interfaces at all. Instead, reference the real interface names
in their place. I'm betting it wont crash.
(netdev added to CC since we're aleady bouncing there)
--
"A search of his car uncovered pornography, a homemade sex aid, women's
stockings and a Jack Russell terrier."
- http://www.dailytelegraph.com.au/news/wacky/indeed/story-e6frev20-1111118083480
^ permalink raw reply [flat|nested] 111+ messages in thread
* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
@ 2011-06-01 11:18 ` CaT
0 siblings, 0 replies; 111+ messages in thread
From: CaT @ 2011-06-01 11:18 UTC (permalink / raw)
To: Brad Campbell
Cc: Avi Kivity, Hugh Dickins, Andrea Arcangeli, Borislav Petkov,
linux-kernel, kvm, linux-mm, netdev
On Wed, Jun 01, 2011 at 06:53:31PM +0800, Brad Campbell wrote:
> I rebooted into a netfilter kernel, and did all the steps I'd used
> on the no-netfilter kernel and it ticked along happily.
>
> So the result of the experiment is inconclusive. Having said that,
> the backtraces certainly smell networky.
>
> To get it to crash, I have to start IE in the VM and https to the
> public address of the machine, which is then redirected by netfilter
> back into another of the VM's.
>
> I can https directly to the other VM's address, but that does not
> cause it to crash, however without netfilter loaded I can't bounce
> off the public IP. It's all rather confusing really.
>
> What next Sherlock?
I think you're hitting something I've seen. Can you try rewriting
your firewall rules so that it does not reference any bridge
interfaces at all. Instead, reference the real interface names
in their place. I'm betting it wont crash.
(netdev added to CC since we're aleady bouncing there)
--
"A search of his car uncovered pornography, a homemade sex aid, women's
stockings and a Jack Russell terrier."
- http://www.dailytelegraph.com.au/news/wacky/indeed/story-e6frev20-1111118083480
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 111+ messages in thread
* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
2011-06-01 11:18 ` CaT
@ 2011-06-01 11:52 ` Brad Campbell
-1 siblings, 0 replies; 111+ messages in thread
From: Brad Campbell @ 2011-06-01 11:52 UTC (permalink / raw)
To: CaT
Cc: Avi Kivity, Hugh Dickins, Andrea Arcangeli, Borislav Petkov,
linux-kernel, kvm, linux-mm, netdev
On 01/06/11 19:18, CaT wrote:
> On Wed, Jun 01, 2011 at 06:53:31PM +0800, Brad Campbell wrote:
>> I rebooted into a netfilter kernel, and did all the steps I'd used
>> on the no-netfilter kernel and it ticked along happily.
>>
>> So the result of the experiment is inconclusive. Having said that,
>> the backtraces certainly smell networky.
>>
>> To get it to crash, I have to start IE in the VM and https to the
>> public address of the machine, which is then redirected by netfilter
>> back into another of the VM's.
>>
>> I can https directly to the other VM's address, but that does not
>> cause it to crash, however without netfilter loaded I can't bounce
>> off the public IP. It's all rather confusing really.
>>
>> What next Sherlock?
>
> I think you're hitting something I've seen. Can you try rewriting
> your firewall rules so that it does not reference any bridge
> interfaces at all. Instead, reference the real interface names
> in their place. I'm betting it wont crash.
>
Unfortunately the only interface that is mentioned by name anywhere in
my firewall is $DMZ (which is ppp0 and not part of any bridge).
All of the nat/dnat and other horrible hacks are based on IP addresses.
^ permalink raw reply [flat|nested] 111+ messages in thread
* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
@ 2011-06-01 11:52 ` Brad Campbell
0 siblings, 0 replies; 111+ messages in thread
From: Brad Campbell @ 2011-06-01 11:52 UTC (permalink / raw)
To: CaT
Cc: Avi Kivity, Hugh Dickins, Andrea Arcangeli, Borislav Petkov,
linux-kernel, kvm, linux-mm, netdev
On 01/06/11 19:18, CaT wrote:
> On Wed, Jun 01, 2011 at 06:53:31PM +0800, Brad Campbell wrote:
>> I rebooted into a netfilter kernel, and did all the steps I'd used
>> on the no-netfilter kernel and it ticked along happily.
>>
>> So the result of the experiment is inconclusive. Having said that,
>> the backtraces certainly smell networky.
>>
>> To get it to crash, I have to start IE in the VM and https to the
>> public address of the machine, which is then redirected by netfilter
>> back into another of the VM's.
>>
>> I can https directly to the other VM's address, but that does not
>> cause it to crash, however without netfilter loaded I can't bounce
>> off the public IP. It's all rather confusing really.
>>
>> What next Sherlock?
>
> I think you're hitting something I've seen. Can you try rewriting
> your firewall rules so that it does not reference any bridge
> interfaces at all. Instead, reference the real interface names
> in their place. I'm betting it wont crash.
>
Unfortunately the only interface that is mentioned by name anywhere in
my firewall is $DMZ (which is ppp0 and not part of any bridge).
All of the nat/dnat and other horrible hacks are based on IP addresses.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 111+ messages in thread
* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
2011-06-01 11:52 ` Brad Campbell
@ 2011-06-01 23:03 ` CaT
-1 siblings, 0 replies; 111+ messages in thread
From: CaT @ 2011-06-01 23:03 UTC (permalink / raw)
To: Brad Campbell
Cc: Avi Kivity, Hugh Dickins, Andrea Arcangeli, Borislav Petkov,
linux-kernel, kvm, linux-mm, netdev
On Wed, Jun 01, 2011 at 07:52:33PM +0800, Brad Campbell wrote:
> Unfortunately the only interface that is mentioned by name anywhere
> in my firewall is $DMZ (which is ppp0 and not part of any bridge).
>
> All of the nat/dnat and other horrible hacks are based on IP addresses.
Damn. Not referencing the bridge interfaces at all stopped our host from
going down in flames when we passed it a few packets. These are two
of the oopses we got from it. Whilst the kernel here is .35 we got the
same issue from a range of kernels. Seems related.
The oopses may be a bit weird. Copy and paste from an ipmi terminal.
slab error in cache_alloc_debugcheck_after(): cache `size-64': double
free, or n
Pid: 2431, comm: kvm Tainted: G D
2.6.35.9-local.20110314-141930 #1
Call Trace:
<IRQ> [<ffffffff810fb8bf>] ? __slab_error+0x1f/0x30
[<ffffffff810fc22b>] ? cache_alloc_debugcheck_after+0x6b/0x1f0
[<ffffffff81530a00>] ? br_nf_pre_routing_finish+0x0/0x370
[<ffffffff8153106b>] ? br_nf_pre_routing+0x2fb/0x980
[<ffffffff810fdd3d>] ? kmem_cache_alloc_notrace+0x7d/0xf0
[<ffffffff8153106b>] ? br_nf_pre_routing+0x2fb/0x980
[<ffffffff81466e66>] ? nf_iterate+0x66/0xb0
[<ffffffff8152b9f0>] ? br_handle_frame_finish+0x0/0x1c0
[<ffffffff81466f14>] ? nf_hook_slow+0x64/0xf0
[<ffffffff8152b9f0>] ? br_handle_frame_finish+0x0/0x1c0
[<ffffffff8152bd3c>] ? br_handle_frame+0x18c/0x250
[<ffffffff81445459>] ? __netif_receive_skb+0x169/0x2a0
[<ffffffff81445673>] ? process_backlog+0xe3/0x1d0
[<ffffffff81446347>] ? net_rx_action+0x87/0x1c0
[<ffffffff810793f7>] ? __do_softirq+0xa7/0x1d0
[<ffffffff81035b8c>] ? call_softirq+0x1c/0x30
<EOI> [<ffffffff81037c6d>] ? do_softirq+0x4d/0x80
[<ffffffff81446b4e>] ? netif_rx_ni+0x1e/0x30
[<ffffffff8139541a>] ? tun_chr_aio_write+0x36a/0x510
[<ffffffff813950b0>] ? tun_chr_aio_write+0x0/0x510
[<ffffffff81102859>] ? do_sync_readv_writev+0xa9/0xf0
[<ffffffff810973fb>] ? ktime_get+0x5b/0xe0
[<ffffffff8104f958>] ? lapic_next_event+0x18/0x20
[<ffffffff8109be18>] ? tick_dev_program_event+0x38/0x100
[<ffffffff81102697>] ? rw_copy_check_uvector+0x77/0x130
[<ffffffff81102f0c>] ? do_readv_writev+0xdc/0x200
[<ffffffff8108dfec>] ? sys_timer_settime+0x13c/0x2e0
[<ffffffff8110317e>] ? sys_writev+0x4e/0x90
[<ffffffff81034d6b>] ? system_call_fastpath+0x16/0x1b
ffff8801e7621500: redzone 1:0xbf05bd0100000006, redzone 2:0x9f911029d74e35b
----------
Code: 40 01 00 00 4c 8b a4 24 48 01 00 00 4c 8b ac 24 50 01 00 00 4c 8b
b4 24 5
RIP [<ffffffff81652c67>] icmp_send+0x297/0x650
RSP <ffff880001c036b8>
---[ end trace 9d3f7be7684ac91e ]---
Kernel panic - not syncing: Fatal exception in interrupt
Pid: 0, comm: swapper Tainted: G D
2.6.35.9-local.20110314-144920 #2
Call Trace:
<IRQ> [<ffffffff8170eada>] ? panic+0x94/0x116
[<ffffffff81711326>] ? _raw_spin_lock_irqsave+0x26/0x40
[<ffffffff8103a05f>] ? oops_end+0xef/0xf0
[<ffffffff81711a15>] ? general_protection+0x25/0x30
[<ffffffff81652c2f>] ? icmp_send+0x25f/0x650
[<ffffffff81652c67>] ? icmp_send+0x297/0x650
[<ffffffff815fd8e6>] ? nf_iterate+0x66/0xb0
[<ffffffff816dbfa0>] ? br_nf_forward_finish+0x0/0x170
[<ffffffff815fd994>] ? nf_hook_slow+0x64/0xf0
[<ffffffff816dbfa0>] ? br_nf_forward_finish+0x0/0x170
[<ffffffff816dc461>] ? br_nf_forward_ip+0x201/0x3e0
[<ffffffff815fd8e6>] ? nf_iterate+0x66/0xb0
[<ffffffff816d6620>] ? br_forward_finish+0x0/0x60
[<ffffffff815fd994>] ? nf_hook_slow+0x64/0xf0
[<ffffffff816d6620>] ? br_forward_finish+0x0/0x60
[<ffffffff816d66e9>] ? __br_forward+0x69/0xb0
[<ffffffff816d741a>] ? br_handle_frame_finish+0x12a/0x280
[<ffffffff816dcac8>] ? br_nf_pre_routing_finish+0x208/0x370
[<ffffffff815fd994>] ? nf_hook_slow+0x64/0xf0
[<ffffffff816dc8c0>] ? br_nf_pre_routing_finish+0x0/0x370
[<ffffffff816dc538>] ? br_nf_forward_ip+0x2d8/0x3e0
[<ffffffff816dd3b5>] ? br_nf_pre_routing+0x785/0x980
[<ffffffff815fd8e6>] ? nf_iterate+0x66/0xb0
[<ffffffff815fd994>] ? nf_hook_slow+0x64/0xf0
[<ffffffff816d72f0>] ? br_handle_frame_finish+0x0/0x280
[<ffffffff815fd994>] ? nf_hook_slow+0x64/0xf0
[<ffffffff816d72f0>] ? br_handle_frame_finish+0x0/0x280
[<ffffffff816d76fc>] ? br_handle_frame+0x18c/0x250
[<ffffffff815dec5b>] ? __netif_receive_skb+0x1cb/0x350
[<ffffffff8103d115>] ? read_tsc+0x5/0x20
[<ffffffff815dfa18>] ? netif_receive_skb+0x78/0x80
[<ffffffff815e0217>] ? napi_gro_receive+0x27/0x40
[<ffffffff815e01d8>] ? napi_skb_finish+0x38/0x50
[<ffffffff8152586d>] ? bnx2_poll_work+0xd0d/0x13d0
[<ffffffff8160c950>] ? ctnetlink_conntrack_event+0x210/0x7d0
[<ffffffff81092029>] ? autoremove_wake_function+0x9/0x30
[<ffffffff8109a71b>] ? ktime_get+0x5b/0xe0
[<ffffffff81526051>] ? bnx2_poll+0x61/0x230
[<ffffffff81051db8>] ? lapic_next_event+0x18/0x20
[<ffffffff815dfbef>] ? net_rx_action+0x9f/0x200
[<ffffffff8109636f>] ? __hrtimer_start_range_ns+0x22f/0x410
[<ffffffff8107c35f>] ? __do_softirq+0xaf/0x1e0
[<ffffffff810ab547>] ? handle_IRQ_event+0x47/0x160
[<ffffffff81036d5c>] ? call_softirq+0x1c/0x30
[<ffffffff81038c85>] ? do_softirq+0x65/0xa0
[<ffffffff8107c235>] ? irq_exit+0x85/0x90
[<ffffffff8103820b>] ? do_IRQ+0x6b/0xe0
[<ffffffff817117d3>] ? ret_from_intr+0x0/0x11
<EOI> [<ffffffff81269340>] ? intel_idle+0xf0/0x180
[<ffffffff81269320>] ? intel_idle+0xd0/0x180
[<ffffffff815b0b0f>] ? cpuidle_idle_call+0x9f/0x140
[<ffffffff81035032>] ? cpu_idle+0x62/0xb0
[<ffffffff81a40c77>] ? start_kernel+0x2ef/0x2fa
[<ffffffff81a403e3>] ? x86_64_start_kernel+0xfb/0x10a
--
"A search of his car uncovered pornography, a homemade sex aid, women's
stockings and a Jack Russell terrier."
- http://www.dailytelegraph.com.au/news/wacky/indeed/story-e6frev20-1111118083480
^ permalink raw reply [flat|nested] 111+ messages in thread
* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
@ 2011-06-01 23:03 ` CaT
0 siblings, 0 replies; 111+ messages in thread
From: CaT @ 2011-06-01 23:03 UTC (permalink / raw)
To: Brad Campbell
Cc: Avi Kivity, Hugh Dickins, Andrea Arcangeli, Borislav Petkov,
linux-kernel, kvm, linux-mm, netdev
On Wed, Jun 01, 2011 at 07:52:33PM +0800, Brad Campbell wrote:
> Unfortunately the only interface that is mentioned by name anywhere
> in my firewall is $DMZ (which is ppp0 and not part of any bridge).
>
> All of the nat/dnat and other horrible hacks are based on IP addresses.
Damn. Not referencing the bridge interfaces at all stopped our host from
going down in flames when we passed it a few packets. These are two
of the oopses we got from it. Whilst the kernel here is .35 we got the
same issue from a range of kernels. Seems related.
The oopses may be a bit weird. Copy and paste from an ipmi terminal.
slab error in cache_alloc_debugcheck_after(): cache `size-64': double
free, or n
Pid: 2431, comm: kvm Tainted: G D
2.6.35.9-local.20110314-141930 #1
Call Trace:
<IRQ> [<ffffffff810fb8bf>] ? __slab_error+0x1f/0x30
[<ffffffff810fc22b>] ? cache_alloc_debugcheck_after+0x6b/0x1f0
[<ffffffff81530a00>] ? br_nf_pre_routing_finish+0x0/0x370
[<ffffffff8153106b>] ? br_nf_pre_routing+0x2fb/0x980
[<ffffffff810fdd3d>] ? kmem_cache_alloc_notrace+0x7d/0xf0
[<ffffffff8153106b>] ? br_nf_pre_routing+0x2fb/0x980
[<ffffffff81466e66>] ? nf_iterate+0x66/0xb0
[<ffffffff8152b9f0>] ? br_handle_frame_finish+0x0/0x1c0
[<ffffffff81466f14>] ? nf_hook_slow+0x64/0xf0
[<ffffffff8152b9f0>] ? br_handle_frame_finish+0x0/0x1c0
[<ffffffff8152bd3c>] ? br_handle_frame+0x18c/0x250
[<ffffffff81445459>] ? __netif_receive_skb+0x169/0x2a0
[<ffffffff81445673>] ? process_backlog+0xe3/0x1d0
[<ffffffff81446347>] ? net_rx_action+0x87/0x1c0
[<ffffffff810793f7>] ? __do_softirq+0xa7/0x1d0
[<ffffffff81035b8c>] ? call_softirq+0x1c/0x30
<EOI> [<ffffffff81037c6d>] ? do_softirq+0x4d/0x80
[<ffffffff81446b4e>] ? netif_rx_ni+0x1e/0x30
[<ffffffff8139541a>] ? tun_chr_aio_write+0x36a/0x510
[<ffffffff813950b0>] ? tun_chr_aio_write+0x0/0x510
[<ffffffff81102859>] ? do_sync_readv_writev+0xa9/0xf0
[<ffffffff810973fb>] ? ktime_get+0x5b/0xe0
[<ffffffff8104f958>] ? lapic_next_event+0x18/0x20
[<ffffffff8109be18>] ? tick_dev_program_event+0x38/0x100
[<ffffffff81102697>] ? rw_copy_check_uvector+0x77/0x130
[<ffffffff81102f0c>] ? do_readv_writev+0xdc/0x200
[<ffffffff8108dfec>] ? sys_timer_settime+0x13c/0x2e0
[<ffffffff8110317e>] ? sys_writev+0x4e/0x90
[<ffffffff81034d6b>] ? system_call_fastpath+0x16/0x1b
ffff8801e7621500: redzone 1:0xbf05bd0100000006, redzone 2:0x9f911029d74e35b
----------
Code: 40 01 00 00 4c 8b a4 24 48 01 00 00 4c 8b ac 24 50 01 00 00 4c 8b
b4 24 5
RIP [<ffffffff81652c67>] icmp_send+0x297/0x650
RSP <ffff880001c036b8>
---[ end trace 9d3f7be7684ac91e ]---
Kernel panic - not syncing: Fatal exception in interrupt
Pid: 0, comm: swapper Tainted: G D
2.6.35.9-local.20110314-144920 #2
Call Trace:
<IRQ> [<ffffffff8170eada>] ? panic+0x94/0x116
[<ffffffff81711326>] ? _raw_spin_lock_irqsave+0x26/0x40
[<ffffffff8103a05f>] ? oops_end+0xef/0xf0
[<ffffffff81711a15>] ? general_protection+0x25/0x30
[<ffffffff81652c2f>] ? icmp_send+0x25f/0x650
[<ffffffff81652c67>] ? icmp_send+0x297/0x650
[<ffffffff815fd8e6>] ? nf_iterate+0x66/0xb0
[<ffffffff816dbfa0>] ? br_nf_forward_finish+0x0/0x170
[<ffffffff815fd994>] ? nf_hook_slow+0x64/0xf0
[<ffffffff816dbfa0>] ? br_nf_forward_finish+0x0/0x170
[<ffffffff816dc461>] ? br_nf_forward_ip+0x201/0x3e0
[<ffffffff815fd8e6>] ? nf_iterate+0x66/0xb0
[<ffffffff816d6620>] ? br_forward_finish+0x0/0x60
[<ffffffff815fd994>] ? nf_hook_slow+0x64/0xf0
[<ffffffff816d6620>] ? br_forward_finish+0x0/0x60
[<ffffffff816d66e9>] ? __br_forward+0x69/0xb0
[<ffffffff816d741a>] ? br_handle_frame_finish+0x12a/0x280
[<ffffffff816dcac8>] ? br_nf_pre_routing_finish+0x208/0x370
[<ffffffff815fd994>] ? nf_hook_slow+0x64/0xf0
[<ffffffff816dc8c0>] ? br_nf_pre_routing_finish+0x0/0x370
[<ffffffff816dc538>] ? br_nf_forward_ip+0x2d8/0x3e0
[<ffffffff816dd3b5>] ? br_nf_pre_routing+0x785/0x980
[<ffffffff815fd8e6>] ? nf_iterate+0x66/0xb0
[<ffffffff815fd994>] ? nf_hook_slow+0x64/0xf0
[<ffffffff816d72f0>] ? br_handle_frame_finish+0x0/0x280
[<ffffffff815fd994>] ? nf_hook_slow+0x64/0xf0
[<ffffffff816d72f0>] ? br_handle_frame_finish+0x0/0x280
[<ffffffff816d76fc>] ? br_handle_frame+0x18c/0x250
[<ffffffff815dec5b>] ? __netif_receive_skb+0x1cb/0x350
[<ffffffff8103d115>] ? read_tsc+0x5/0x20
[<ffffffff815dfa18>] ? netif_receive_skb+0x78/0x80
[<ffffffff815e0217>] ? napi_gro_receive+0x27/0x40
[<ffffffff815e01d8>] ? napi_skb_finish+0x38/0x50
[<ffffffff8152586d>] ? bnx2_poll_work+0xd0d/0x13d0
[<ffffffff8160c950>] ? ctnetlink_conntrack_event+0x210/0x7d0
[<ffffffff81092029>] ? autoremove_wake_function+0x9/0x30
[<ffffffff8109a71b>] ? ktime_get+0x5b/0xe0
[<ffffffff81526051>] ? bnx2_poll+0x61/0x230
[<ffffffff81051db8>] ? lapic_next_event+0x18/0x20
[<ffffffff815dfbef>] ? net_rx_action+0x9f/0x200
[<ffffffff8109636f>] ? __hrtimer_start_range_ns+0x22f/0x410
[<ffffffff8107c35f>] ? __do_softirq+0xaf/0x1e0
[<ffffffff810ab547>] ? handle_IRQ_event+0x47/0x160
[<ffffffff81036d5c>] ? call_softirq+0x1c/0x30
[<ffffffff81038c85>] ? do_softirq+0x65/0xa0
[<ffffffff8107c235>] ? irq_exit+0x85/0x90
[<ffffffff8103820b>] ? do_IRQ+0x6b/0xe0
[<ffffffff817117d3>] ? ret_from_intr+0x0/0x11
<EOI> [<ffffffff81269340>] ? intel_idle+0xf0/0x180
[<ffffffff81269320>] ? intel_idle+0xd0/0x180
[<ffffffff815b0b0f>] ? cpuidle_idle_call+0x9f/0x140
[<ffffffff81035032>] ? cpu_idle+0x62/0xb0
[<ffffffff81a40c77>] ? start_kernel+0x2ef/0x2fa
[<ffffffff81a403e3>] ? x86_64_start_kernel+0xfb/0x10a
--
"A search of his car uncovered pornography, a homemade sex aid, women's
stockings and a Jack Russell terrier."
- http://www.dailytelegraph.com.au/news/wacky/indeed/story-e6frev20-1111118083480
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 111+ messages in thread
* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
2011-06-01 23:03 ` CaT
@ 2011-06-03 13:38 ` Brad Campbell
-1 siblings, 0 replies; 111+ messages in thread
From: Brad Campbell @ 2011-06-03 13:38 UTC (permalink / raw)
To: CaT
Cc: Avi Kivity, Hugh Dickins, Andrea Arcangeli, Borislav Petkov,
linux-kernel, kvm, linux-mm, netdev
On 02/06/11 07:03, CaT wrote:
> On Wed, Jun 01, 2011 at 07:52:33PM +0800, Brad Campbell wrote:
>> Unfortunately the only interface that is mentioned by name anywhere
>> in my firewall is $DMZ (which is ppp0 and not part of any bridge).
>>
>> All of the nat/dnat and other horrible hacks are based on IP addresses.
>
> Damn. Not referencing the bridge interfaces at all stopped our host from
> going down in flames when we passed it a few packets. These are two
> of the oopses we got from it. Whilst the kernel here is .35 we got the
> same issue from a range of kernels. Seems related.
Well, I tried sending an explanatory message to netdev, netfilter & cc'd
to kvm, but it appears not to have made it to kvm or netfilter, and the
cc to netdev has not elicited a response. My resend to netfilter seems
to have dropped into the bit bucket also.
Is there anyone who can point me at the appropriate cage to rattle? I
know it appears to be a netfilter issue, but I don't seem to be able to
get a message to the list (and I am subscribed to it and have been
getting mail for months) and I'm not sure who to pester. The other
alternative is I just stop doing "that" and wait for it to bite someone
else.
Cheers.
Brad
^ permalink raw reply [flat|nested] 111+ messages in thread
* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
@ 2011-06-03 13:38 ` Brad Campbell
0 siblings, 0 replies; 111+ messages in thread
From: Brad Campbell @ 2011-06-03 13:38 UTC (permalink / raw)
To: CaT
Cc: Avi Kivity, Hugh Dickins, Andrea Arcangeli, Borislav Petkov,
linux-kernel, kvm, linux-mm, netdev
On 02/06/11 07:03, CaT wrote:
> On Wed, Jun 01, 2011 at 07:52:33PM +0800, Brad Campbell wrote:
>> Unfortunately the only interface that is mentioned by name anywhere
>> in my firewall is $DMZ (which is ppp0 and not part of any bridge).
>>
>> All of the nat/dnat and other horrible hacks are based on IP addresses.
>
> Damn. Not referencing the bridge interfaces at all stopped our host from
> going down in flames when we passed it a few packets. These are two
> of the oopses we got from it. Whilst the kernel here is .35 we got the
> same issue from a range of kernels. Seems related.
Well, I tried sending an explanatory message to netdev, netfilter & cc'd
to kvm, but it appears not to have made it to kvm or netfilter, and the
cc to netdev has not elicited a response. My resend to netfilter seems
to have dropped into the bit bucket also.
Is there anyone who can point me at the appropriate cage to rattle? I
know it appears to be a netfilter issue, but I don't seem to be able to
get a message to the list (and I am subscribed to it and have been
getting mail for months) and I'm not sure who to pester. The other
alternative is I just stop doing "that" and wait for it to bite someone
else.
Cheers.
Brad
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 111+ messages in thread
* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
2011-06-03 13:38 ` Brad Campbell
(?)
@ 2011-06-03 15:50 ` Bernhard Held
-1 siblings, 0 replies; 111+ messages in thread
From: Bernhard Held @ 2011-06-03 15:50 UTC (permalink / raw)
To: linux-kernel; +Cc: linux-mm, kvm, netdev
Am 03.06.2011 15:38, schrieb Brad Campbell:
> On 02/06/11 07:03, CaT wrote:
>> On Wed, Jun 01, 2011 at 07:52:33PM +0800, Brad Campbell wrote:
>>> Unfortunately the only interface that is mentioned by name anywhere
>>> in my firewall is $DMZ (which is ppp0 and not part of any bridge).
>>>
>>> All of the nat/dnat and other horrible hacks are based on IP addresses.
>>
>> Damn. Not referencing the bridge interfaces at all stopped our host from
>> going down in flames when we passed it a few packets. These are two
>> of the oopses we got from it. Whilst the kernel here is .35 we got the
>> same issue from a range of kernels. Seems related.
>
> Well, I tried sending an explanatory message to netdev, netfilter & cc'd to kvm,
> but it appears not to have made it to kvm or netfilter, and the cc to netdev has
> not elicited a response. My resend to netfilter seems to have dropped into the
> bit bucket also.
Just another reference 3.5 months ago:
http://www.spinics.net/lists/netfilter-devel/msg17239.html
Bernhard
^ permalink raw reply [flat|nested] 111+ messages in thread
* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
@ 2011-06-03 15:50 ` Bernhard Held
0 siblings, 0 replies; 111+ messages in thread
From: Bernhard Held @ 2011-06-03 15:50 UTC (permalink / raw)
To: kvm; +Cc: linux-mm, linux-kernel, netdev, linux-mm, kvm, netdev
Am 03.06.2011 15:38, schrieb Brad Campbell:
> On 02/06/11 07:03, CaT wrote:
>> On Wed, Jun 01, 2011 at 07:52:33PM +0800, Brad Campbell wrote:
>>> Unfortunately the only interface that is mentioned by name anywhere
>>> in my firewall is $DMZ (which is ppp0 and not part of any bridge).
>>>
>>> All of the nat/dnat and other horrible hacks are based on IP addresses.
>>
>> Damn. Not referencing the bridge interfaces at all stopped our host from
>> going down in flames when we passed it a few packets. These are two
>> of the oopses we got from it. Whilst the kernel here is .35 we got the
>> same issue from a range of kernels. Seems related.
>
> Well, I tried sending an explanatory message to netdev, netfilter & cc'd to kvm,
> but it appears not to have made it to kvm or netfilter, and the cc to netdev has
> not elicited a response. My resend to netfilter seems to have dropped into the
> bit bucket also.
Just another reference 3.5 months ago:
http://www.spinics.net/lists/netfilter-devel/msg17239.html
Bernhard
^ permalink raw reply [flat|nested] 111+ messages in thread
* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
@ 2011-06-03 15:50 ` Bernhard Held
0 siblings, 0 replies; 111+ messages in thread
From: Bernhard Held @ 2011-06-03 15:50 UTC (permalink / raw)
To: linux-mm; +Cc: linux-kernel, kvm, netdev
Am 03.06.2011 15:38, schrieb Brad Campbell:
> On 02/06/11 07:03, CaT wrote:
>> On Wed, Jun 01, 2011 at 07:52:33PM +0800, Brad Campbell wrote:
>>> Unfortunately the only interface that is mentioned by name anywhere
>>> in my firewall is $DMZ (which is ppp0 and not part of any bridge).
>>>
>>> All of the nat/dnat and other horrible hacks are based on IP addresses.
>>
>> Damn. Not referencing the bridge interfaces at all stopped our host from
>> going down in flames when we passed it a few packets. These are two
>> of the oopses we got from it. Whilst the kernel here is .35 we got the
>> same issue from a range of kernels. Seems related.
>
> Well, I tried sending an explanatory message to netdev, netfilter & cc'd to kvm,
> but it appears not to have made it to kvm or netfilter, and the cc to netdev has
> not elicited a response. My resend to netfilter seems to have dropped into the
> bit bucket also.
Just another reference 3.5 months ago:
http://www.spinics.net/lists/netfilter-devel/msg17239.html
Bernhard
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 111+ messages in thread
* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
2011-06-03 15:50 ` Bernhard Held
@ 2011-06-03 16:07 ` Brad Campbell
-1 siblings, 0 replies; 111+ messages in thread
From: Brad Campbell @ 2011-06-03 16:07 UTC (permalink / raw)
To: kvm; +Cc: linux-mm, linux-kernel, netdev, netfilter-devel
On 03/06/11 23:50, Bernhard Held wrote:
> Am 03.06.2011 15:38, schrieb Brad Campbell:
>> On 02/06/11 07:03, CaT wrote:
>>> On Wed, Jun 01, 2011 at 07:52:33PM +0800, Brad Campbell wrote:
>>>> Unfortunately the only interface that is mentioned by name anywhere
>>>> in my firewall is $DMZ (which is ppp0 and not part of any bridge).
>>>>
>>>> All of the nat/dnat and other horrible hacks are based on IP addresses.
>>>
>>> Damn. Not referencing the bridge interfaces at all stopped our host from
>>> going down in flames when we passed it a few packets. These are two
>>> of the oopses we got from it. Whilst the kernel here is .35 we got the
>>> same issue from a range of kernels. Seems related.
>>
>> Well, I tried sending an explanatory message to netdev, netfilter &
>> cc'd to kvm,
>> but it appears not to have made it to kvm or netfilter, and the cc to
>> netdev has
>> not elicited a response. My resend to netfilter seems to have dropped
>> into the
>> bit bucket also.
> Just another reference 3.5 months ago:
> http://www.spinics.net/lists/netfilter-devel/msg17239.html
<waves hands around shouting "I have a reproducible test case for this
and don't mind patching and crashing the machine to get it fixed">
Attempted to add netfilter-devel to the cc this time.
^ permalink raw reply [flat|nested] 111+ messages in thread
* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
@ 2011-06-03 16:07 ` Brad Campbell
0 siblings, 0 replies; 111+ messages in thread
From: Brad Campbell @ 2011-06-03 16:07 UTC (permalink / raw)
To: kvm; +Cc: linux-mm, linux-kernel, netdev, netfilter-devel
On 03/06/11 23:50, Bernhard Held wrote:
>Am 03.06.2011 15:38, schrieb Brad Campbell:
>>On 02/06/11 07:03, CaT wrote:
>>>On Wed, Jun 01, 2011 at 07:52:33PM +0800, Brad Campbell wrote:
>>>>Unfortunately the only interface that is mentioned by name anywhere
>>>>in my firewall is $DMZ (which is ppp0 and not part of any bridge).
>>>>
>>>>All of the nat/dnat and other horrible hacks are based on IP addresses.
>>>
>>>Damn. Not referencing the bridge interfaces at all stopped our host from
>>>going down in flames when we passed it a few packets. These are two
>>>of the oopses we got from it. Whilst the kernel here is .35 we got the
>>>same issue from a range of kernels. Seems related.
>>
>>Well, I tried sending an explanatory message to netdev, netfilter &
>>cc'd to kvm,
>>but it appears not to have made it to kvm or netfilter, and the cc to
>>netdev has
>>not elicited a response. My resend to netfilter seems to have dropped
>>into the
>>bit bucket also.
>Just another reference 3.5 months ago:
>http://www.spinics.net/lists/netfilter-devel/msg17239.html
<waves hands around shouting "I have a reproducible test case for this
and don't mind patching and crashing the machine to get it fixed">
Attempted to add netfilter-devel to the cc this time.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 111+ messages in thread
* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
2011-06-03 13:38 ` Brad Campbell
@ 2011-06-05 8:14 ` Avi Kivity
-1 siblings, 0 replies; 111+ messages in thread
From: Avi Kivity @ 2011-06-05 8:14 UTC (permalink / raw)
To: Brad Campbell
Cc: CaT, Hugh Dickins, Andrea Arcangeli, Borislav Petkov,
linux-kernel, kvm, linux-mm, netdev
On 06/03/2011 04:38 PM, Brad Campbell wrote:
>
> Is there anyone who can point me at the appropriate cage to rattle? I
> know it appears to be a netfilter issue, but I don't seem to be able
> to get a message to the list (and I am subscribed to it and have been
> getting mail for months) and I'm not sure who to pester. The other
> alternative is I just stop doing "that" and wait for it to bite
> someone else.
The mailing list might be set not to send your own mails back to you.
Check the list archive.
--
error compiling committee.c: too many arguments to function
^ permalink raw reply [flat|nested] 111+ messages in thread
* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
@ 2011-06-05 8:14 ` Avi Kivity
0 siblings, 0 replies; 111+ messages in thread
From: Avi Kivity @ 2011-06-05 8:14 UTC (permalink / raw)
To: Brad Campbell
Cc: CaT, Hugh Dickins, Andrea Arcangeli, Borislav Petkov,
linux-kernel, kvm, linux-mm, netdev
On 06/03/2011 04:38 PM, Brad Campbell wrote:
>
> Is there anyone who can point me at the appropriate cage to rattle? I
> know it appears to be a netfilter issue, but I don't seem to be able
> to get a message to the list (and I am subscribed to it and have been
> getting mail for months) and I'm not sure who to pester. The other
> alternative is I just stop doing "that" and wait for it to bite
> someone else.
The mailing list might be set not to send your own mails back to you.
Check the list archive.
--
error compiling committee.c: too many arguments to function
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 111+ messages in thread
* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
2011-06-05 8:14 ` Avi Kivity
@ 2011-06-05 13:45 ` Brad Campbell
-1 siblings, 0 replies; 111+ messages in thread
From: Brad Campbell @ 2011-06-05 13:45 UTC (permalink / raw)
To: Avi Kivity
Cc: CaT, Hugh Dickins, Andrea Arcangeli, Borislav Petkov,
linux-kernel, kvm, linux-mm, netdev
On 05/06/11 16:14, Avi Kivity wrote:
> On 06/03/2011 04:38 PM, Brad Campbell wrote:
>>
>> Is there anyone who can point me at the appropriate cage to rattle? I
>> know it appears to be a netfilter issue, but I don't seem to be able
>> to get a message to the list (and I am subscribed to it and have been
>> getting mail for months) and I'm not sure who to pester. The other
>> alternative is I just stop doing "that" and wait for it to bite
>> someone else.
>
> The mailing list might be set not to send your own mails back to you.
> Check the list archive.
Yep, I did that first..
Given the response to previous issues along the same line, it looks a
bit like I just remember not to actually use the system in the way that
triggers the bug and be happy that 99% of the time the kernel does not
panic, but have that lovely feeling in the back of the skull that says
"any time now, and without obvious reason the whole machine might just
come crashing down"..
I guess it's still better than running Xen or Windows..
^ permalink raw reply [flat|nested] 111+ messages in thread
* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
@ 2011-06-05 13:45 ` Brad Campbell
0 siblings, 0 replies; 111+ messages in thread
From: Brad Campbell @ 2011-06-05 13:45 UTC (permalink / raw)
To: Avi Kivity
Cc: CaT, Hugh Dickins, Andrea Arcangeli, Borislav Petkov,
linux-kernel, kvm, linux-mm, netdev
On 05/06/11 16:14, Avi Kivity wrote:
> On 06/03/2011 04:38 PM, Brad Campbell wrote:
>>
>> Is there anyone who can point me at the appropriate cage to rattle? I
>> know it appears to be a netfilter issue, but I don't seem to be able
>> to get a message to the list (and I am subscribed to it and have been
>> getting mail for months) and I'm not sure who to pester. The other
>> alternative is I just stop doing "that" and wait for it to bite
>> someone else.
>
> The mailing list might be set not to send your own mails back to you.
> Check the list archive.
Yep, I did that first..
Given the response to previous issues along the same line, it looks a
bit like I just remember not to actually use the system in the way that
triggers the bug and be happy that 99% of the time the kernel does not
panic, but have that lovely feeling in the back of the skull that says
"any time now, and without obvious reason the whole machine might just
come crashing down"..
I guess it's still better than running Xen or Windows..
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 111+ messages in thread
* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
2011-06-05 13:45 ` Brad Campbell
@ 2011-06-05 13:58 ` Avi Kivity
-1 siblings, 0 replies; 111+ messages in thread
From: Avi Kivity @ 2011-06-05 13:58 UTC (permalink / raw)
To: Brad Campbell
Cc: CaT, Hugh Dickins, Andrea Arcangeli, Borislav Petkov,
linux-kernel, kvm, linux-mm, netdev, netfilter-devel
On 06/05/2011 04:45 PM, Brad Campbell wrote:
>> The mailing list might be set not to send your own mails back to you.
>> Check the list archive.
>
>
> Yep, I did that first..
>
> Given the response to previous issues along the same line, it looks a
> bit like I just remember not to actually use the system in the way
> that triggers the bug and be happy that 99% of the time the kernel
> does not panic, but have that lovely feeling in the back of the skull
> that says "any time now, and without obvious reason the whole machine
> might just come crashing down"..
>
> I guess it's still better than running Xen or Windows..
Not at all. Can some networking/netfilter expert look at this?
Please file a bug with all the relevant information in this thread. If
you can look for a previous version that worked, that might increase the
chances of the bug being resolved faster.
--
error compiling committee.c: too many arguments to function
^ permalink raw reply [flat|nested] 111+ messages in thread
* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
@ 2011-06-05 13:58 ` Avi Kivity
0 siblings, 0 replies; 111+ messages in thread
From: Avi Kivity @ 2011-06-05 13:58 UTC (permalink / raw)
To: Brad Campbell
Cc: CaT, Hugh Dickins, Andrea Arcangeli, Borislav Petkov,
linux-kernel, kvm, linux-mm, netdev, netfilter-devel
On 06/05/2011 04:45 PM, Brad Campbell wrote:
>> The mailing list might be set not to send your own mails back to you.
>> Check the list archive.
>
>
> Yep, I did that first..
>
> Given the response to previous issues along the same line, it looks a
> bit like I just remember not to actually use the system in the way
> that triggers the bug and be happy that 99% of the time the kernel
> does not panic, but have that lovely feeling in the back of the skull
> that says "any time now, and without obvious reason the whole machine
> might just come crashing down"..
>
> I guess it's still better than running Xen or Windows..
Not at all. Can some networking/netfilter expert look at this?
Please file a bug with all the relevant information in this thread. If
you can look for a previous version that worked, that might increase the
chances of the bug being resolved faster.
--
error compiling committee.c: too many arguments to function
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 111+ messages in thread
* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
2011-06-03 16:07 ` Brad Campbell
@ 2011-06-06 20:10 ` Bart De Schuymer
-1 siblings, 0 replies; 111+ messages in thread
From: Bart De Schuymer @ 2011-06-06 20:10 UTC (permalink / raw)
To: Brad Campbell; +Cc: kvm, linux-mm, linux-kernel, netdev, netfilter-devel
Hi Brad,
This has probably nothing to do with ebtables, so please rmmod in case
it's loaded.
A few questions I didn't directly see an answer to in the threads I
scanned...
I'm assuming you actually use the bridging firewall functionality. So,
what iptables modules do you use? Can you reduce your iptables rules to
a core that triggers the bug?
Or does it get triggered even with an empty set of firewall rules?
Are you using a stock .35 kernel or is it patched?
Is this something I can trigger on a poor guy's laptop or does it
require specialized hardware (I'm catching up on qemu/kvm...)?
cheers,
Bart
PS: I'm not sure if we should keep CC-ing everybody, netfilter-devel
together with kvm should probably do fine.
Op 3/06/2011 18:07, Brad Campbell schreef:
> On 03/06/11 23:50, Bernhard Held wrote:
>> Am 03.06.2011 15:38, schrieb Brad Campbell:
>>> On 02/06/11 07:03, CaT wrote:
>>>> On Wed, Jun 01, 2011 at 07:52:33PM +0800, Brad Campbell wrote:
>>>>> Unfortunately the only interface that is mentioned by name anywhere
>>>>> in my firewall is $DMZ (which is ppp0 and not part of any bridge).
>>>>>
>>>>> All of the nat/dnat and other horrible hacks are based on IP
>>>>> addresses.
>>>>
>>>> Damn. Not referencing the bridge interfaces at all stopped our host
>>>> from
>>>> going down in flames when we passed it a few packets. These are two
>>>> of the oopses we got from it. Whilst the kernel here is .35 we got the
>>>> same issue from a range of kernels. Seems related.
>>>
>>> Well, I tried sending an explanatory message to netdev, netfilter &
>>> cc'd to kvm,
>>> but it appears not to have made it to kvm or netfilter, and the cc to
>>> netdev has
>>> not elicited a response. My resend to netfilter seems to have dropped
>>> into the
>>> bit bucket also.
>> Just another reference 3.5 months ago:
>> http://www.spinics.net/lists/netfilter-devel/msg17239.html
>
> <waves hands around shouting "I have a reproducible test case for this
> and don't mind patching and crashing the machine to get it fixed">
>
> Attempted to add netfilter-devel to the cc this time.
> --
> To unsubscribe from this list: send the line "unsubscribe
> netfilter-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
--
Bart De Schuymer
www.artinalgorithms.be
^ permalink raw reply [flat|nested] 111+ messages in thread
* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
@ 2011-06-06 20:10 ` Bart De Schuymer
0 siblings, 0 replies; 111+ messages in thread
From: Bart De Schuymer @ 2011-06-06 20:10 UTC (permalink / raw)
To: Brad Campbell; +Cc: kvm, linux-mm, linux-kernel, netdev, netfilter-devel
Hi Brad,
This has probably nothing to do with ebtables, so please rmmod in case
it's loaded.
A few questions I didn't directly see an answer to in the threads I
scanned...
I'm assuming you actually use the bridging firewall functionality. So,
what iptables modules do you use? Can you reduce your iptables rules to
a core that triggers the bug?
Or does it get triggered even with an empty set of firewall rules?
Are you using a stock .35 kernel or is it patched?
Is this something I can trigger on a poor guy's laptop or does it
require specialized hardware (I'm catching up on qemu/kvm...)?
cheers,
Bart
PS: I'm not sure if we should keep CC-ing everybody, netfilter-devel
together with kvm should probably do fine.
Op 3/06/2011 18:07, Brad Campbell schreef:
> On 03/06/11 23:50, Bernhard Held wrote:
>> Am 03.06.2011 15:38, schrieb Brad Campbell:
>>> On 02/06/11 07:03, CaT wrote:
>>>> On Wed, Jun 01, 2011 at 07:52:33PM +0800, Brad Campbell wrote:
>>>>> Unfortunately the only interface that is mentioned by name anywhere
>>>>> in my firewall is $DMZ (which is ppp0 and not part of any bridge).
>>>>>
>>>>> All of the nat/dnat and other horrible hacks are based on IP
>>>>> addresses.
>>>>
>>>> Damn. Not referencing the bridge interfaces at all stopped our host
>>>> from
>>>> going down in flames when we passed it a few packets. These are two
>>>> of the oopses we got from it. Whilst the kernel here is .35 we got the
>>>> same issue from a range of kernels. Seems related.
>>>
>>> Well, I tried sending an explanatory message to netdev, netfilter &
>>> cc'd to kvm,
>>> but it appears not to have made it to kvm or netfilter, and the cc to
>>> netdev has
>>> not elicited a response. My resend to netfilter seems to have dropped
>>> into the
>>> bit bucket also.
>> Just another reference 3.5 months ago:
>> http://www.spinics.net/lists/netfilter-devel/msg17239.html
>
> <waves hands around shouting "I have a reproducible test case for this
> and don't mind patching and crashing the machine to get it fixed">
>
> Attempted to add netfilter-devel to the cc this time.
> --
> To unsubscribe from this list: send the line "unsubscribe
> netfilter-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
--
Bart De Schuymer
www.artinalgorithms.be
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 111+ messages in thread
* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
2011-06-05 13:45 ` Brad Campbell
(?)
@ 2011-06-06 20:22 ` Eric Dumazet
-1 siblings, 0 replies; 111+ messages in thread
From: Eric Dumazet @ 2011-06-06 20:22 UTC (permalink / raw)
To: Brad Campbell
Cc: Avi Kivity, CaT, Hugh Dickins, Andrea Arcangeli, Borislav Petkov,
linux-kernel, kvm, linux-mm, netdev
Le dimanche 05 juin 2011 à 21:45 +0800, Brad Campbell a écrit :
> On 05/06/11 16:14, Avi Kivity wrote:
> > On 06/03/2011 04:38 PM, Brad Campbell wrote:
> >>
> >> Is there anyone who can point me at the appropriate cage to rattle? I
> >> know it appears to be a netfilter issue, but I don't seem to be able
> >> to get a message to the list (and I am subscribed to it and have been
> >> getting mail for months) and I'm not sure who to pester. The other
> >> alternative is I just stop doing "that" and wait for it to bite
> >> someone else.
> >
> > The mailing list might be set not to send your own mails back to you.
> > Check the list archive.
>
> Yep, I did that first..
>
> Given the response to previous issues along the same line, it looks a
> bit like I just remember not to actually use the system in the way that
> triggers the bug and be happy that 99% of the time the kernel does not
> panic, but have that lovely feeling in the back of the skull that says
> "any time now, and without obvious reason the whole machine might just
> come crashing down"..
>
> I guess it's still better than running Xen or Windows..
Could you please try latest linux-2.6 tree ?
We fixed many networking bugs that could explain your crash.
^ permalink raw reply [flat|nested] 111+ messages in thread
* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
@ 2011-06-06 20:22 ` Eric Dumazet
0 siblings, 0 replies; 111+ messages in thread
From: Eric Dumazet @ 2011-06-06 20:22 UTC (permalink / raw)
To: Brad Campbell
Cc: Avi Kivity, CaT, Hugh Dickins, Andrea Arcangeli, Borislav Petkov,
linux-kernel, kvm, linux-mm, netdev
Le dimanche 05 juin 2011 à 21:45 +0800, Brad Campbell a écrit :
> On 05/06/11 16:14, Avi Kivity wrote:
> > On 06/03/2011 04:38 PM, Brad Campbell wrote:
> >>
> >> Is there anyone who can point me at the appropriate cage to rattle? I
> >> know it appears to be a netfilter issue, but I don't seem to be able
> >> to get a message to the list (and I am subscribed to it and have been
> >> getting mail for months) and I'm not sure who to pester. The other
> >> alternative is I just stop doing "that" and wait for it to bite
> >> someone else.
> >
> > The mailing list might be set not to send your own mails back to you.
> > Check the list archive.
>
> Yep, I did that first..
>
> Given the response to previous issues along the same line, it looks a
> bit like I just remember not to actually use the system in the way that
> triggers the bug and be happy that 99% of the time the kernel does not
> panic, but have that lovely feeling in the back of the skull that says
> "any time now, and without obvious reason the whole machine might just
> come crashing down"..
>
> I guess it's still better than running Xen or Windows..
Could you please try latest linux-2.6 tree ?
We fixed many networking bugs that could explain your crash.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 111+ messages in thread
* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
@ 2011-06-06 20:22 ` Eric Dumazet
0 siblings, 0 replies; 111+ messages in thread
From: Eric Dumazet @ 2011-06-06 20:22 UTC (permalink / raw)
To: Brad Campbell
Cc: Avi Kivity, CaT, Hugh Dickins, Andrea Arcangeli, Borislav Petkov,
linux-kernel, kvm, linux-mm, netdev
Le dimanche 05 juin 2011 A 21:45 +0800, Brad Campbell a A(C)crit :
> On 05/06/11 16:14, Avi Kivity wrote:
> > On 06/03/2011 04:38 PM, Brad Campbell wrote:
> >>
> >> Is there anyone who can point me at the appropriate cage to rattle? I
> >> know it appears to be a netfilter issue, but I don't seem to be able
> >> to get a message to the list (and I am subscribed to it and have been
> >> getting mail for months) and I'm not sure who to pester. The other
> >> alternative is I just stop doing "that" and wait for it to bite
> >> someone else.
> >
> > The mailing list might be set not to send your own mails back to you.
> > Check the list archive.
>
> Yep, I did that first..
>
> Given the response to previous issues along the same line, it looks a
> bit like I just remember not to actually use the system in the way that
> triggers the bug and be happy that 99% of the time the kernel does not
> panic, but have that lovely feeling in the back of the skull that says
> "any time now, and without obvious reason the whole machine might just
> come crashing down"..
>
> I guess it's still better than running Xen or Windows..
Could you please try latest linux-2.6 tree ?
We fixed many networking bugs that could explain your crash.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 111+ messages in thread
* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
2011-06-06 20:10 ` Bart De Schuymer
(?)
@ 2011-06-06 20:23 ` Eric Dumazet
-1 siblings, 0 replies; 111+ messages in thread
From: Eric Dumazet @ 2011-06-06 20:23 UTC (permalink / raw)
To: Bart De Schuymer
Cc: Brad Campbell, kvm, linux-mm, linux-kernel, netdev, netfilter-devel
Le lundi 06 juin 2011 à 22:10 +0200, Bart De Schuymer a écrit :
> Hi Brad,
>
> This has probably nothing to do with ebtables, so please rmmod in case
> it's loaded.
> A few questions I didn't directly see an answer to in the threads I
> scanned...
> I'm assuming you actually use the bridging firewall functionality. So,
> what iptables modules do you use? Can you reduce your iptables rules to
> a core that triggers the bug?
> Or does it get triggered even with an empty set of firewall rules?
> Are you using a stock .35 kernel or is it patched?
> Is this something I can trigger on a poor guy's laptop or does it
> require specialized hardware (I'm catching up on qemu/kvm...)?
>
Keep netdev, as this most probably is a networking bug.
^ permalink raw reply [flat|nested] 111+ messages in thread
* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
@ 2011-06-06 20:23 ` Eric Dumazet
0 siblings, 0 replies; 111+ messages in thread
From: Eric Dumazet @ 2011-06-06 20:23 UTC (permalink / raw)
To: Bart De Schuymer
Cc: Brad Campbell, kvm, linux-mm, linux-kernel, netdev, netfilter-devel
Le lundi 06 juin 2011 à 22:10 +0200, Bart De Schuymer a écrit :
> Hi Brad,
>
> This has probably nothing to do with ebtables, so please rmmod in case
> it's loaded.
> A few questions I didn't directly see an answer to in the threads I
> scanned...
> I'm assuming you actually use the bridging firewall functionality. So,
> what iptables modules do you use? Can you reduce your iptables rules to
> a core that triggers the bug?
> Or does it get triggered even with an empty set of firewall rules?
> Are you using a stock .35 kernel or is it patched?
> Is this something I can trigger on a poor guy's laptop or does it
> require specialized hardware (I'm catching up on qemu/kvm...)?
>
Keep netdev, as this most probably is a networking bug.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 111+ messages in thread
* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
@ 2011-06-06 20:23 ` Eric Dumazet
0 siblings, 0 replies; 111+ messages in thread
From: Eric Dumazet @ 2011-06-06 20:23 UTC (permalink / raw)
To: Bart De Schuymer
Cc: Brad Campbell, kvm, linux-mm, linux-kernel, netdev, netfilter-devel
Le lundi 06 juin 2011 A 22:10 +0200, Bart De Schuymer a A(C)crit :
> Hi Brad,
>
> This has probably nothing to do with ebtables, so please rmmod in case
> it's loaded.
> A few questions I didn't directly see an answer to in the threads I
> scanned...
> I'm assuming you actually use the bridging firewall functionality. So,
> what iptables modules do you use? Can you reduce your iptables rules to
> a core that triggers the bug?
> Or does it get triggered even with an empty set of firewall rules?
> Are you using a stock .35 kernel or is it patched?
> Is this something I can trigger on a poor guy's laptop or does it
> require specialized hardware (I'm catching up on qemu/kvm...)?
>
Keep netdev, as this most probably is a networking bug.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 111+ messages in thread
* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
2011-06-06 20:10 ` Bart De Schuymer
@ 2011-06-07 3:33 ` Brad Campbell
-1 siblings, 0 replies; 111+ messages in thread
From: Brad Campbell @ 2011-06-07 3:33 UTC (permalink / raw)
To: Bart De Schuymer; +Cc: kvm, linux-mm, linux-kernel, netdev, netfilter-devel
On 07/06/11 04:10, Bart De Schuymer wrote:
> Hi Brad,
>
> This has probably nothing to do with ebtables, so please rmmod in case
> it's loaded.
> A few questions I didn't directly see an answer to in the threads I
> scanned...
> I'm assuming you actually use the bridging firewall functionality. So,
> what iptables modules do you use? Can you reduce your iptables rules to
> a core that triggers the bug?
> Or does it get triggered even with an empty set of firewall rules?
> Are you using a stock .35 kernel or is it patched?
> Is this something I can trigger on a poor guy's laptop or does it
> require specialized hardware (I'm catching up on qemu/kvm...)?
Not specialised hardware as such, I've just not been able to reproduce
it outside of this specific operating scenario.
I can't trigger it with empty firewall rules as it relies on a DNAT to
occur. If I try it directly to the internal IP address (as I have to
without netfilter loaded) then of course nothing fails.
It's a pain in the bum as a fault, but it's one I can easily reproduce
as long as I use the same set of circumstances.
I'll try using 3.0-rc2 (current git) tonight, and if I can reproduce it
on that then I'll attempt to pare down the IPTABLES rules to a bare minimum.
It is nothing to do with ebtables as I don't compile it. I'm not really
sure about "bridging firewall" functionality. I just use a couple of
hand coded bash scripts to set the tables up.
brad@srv:~$ lsmod
Module Size Used by
xt_iprange 1637 1
xt_DSCP 2077 2
xt_length 1216 1
xt_CLASSIFY 1091 26
sch_sfq 6681 4
xt_CHECKSUM 1229 2 brad@srv:~$ lsmod
Module Size Used by
xt_iprange 1637 1
xt_DSCP 2077 2
xt_length 1216 1
xt_CLASSIFY 1091 26
sch_sfq 6681 4
xt_CHECKSUM 1229 2
ipt_REJECT 2277 1
ipt_MASQUERADE 1759 7
ipt_REDIRECT 1133 1
xt_recent 8223 2
xt_state 1226 5
iptable_nat 3993 1
nf_nat 16773 3 ipt_MASQUERADE,ipt_REDIRECT,iptable_nat
nf_conntrack_ipv4 11868 8 iptable_nat,nf_nat
nf_conntrack 60962 5
ipt_MASQUERADE,xt_state,iptable_nat,nf_nat,nf_conntrack_ipv4
nf_defrag_ipv4 1417 1 nf_conntrack_ipv4
xt_TCPMSS 2567 2
xt_tcpmss 1469 0
xt_tcpudp 2467 56
iptable_mangle 1487 1
pppoe 9574 2
pppox 2188 1 pppoe
iptable_filter 1442 1
ip_tables 16762 3 iptable_nat,iptable_mangle,iptable_filter
x_tables 20462 17
xt_iprange,xt_DSCP,xt_length,xt_CLASSIFY,xt_CHECKSUM,ipt_REJECT,ipt_MASQUERADE,ipt_REDIRECT,xt_recent,xt_state,iptable_nat,xt_TCPMSS,xt_tcpmss,xt_tcpudp,iptable_mangle,iptable_filter,ip_tables
ppp_generic 24243 6 pppoe,pppox
slhc 5293 1 ppp_generic
cls_u32 6468 6
sch_htb 14432 2
deflate 1937 0
zlib_deflate 21228 1 deflate
des_generic 16135 0
cbc 2721 0
ecb 1975 0
crypto_blkcipher 13645 2 cbc,ecb
sha1_generic 2095 0
md5 4001 0
hmac 2977 0
crypto_hash 14519 3 sha1_generic,md5,hmac
cryptomgr 2636 0
aead 6137 1 cryptomgr
crypto_algapi 15289 9
deflate,des_generic,cbc,ecb,crypto_blkcipher,hmac,crypto_hash,cryptomgr,aead
af_key 27372 0
fuse 66747 1
w83627ehf 32052 0
hwmon_vid 2867 1 w83627ehf
vhost_net 16802 6
powernow_k8 12932 0
mperf 1263 1 powernow_k8
kvm_amd 53431 24
kvm 235155 1 kvm_amd
pl2303 12732 1
xhci_hcd 62865 0
i2c_piix4 8391 0
k10temp 3183 0
usbserial 34452 3 pl2303
usb_storage 37887 1
usb_libusual 10999 1 usb_storage
ohci_hcd 18105 0
ehci_hcd 33641 0
ahci 20748 4
usbcore 130936 7
pl2303,xhci_hcd,usbserial,usb_storage,usb_libusual,ohci_hcd,ehci_hcd
libahci 21202 1 ahci
sata_mv 26939 0
megaraid_sas 71659 14
Nat Table (external ip substituted for xxx.xxx.xxx.xxx)
Chain PREROUTING (policy ACCEPT 1761K packets, 152M bytes)
pkts bytes target prot opt in out source
destination
5 210 DNAT udp -- ppp0 * 0.0.0.0/0
0.0.0.0/0 udp dpt:1195 to:192.168.253.199
6 252 DNAT udp -- !ppp0 * 0.0.0.0/0
xxx.xxx.xxx.xxx udp dpt:1195 to:192.168.253.199
0 0 DNAT tcp -- ppp0 * 0.0.0.0/0
0.0.0.0/0 tcp dpt:25001 to:192.168.253.199:465
0 0 DNAT tcp -- ppp0 * 0.0.0.0/0
0.0.0.0/0 tcp dpt:25000 to:192.168.253.199:993
0 0 DNAT tcp -- !ppp0 * 0.0.0.0/0
xxx.xxx.xxx.xxx tcp dpt:25001 to:192.168.253.199:465
0 0 DNAT tcp -- !ppp0 * 0.0.0.0/0
xxx.xxx.xxx.xxx tcp dpt:25000 to:192.168.253.199:993
2 142 DNAT 47 -- ppp0 * 0.0.0.0/0
0.0.0.0/0 to:192.168.253.199
18 880 DNAT tcp -- ppp0 * 0.0.0.0/0
0.0.0.0/0 tcp dpt:1723 to:192.168.253.199
0 0 DNAT 47 -- !ppp0 * 0.0.0.0/0
xxx.xxx.xxx.xxx to:192.168.253.199
0 0 DNAT tcp -- !ppp0 * 0.0.0.0/0
xxx.xxx.xxx.xxx tcp dpt:1723 to:192.168.253.199
2969 149K DNAT tcp -- ppp0 * 0.0.0.0/0
0.0.0.0/0 tcp dpt:443 to:192.168.253.198
20 1280 DNAT tcp -- !ppp0 * 0.0.0.0/0
xxx.xxx.xxx.xxx tcp dpt:443 to:192.168.253.198
0 0 DNAT tcp -- ppp0 * 0.0.0.0/0
0.0.0.0/0 tcp dpt:3101 to:192.168.253.197
0 0 DNAT tcp -- !ppp0 * 0.0.0.0/0
xxx.xxx.xxx.xxx tcp dpt:3101 to:192.168.253.197
0 0 DNAT tcp -- !ppp0 * 0.0.0.0/0
xxx.xxx.xxx.xxx tcp dpt:4101 to:192.168.253.197
44528 2718K REDIRECT tcp -- !ppp0 * 0.0.0.0/0
!192.168.0.0/16 tcp dpt:80 redir ports 8080
0 0 DNAT tcp -- * * 0.0.0.0/0
0.0.0.0/0 tcp dpt:3724 to:192.168.2.107
596K 33M DNAT tcp -- ppp0 * 0.0.0.0/0
0.0.0.0/0 tcp dpts:2001:2030 to:10.99.99.2
1420K 119M DNAT udp -- ppp0 * 0.0.0.0/0
0.0.0.0/0 udp dpts:2001:2030 to:10.99.99.2
7483 449K DNAT all -- !ppp0 * 0.0.0.0/0
xxx.xxx.xxx.xxx to:192.168.2.1
Mangle Table
Chain INPUT (policy ACCEPT 270K packets, 17M bytes)
pkts bytes target prot opt in out source
destination
Chain OUTPUT (policy ACCEPT 170K packets, 12M bytes)
pkts bytes target prot opt in out source
destination
Chain POSTROUTING (policy ACCEPT 2205K packets, 166M bytes)
pkts bytes target prot opt in out source
destination
0 0 MASQUERADE all -- * * 0.0.0.0/0
192.168.254.3
6 360 ACCEPT all -- * * 0.0.0.0/0
xxx.xxx.xxx.xxx
20424 2120K MASQUERADE all -- * ppp0 192.168.0.0/16
!192.168.0.0/16
0 0 MASQUERADE all -- * ppp0 10.0.0.0/24
0.0.0.0/0
3 204 MASQUERADE all -- * * 192.168.2.0/24
10.8.0.0/24
1418K 128M MASQUERADE all -- * * 10.99.99.0/24
0.0.0.0/0
68248 4095K MASQUERADE all -- * * 192.168.253.0/24
10.8.0.0/16
13305 2405K MASQUERADE all -- * * 192.168.253.0/24
!192.168.0.0/16
Chain PREROUTING (policy ACCEPT 278M packets, 293G bytes)
pkts bytes target prot opt in out source
destination
169 55528 CHECKSUM udp -- br1 * 0.0.0.0/0
0.0.0.0/0 udp dpt:67 CHECKSUM fill
Chain INPUT (policy ACCEPT 180M packets, 250G bytes)
pkts bytes target prot opt in out source
destination
Chain FORWARD (policy ACCEPT 98M packets, 44G bytes)
pkts bytes target prot opt in out source
destination
Chain OUTPUT (policy ACCEPT 155M packets, 180G bytes)
pkts bytes target prot opt in out source
destination
Chain POSTROUTING (policy ACCEPT 253M packets, 223G bytes)
pkts bytes target prot opt in out source
destination
165 54182 CHECKSUM udp -- * br1 0.0.0.0/0
0.0.0.0/0 udp spt:67 CHECKSUM fill
51 3712 CLASSIFY tcp -- * ppp0 0.0.0.0/0
0.0.0.0/0 tcp dpt:53 CLASSIFY set 1:20
85274 6454K CLASSIFY udp -- * ppp0 0.0.0.0/0
0.0.0.0/0 udp dpt:53 CLASSIFY set 1:20
187 257K CLASSIFY tcp -- * ppp0 0.0.0.0/0
0.0.0.0/0 tcp spt:81 CLASSIFY set 1:20
25M 1180M CLASSIFY tcp -- * ppp0 0.0.0.0/0
0.0.0.0/0 tcp flags:0x3F/0x10 state ESTABLISHED length 40:100
CLASSIFY set 1:15
728K 67M CLASSIFY icmp -- * ppp0 0.0.0.0/0
0.0.0.0/0 CLASSIFY set 1:15
231 23484 CLASSIFY tcp -- * ppp0 0.0.0.0/0
0.0.0.0/0 tcp dpt:2401 CLASSIFY set 1:15
65636 5610K CLASSIFY tcp -- * ppp0 0.0.0.0/0
0.0.0.0/0 tcp dpt:22 CLASSIFY set 1:10
2018 315K CLASSIFY tcp -- * ppp0 0.0.0.0/0
0.0.0.0/0 tcp spt:22 CLASSIFY set 1:10
80 10092 CLASSIFY tcp -- * ppp0 0.0.0.0/0
0.0.0.0/0 tcp dpt:3389 CLASSIFY set 1:10
26063 8910K CLASSIFY tcp -- * ppp0 0.0.0.0/0
0.0.0.0/0 tcp dpt:8080 CLASSIFY set 1:15
932K 131M CLASSIFY tcp -- * ppp0 0.0.0.0/0
0.0.0.0/0 tcp dpt:80 CLASSIFY set 1:15
3511 267K CLASSIFY udp -- * ppp0 0.0.0.0/0
0.0.0.0/0 udp dpt:123 CLASSIFY set 1:10
0 0 CLASSIFY tcp -- * ppp0 0.0.0.0/0
0.0.0.0/0 tcp spt:20 CLASSIFY set 1:15
3 180 CLASSIFY tcp -- * ppp0 0.0.0.0/0
0.0.0.0/0 tcp dpt:20 CLASSIFY set 1:15
94058 38M CLASSIFY 47 -- * ppp0 0.0.0.0/0
0.0.0.0/0 CLASSIFY set 1:10
1086K 183M CLASSIFY udp -- * ppp0 0.0.0.0/0
0.0.0.0/0 udp spt:1194 CLASSIFY set 1:10
1086K 183M TOS udp -- * ppp0 0.0.0.0/0
0.0.0.0/0 udp spt:1194 TOS set 0x10/0x3f
48817 10M CLASSIFY udp -- * ppp0 0.0.0.0/0
0.0.0.0/0 udp spt:1195 CLASSIFY set 1:10
48817 10M TOS udp -- * ppp0 0.0.0.0/0
0.0.0.0/0 udp spt:1195 TOS set 0x10/0x3f
94058 38M CLASSIFY 47 -- * ppp0 0.0.0.0/0
0.0.0.0/0 CLASSIFY set 1:15
106 7207 CLASSIFY tcp -- * ppp0 0.0.0.0/0
0.0.0.0/0 tcp dpt:1863 CLASSIFY set 1:15
188K 34M CLASSIFY tcp -- * ppp0 0.0.0.0/0
0.0.0.0/0 tcp dpt:443 CLASSIFY set 1:15
51541 3327K CLASSIFY tcp -- * ppp0 0.0.0.0/0
0.0.0.0/0 tcp dpts:6660:6669 CLASSIFY set 1:15
0 0 CLASSIFY tcp -- * ppp0 0.0.0.0/0
0.0.0.0/0 tcp spts:2021:2030 CLASSIFY set 1:15
85 4944 CLASSIFY tcp -- * ppp0 0.0.0.0/0
0.0.0.0/0 tcp dpt:19999 CLASSIFY set 1:15
208K 86M CLASSIFY udp -- * * 0.0.0.0/0
0.0.0.0/0 source IP range 192.168.2.80-192.168.2.120 CLASSIFY
set 1:10
0 0 CLASSIFY tcp -- * ppp0 0.0.0.0/0
0.0.0.0/0 tcp spt:12345 CLASSIFY set 1:15
1 80 CLASSIFY udp -- * ppp0 0.0.0.0/0
0.0.0.0/0 udp spt:12345 CLASSIFY set 1:15
Default table
Chain INPUT (policy ACCEPT 176M packets, 247G bytes)
pkts bytes target prot opt in out source
destination
0 0 ACCEPT udp -- ppp0 * 0.0.0.0/0
0.0.0.0/0 udp dpt:4569
1187K 582M ACCEPT udp -- ppp0 * 0.0.0.0/0
0.0.0.0/0 udp dpt:1194
2 577 ACCEPT udp -- ppp0 * 0.0.0.0/0
0.0.0.0/0 udp dpt:1195
28 1224 ACCEPT tcp -- ppp0 * 0.0.0.0/0
0.0.0.0/0 tcp dpt:3389
230 12372 tcp -- ppp0 * 0.0.0.0/0
0.0.0.0/0 tcp dpt:22 state NEW recent: SET name: DEFAULT side:
source
3 180 DROP tcp -- ppp0 * 0.0.0.0/0
0.0.0.0/0 tcp dpt:22 state NEW recent: UPDATE seconds: 300
hit_count: 4 name: DEFAULT side: source
1750 143K ACCEPT tcp -- ppp0 * 0.0.0.0/0
0.0.0.0/0 tcp dpt:22
3 144 ACCEPT tcp -- ppp0 * 0.0.0.0/0
0.0.0.0/0 tcp dpt:113
120 6090 ACCEPT tcp -- ppp0 * 0.0.0.0/0
0.0.0.0/0 tcp dpt:81
36094 29M ACCEPT tcp -- ppp0 * 0.0.0.0/0
0.0.0.0/0 tcp dpt:25
1456K 1706M ACCEPT all -- ppp0 * 0.0.0.0/0
0.0.0.0/0 state RELATED,ESTABLISHED
31047 2334K REJECT tcp -- ppp0 * 0.0.0.0/0
0.0.0.0/0 tcp option=!2 reject-with tcp-reset
552K 60M ACCEPT all -- !ppp0 * 0.0.0.0/0
0.0.0.0/0 state NEW
13552 1207K ACCEPT icmp -- ppp0 * 0.0.0.0/0
0.0.0.0/0
5712 392K DROP all -- ppp0 * 0.0.0.0/0
0.0.0.0/0
Chain FORWARD (policy ACCEPT 98M packets, 44G bytes)
pkts bytes target prot opt in out source
destination
1207K 68M TCPMSS tcp -- * ppp0 0.0.0.0/0
0.0.0.0/0 tcp flags:0x06/0x02 TCPMSS clamp to PMTU
Chain OUTPUT (policy ACCEPT 155M packets, 180G bytes)
pkts bytes target prot opt in out source
destination
31675 1895K TCPMSS tcp -- * ppp0 0.0.0.0/0
0.0.0.0/0 tcp flags:0x06/0x02 TCPMSS clamp to PMTU
lsmod
ipt_REJECT 2277 1
ipt_MASQUERADE 1759 7
ipt_REDIRECT 1133 1
xt_recent 8223 2
xt_state 1226 5
iptable_nat 3993 1
nf_nat 16773 3 ipt_MASQUERADE,ipt_REDIRECT,iptable_nat
nf_conntrack_ipv4 11868 8 iptable_nat,nf_nat
nf_conntrack 60962 5
ipt_MASQUERADE,xt_state,iptable_nat,nf_nat,nf_conntrack_ipv4
nf_defrag_ipv4 1417 1 nf_conntrack_ipv4
xt_TCPMSS 2567 2
xt_tcpmss 1469 0
xt_tcpudp 2467 56
iptable_mangle 1487 1
pppoe 9574 2
pppox 2188 1 pppoe
iptable_filter 1442 1
ip_tables 16762 3 iptable_nat,iptable_mangle,iptable_filter
x_tables 20462 17
xt_iprange,xt_DSCP,xt_length,xt_CLASSIFY,xt_CHECKSUM,ipt_REJECT,ipt_MASQUERADE,ipt_REDIRECT,xt_recent,xt_state,iptable_nat,xt_TCPMSS,xt_tcpmss,xt_tcpudp,iptable_mangle,iptable_filter,ip_tables
ppp_generic 24243 6 pppoe,pppox
slhc 5293 1 ppp_generic
cls_u32 6468 6
sch_htb 14432 2
deflate 1937 0
zlib_deflate 21228 1 deflate
des_generic 16135 0
cbc 2721 0
ecb 1975 0
crypto_blkcipher 13645 2 cbc,ecb
sha1_generic 2095 0
md5 4001 0
hmac 2977 0
crypto_hash 14519 3 sha1_generic,md5,hmac
cryptomgr 2636 0
aead 6137 1 cryptomgr
crypto_algapi 15289 9
deflate,des_generic,cbc,ecb,crypto_blkcipher,hmac,crypto_hash,cryptomgr,aead
af_key 27372 0
fuse 66747 1
w83627ehf 32052 0
hwmon_vid 2867 1 w83627ehf
vhost_net 16802 6
powernow_k8 12932 0
mperf 1263 1 powernow_k8
kvm_amd 53431 24
kvm 235155 1 kvm_amd
pl2303 12732 1
xhci_hcd 62865 0
i2c_piix4 8391 0
k10temp 3183 0
usbserial 34452 3 pl2303
usb_storage 37887 1
usb_libusual 10999 1 usb_storage
ohci_hcd 18105 0
ehci_hcd 33641 0
ahci 20748 4
usbcore 130936 7
pl2303,xhci_hcd,usbserial,usb_storage,usb_libusual,ohci_hcd,ehci_hcd
libahci 21202 1 ahci
sata_mv 26939 0
megaraid_sas 71659 14
^ permalink raw reply [flat|nested] 111+ messages in thread
* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
@ 2011-06-07 3:33 ` Brad Campbell
0 siblings, 0 replies; 111+ messages in thread
From: Brad Campbell @ 2011-06-07 3:33 UTC (permalink / raw)
To: Bart De Schuymer; +Cc: kvm, linux-mm, linux-kernel, netdev, netfilter-devel
On 07/06/11 04:10, Bart De Schuymer wrote:
> Hi Brad,
>
> This has probably nothing to do with ebtables, so please rmmod in case
> it's loaded.
> A few questions I didn't directly see an answer to in the threads I
> scanned...
> I'm assuming you actually use the bridging firewall functionality. So,
> what iptables modules do you use? Can you reduce your iptables rules to
> a core that triggers the bug?
> Or does it get triggered even with an empty set of firewall rules?
> Are you using a stock .35 kernel or is it patched?
> Is this something I can trigger on a poor guy's laptop or does it
> require specialized hardware (I'm catching up on qemu/kvm...)?
Not specialised hardware as such, I've just not been able to reproduce
it outside of this specific operating scenario.
I can't trigger it with empty firewall rules as it relies on a DNAT to
occur. If I try it directly to the internal IP address (as I have to
without netfilter loaded) then of course nothing fails.
It's a pain in the bum as a fault, but it's one I can easily reproduce
as long as I use the same set of circumstances.
I'll try using 3.0-rc2 (current git) tonight, and if I can reproduce it
on that then I'll attempt to pare down the IPTABLES rules to a bare minimum.
It is nothing to do with ebtables as I don't compile it. I'm not really
sure about "bridging firewall" functionality. I just use a couple of
hand coded bash scripts to set the tables up.
brad@srv:~$ lsmod
Module Size Used by
xt_iprange 1637 1
xt_DSCP 2077 2
xt_length 1216 1
xt_CLASSIFY 1091 26
sch_sfq 6681 4
xt_CHECKSUM 1229 2 brad@srv:~$ lsmod
Module Size Used by
xt_iprange 1637 1
xt_DSCP 2077 2
xt_length 1216 1
xt_CLASSIFY 1091 26
sch_sfq 6681 4
xt_CHECKSUM 1229 2
ipt_REJECT 2277 1
ipt_MASQUERADE 1759 7
ipt_REDIRECT 1133 1
xt_recent 8223 2
xt_state 1226 5
iptable_nat 3993 1
nf_nat 16773 3 ipt_MASQUERADE,ipt_REDIRECT,iptable_nat
nf_conntrack_ipv4 11868 8 iptable_nat,nf_nat
nf_conntrack 60962 5
ipt_MASQUERADE,xt_state,iptable_nat,nf_nat,nf_conntrack_ipv4
nf_defrag_ipv4 1417 1 nf_conntrack_ipv4
xt_TCPMSS 2567 2
xt_tcpmss 1469 0
xt_tcpudp 2467 56
iptable_mangle 1487 1
pppoe 9574 2
pppox 2188 1 pppoe
iptable_filter 1442 1
ip_tables 16762 3 iptable_nat,iptable_mangle,iptable_filter
x_tables 20462 17
xt_iprange,xt_DSCP,xt_length,xt_CLASSIFY,xt_CHECKSUM,ipt_REJECT,ipt_MASQUERADE,ipt_REDIRECT,xt_recent,xt_state,iptable_nat,xt_TCPMSS,xt_tcpmss,xt_tcpudp,iptable_mangle,iptable_filter,ip_tables
ppp_generic 24243 6 pppoe,pppox
slhc 5293 1 ppp_generic
cls_u32 6468 6
sch_htb 14432 2
deflate 1937 0
zlib_deflate 21228 1 deflate
des_generic 16135 0
cbc 2721 0
ecb 1975 0
crypto_blkcipher 13645 2 cbc,ecb
sha1_generic 2095 0
md5 4001 0
hmac 2977 0
crypto_hash 14519 3 sha1_generic,md5,hmac
cryptomgr 2636 0
aead 6137 1 cryptomgr
crypto_algapi 15289 9
deflate,des_generic,cbc,ecb,crypto_blkcipher,hmac,crypto_hash,cryptomgr,aead
af_key 27372 0
fuse 66747 1
w83627ehf 32052 0
hwmon_vid 2867 1 w83627ehf
vhost_net 16802 6
powernow_k8 12932 0
mperf 1263 1 powernow_k8
kvm_amd 53431 24
kvm 235155 1 kvm_amd
pl2303 12732 1
xhci_hcd 62865 0
i2c_piix4 8391 0
k10temp 3183 0
usbserial 34452 3 pl2303
usb_storage 37887 1
usb_libusual 10999 1 usb_storage
ohci_hcd 18105 0
ehci_hcd 33641 0
ahci 20748 4
usbcore 130936 7
pl2303,xhci_hcd,usbserial,usb_storage,usb_libusual,ohci_hcd,ehci_hcd
libahci 21202 1 ahci
sata_mv 26939 0
megaraid_sas 71659 14
Nat Table (external ip substituted for xxx.xxx.xxx.xxx)
Chain PREROUTING (policy ACCEPT 1761K packets, 152M bytes)
pkts bytes target prot opt in out source
destination
5 210 DNAT udp -- ppp0 * 0.0.0.0/0
0.0.0.0/0 udp dpt:1195 to:192.168.253.199
6 252 DNAT udp -- !ppp0 * 0.0.0.0/0
xxx.xxx.xxx.xxx udp dpt:1195 to:192.168.253.199
0 0 DNAT tcp -- ppp0 * 0.0.0.0/0
0.0.0.0/0 tcp dpt:25001 to:192.168.253.199:465
0 0 DNAT tcp -- ppp0 * 0.0.0.0/0
0.0.0.0/0 tcp dpt:25000 to:192.168.253.199:993
0 0 DNAT tcp -- !ppp0 * 0.0.0.0/0
xxx.xxx.xxx.xxx tcp dpt:25001 to:192.168.253.199:465
0 0 DNAT tcp -- !ppp0 * 0.0.0.0/0
xxx.xxx.xxx.xxx tcp dpt:25000 to:192.168.253.199:993
2 142 DNAT 47 -- ppp0 * 0.0.0.0/0
0.0.0.0/0 to:192.168.253.199
18 880 DNAT tcp -- ppp0 * 0.0.0.0/0
0.0.0.0/0 tcp dpt:1723 to:192.168.253.199
0 0 DNAT 47 -- !ppp0 * 0.0.0.0/0
xxx.xxx.xxx.xxx to:192.168.253.199
0 0 DNAT tcp -- !ppp0 * 0.0.0.0/0
xxx.xxx.xxx.xxx tcp dpt:1723 to:192.168.253.199
2969 149K DNAT tcp -- ppp0 * 0.0.0.0/0
0.0.0.0/0 tcp dpt:443 to:192.168.253.198
20 1280 DNAT tcp -- !ppp0 * 0.0.0.0/0
xxx.xxx.xxx.xxx tcp dpt:443 to:192.168.253.198
0 0 DNAT tcp -- ppp0 * 0.0.0.0/0
0.0.0.0/0 tcp dpt:3101 to:192.168.253.197
0 0 DNAT tcp -- !ppp0 * 0.0.0.0/0
xxx.xxx.xxx.xxx tcp dpt:3101 to:192.168.253.197
0 0 DNAT tcp -- !ppp0 * 0.0.0.0/0
xxx.xxx.xxx.xxx tcp dpt:4101 to:192.168.253.197
44528 2718K REDIRECT tcp -- !ppp0 * 0.0.0.0/0
!192.168.0.0/16 tcp dpt:80 redir ports 8080
0 0 DNAT tcp -- * * 0.0.0.0/0
0.0.0.0/0 tcp dpt:3724 to:192.168.2.107
596K 33M DNAT tcp -- ppp0 * 0.0.0.0/0
0.0.0.0/0 tcp dpts:2001:2030 to:10.99.99.2
1420K 119M DNAT udp -- ppp0 * 0.0.0.0/0
0.0.0.0/0 udp dpts:2001:2030 to:10.99.99.2
7483 449K DNAT all -- !ppp0 * 0.0.0.0/0
xxx.xxx.xxx.xxx to:192.168.2.1
Mangle Table
Chain INPUT (policy ACCEPT 270K packets, 17M bytes)
pkts bytes target prot opt in out source
destination
Chain OUTPUT (policy ACCEPT 170K packets, 12M bytes)
pkts bytes target prot opt in out source
destination
Chain POSTROUTING (policy ACCEPT 2205K packets, 166M bytes)
pkts bytes target prot opt in out source
destination
0 0 MASQUERADE all -- * * 0.0.0.0/0
192.168.254.3
6 360 ACCEPT all -- * * 0.0.0.0/0
xxx.xxx.xxx.xxx
20424 2120K MASQUERADE all -- * ppp0 192.168.0.0/16
!192.168.0.0/16
0 0 MASQUERADE all -- * ppp0 10.0.0.0/24
0.0.0.0/0
3 204 MASQUERADE all -- * * 192.168.2.0/24
10.8.0.0/24
1418K 128M MASQUERADE all -- * * 10.99.99.0/24
0.0.0.0/0
68248 4095K MASQUERADE all -- * * 192.168.253.0/24
10.8.0.0/16
13305 2405K MASQUERADE all -- * * 192.168.253.0/24
!192.168.0.0/16
Chain PREROUTING (policy ACCEPT 278M packets, 293G bytes)
pkts bytes target prot opt in out source
destination
169 55528 CHECKSUM udp -- br1 * 0.0.0.0/0
0.0.0.0/0 udp dpt:67 CHECKSUM fill
Chain INPUT (policy ACCEPT 180M packets, 250G bytes)
pkts bytes target prot opt in out source
destination
Chain FORWARD (policy ACCEPT 98M packets, 44G bytes)
pkts bytes target prot opt in out source
destination
Chain OUTPUT (policy ACCEPT 155M packets, 180G bytes)
pkts bytes target prot opt in out source
destination
Chain POSTROUTING (policy ACCEPT 253M packets, 223G bytes)
pkts bytes target prot opt in out source
destination
165 54182 CHECKSUM udp -- * br1 0.0.0.0/0
0.0.0.0/0 udp spt:67 CHECKSUM fill
51 3712 CLASSIFY tcp -- * ppp0 0.0.0.0/0
0.0.0.0/0 tcp dpt:53 CLASSIFY set 1:20
85274 6454K CLASSIFY udp -- * ppp0 0.0.0.0/0
0.0.0.0/0 udp dpt:53 CLASSIFY set 1:20
187 257K CLASSIFY tcp -- * ppp0 0.0.0.0/0
0.0.0.0/0 tcp spt:81 CLASSIFY set 1:20
25M 1180M CLASSIFY tcp -- * ppp0 0.0.0.0/0
0.0.0.0/0 tcp flags:0x3F/0x10 state ESTABLISHED length 40:100
CLASSIFY set 1:15
728K 67M CLASSIFY icmp -- * ppp0 0.0.0.0/0
0.0.0.0/0 CLASSIFY set 1:15
231 23484 CLASSIFY tcp -- * ppp0 0.0.0.0/0
0.0.0.0/0 tcp dpt:2401 CLASSIFY set 1:15
65636 5610K CLASSIFY tcp -- * ppp0 0.0.0.0/0
0.0.0.0/0 tcp dpt:22 CLASSIFY set 1:10
2018 315K CLASSIFY tcp -- * ppp0 0.0.0.0/0
0.0.0.0/0 tcp spt:22 CLASSIFY set 1:10
80 10092 CLASSIFY tcp -- * ppp0 0.0.0.0/0
0.0.0.0/0 tcp dpt:3389 CLASSIFY set 1:10
26063 8910K CLASSIFY tcp -- * ppp0 0.0.0.0/0
0.0.0.0/0 tcp dpt:8080 CLASSIFY set 1:15
932K 131M CLASSIFY tcp -- * ppp0 0.0.0.0/0
0.0.0.0/0 tcp dpt:80 CLASSIFY set 1:15
3511 267K CLASSIFY udp -- * ppp0 0.0.0.0/0
0.0.0.0/0 udp dpt:123 CLASSIFY set 1:10
0 0 CLASSIFY tcp -- * ppp0 0.0.0.0/0
0.0.0.0/0 tcp spt:20 CLASSIFY set 1:15
3 180 CLASSIFY tcp -- * ppp0 0.0.0.0/0
0.0.0.0/0 tcp dpt:20 CLASSIFY set 1:15
94058 38M CLASSIFY 47 -- * ppp0 0.0.0.0/0
0.0.0.0/0 CLASSIFY set 1:10
1086K 183M CLASSIFY udp -- * ppp0 0.0.0.0/0
0.0.0.0/0 udp spt:1194 CLASSIFY set 1:10
1086K 183M TOS udp -- * ppp0 0.0.0.0/0
0.0.0.0/0 udp spt:1194 TOS set 0x10/0x3f
48817 10M CLASSIFY udp -- * ppp0 0.0.0.0/0
0.0.0.0/0 udp spt:1195 CLASSIFY set 1:10
48817 10M TOS udp -- * ppp0 0.0.0.0/0
0.0.0.0/0 udp spt:1195 TOS set 0x10/0x3f
94058 38M CLASSIFY 47 -- * ppp0 0.0.0.0/0
0.0.0.0/0 CLASSIFY set 1:15
106 7207 CLASSIFY tcp -- * ppp0 0.0.0.0/0
0.0.0.0/0 tcp dpt:1863 CLASSIFY set 1:15
188K 34M CLASSIFY tcp -- * ppp0 0.0.0.0/0
0.0.0.0/0 tcp dpt:443 CLASSIFY set 1:15
51541 3327K CLASSIFY tcp -- * ppp0 0.0.0.0/0
0.0.0.0/0 tcp dpts:6660:6669 CLASSIFY set 1:15
0 0 CLASSIFY tcp -- * ppp0 0.0.0.0/0
0.0.0.0/0 tcp spts:2021:2030 CLASSIFY set 1:15
85 4944 CLASSIFY tcp -- * ppp0 0.0.0.0/0
0.0.0.0/0 tcp dpt:19999 CLASSIFY set 1:15
208K 86M CLASSIFY udp -- * * 0.0.0.0/0
0.0.0.0/0 source IP range 192.168.2.80-192.168.2.120 CLASSIFY
set 1:10
0 0 CLASSIFY tcp -- * ppp0 0.0.0.0/0
0.0.0.0/0 tcp spt:12345 CLASSIFY set 1:15
1 80 CLASSIFY udp -- * ppp0 0.0.0.0/0
0.0.0.0/0 udp spt:12345 CLASSIFY set 1:15
Default table
Chain INPUT (policy ACCEPT 176M packets, 247G bytes)
pkts bytes target prot opt in out source
destination
0 0 ACCEPT udp -- ppp0 * 0.0.0.0/0
0.0.0.0/0 udp dpt:4569
1187K 582M ACCEPT udp -- ppp0 * 0.0.0.0/0
0.0.0.0/0 udp dpt:1194
2 577 ACCEPT udp -- ppp0 * 0.0.0.0/0
0.0.0.0/0 udp dpt:1195
28 1224 ACCEPT tcp -- ppp0 * 0.0.0.0/0
0.0.0.0/0 tcp dpt:3389
230 12372 tcp -- ppp0 * 0.0.0.0/0
0.0.0.0/0 tcp dpt:22 state NEW recent: SET name: DEFAULT side:
source
3 180 DROP tcp -- ppp0 * 0.0.0.0/0
0.0.0.0/0 tcp dpt:22 state NEW recent: UPDATE seconds: 300
hit_count: 4 name: DEFAULT side: source
1750 143K ACCEPT tcp -- ppp0 * 0.0.0.0/0
0.0.0.0/0 tcp dpt:22
3 144 ACCEPT tcp -- ppp0 * 0.0.0.0/0
0.0.0.0/0 tcp dpt:113
120 6090 ACCEPT tcp -- ppp0 * 0.0.0.0/0
0.0.0.0/0 tcp dpt:81
36094 29M ACCEPT tcp -- ppp0 * 0.0.0.0/0
0.0.0.0/0 tcp dpt:25
1456K 1706M ACCEPT all -- ppp0 * 0.0.0.0/0
0.0.0.0/0 state RELATED,ESTABLISHED
31047 2334K REJECT tcp -- ppp0 * 0.0.0.0/0
0.0.0.0/0 tcp option=!2 reject-with tcp-reset
552K 60M ACCEPT all -- !ppp0 * 0.0.0.0/0
0.0.0.0/0 state NEW
13552 1207K ACCEPT icmp -- ppp0 * 0.0.0.0/0
0.0.0.0/0
5712 392K DROP all -- ppp0 * 0.0.0.0/0
0.0.0.0/0
Chain FORWARD (policy ACCEPT 98M packets, 44G bytes)
pkts bytes target prot opt in out source
destination
1207K 68M TCPMSS tcp -- * ppp0 0.0.0.0/0
0.0.0.0/0 tcp flags:0x06/0x02 TCPMSS clamp to PMTU
Chain OUTPUT (policy ACCEPT 155M packets, 180G bytes)
pkts bytes target prot opt in out source
destination
31675 1895K TCPMSS tcp -- * ppp0 0.0.0.0/0
0.0.0.0/0 tcp flags:0x06/0x02 TCPMSS clamp to PMTU
lsmod
ipt_REJECT 2277 1
ipt_MASQUERADE 1759 7
ipt_REDIRECT 1133 1
xt_recent 8223 2
xt_state 1226 5
iptable_nat 3993 1
nf_nat 16773 3 ipt_MASQUERADE,ipt_REDIRECT,iptable_nat
nf_conntrack_ipv4 11868 8 iptable_nat,nf_nat
nf_conntrack 60962 5
ipt_MASQUERADE,xt_state,iptable_nat,nf_nat,nf_conntrack_ipv4
nf_defrag_ipv4 1417 1 nf_conntrack_ipv4
xt_TCPMSS 2567 2
xt_tcpmss 1469 0
xt_tcpudp 2467 56
iptable_mangle 1487 1
pppoe 9574 2
pppox 2188 1 pppoe
iptable_filter 1442 1
ip_tables 16762 3 iptable_nat,iptable_mangle,iptable_filter
x_tables 20462 17
xt_iprange,xt_DSCP,xt_length,xt_CLASSIFY,xt_CHECKSUM,ipt_REJECT,ipt_MASQUERADE,ipt_REDIRECT,xt_recent,xt_state,iptable_nat,xt_TCPMSS,xt_tcpmss,xt_tcpudp,iptable_mangle,iptable_filter,ip_tables
ppp_generic 24243 6 pppoe,pppox
slhc 5293 1 ppp_generic
cls_u32 6468 6
sch_htb 14432 2
deflate 1937 0
zlib_deflate 21228 1 deflate
des_generic 16135 0
cbc 2721 0
ecb 1975 0
crypto_blkcipher 13645 2 cbc,ecb
sha1_generic 2095 0
md5 4001 0
hmac 2977 0
crypto_hash 14519 3 sha1_generic,md5,hmac
cryptomgr 2636 0
aead 6137 1 cryptomgr
crypto_algapi 15289 9
deflate,des_generic,cbc,ecb,crypto_blkcipher,hmac,crypto_hash,cryptomgr,aead
af_key 27372 0
fuse 66747 1
w83627ehf 32052 0
hwmon_vid 2867 1 w83627ehf
vhost_net 16802 6
powernow_k8 12932 0
mperf 1263 1 powernow_k8
kvm_amd 53431 24
kvm 235155 1 kvm_amd
pl2303 12732 1
xhci_hcd 62865 0
i2c_piix4 8391 0
k10temp 3183 0
usbserial 34452 3 pl2303
usb_storage 37887 1
usb_libusual 10999 1 usb_storage
ohci_hcd 18105 0
ehci_hcd 33641 0
ahci 20748 4
usbcore 130936 7
pl2303,xhci_hcd,usbserial,usb_storage,usb_libusual,ohci_hcd,ehci_hcd
libahci 21202 1 ahci
sata_mv 26939 0
megaraid_sas 71659 14
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 111+ messages in thread
* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
2011-06-06 20:22 ` Eric Dumazet
(?)
(?)
@ 2011-06-07 13:27 ` Brad Campbell
2011-06-07 13:37 ` Eric Dumazet
-1 siblings, 1 reply; 111+ messages in thread
From: Brad Campbell @ 2011-06-07 13:27 UTC (permalink / raw)
To: Eric Dumazet; +Cc: Avi Kivity, CaT, Borislav Petkov, linux-kernel, kvm, netdev
On 07/06/11 04:22, Eric Dumazet wrote:
> Could you please try latest linux-2.6 tree ?
>
> We fixed many networking bugs that could explain your crash.
>
>
>
>
No good I'm afraid.
[ 543.040056]
=============================================================================
[ 543.040136] BUG ip_dst_cache: Padding overwritten.
0xffff8803e4217ffe-0xffff8803e4217fff
[ 543.040194]
-----------------------------------------------------------------------------
[ 543.040198]
[ 543.040298] INFO: Slab 0xffffea000d9e74d0 objects=25 used=25 fp=0x
(null) flags=0x8000000000004081
[ 543.040364] Pid: 4576, comm: kworker/1:2 Not tainted 3.0.0-rc2 #1
[ 543.040415] Call Trace:
[ 543.040472] [<ffffffff810b9c1d>] ? slab_err+0xad/0xd0
[ 543.040528] [<ffffffff8102e034>] ? check_preempt_wakeup+0xa4/0x160
[ 543.040595] [<ffffffff810ba206>] ? slab_pad_check+0x126/0x170
[ 543.040650] [<ffffffff8133045b>] ? dst_destroy+0x8b/0x110
[ 543.040701] [<ffffffff810ba29a>] ? check_slab+0x4a/0xc0
[ 543.040753] [<ffffffff810baf2d>] ? free_debug_processing+0x2d/0x250
[ 543.040808] [<ffffffff810bb27b>] ? __slab_free+0x12b/0x140
[ 543.040862] [<ffffffff810bbe99>] ? kmem_cache_free+0x99/0xa0
[ 543.040915] [<ffffffff8133045b>] ? dst_destroy+0x8b/0x110
[ 543.040967] [<ffffffff813307f6>] ? dst_gc_task+0x196/0x1f0
[ 543.041021] [<ffffffff8104e954>] ? queue_delayed_work_on+0x154/0x160
[ 543.041081] [<ffffffff813066fe>] ? do_dbs_timer+0x20e/0x3d0
[ 543.041133] [<ffffffff81330660>] ? dst_alloc+0x180/0x180
[ 543.041187] [<ffffffff8104f28b>] ? process_one_work+0xfb/0x3b0
[ 543.041242] [<ffffffff8104f964>] ? worker_thread+0x144/0x3d0
[ 543.041296] [<ffffffff8102cc10>] ? __wake_up_common+0x50/0x80
[ 543.041678] [<ffffffff8104f820>] ? rescuer_thread+0x2e0/0x2e0
[ 543.041729] [<ffffffff8104f820>] ? rescuer_thread+0x2e0/0x2e0
[ 543.041782] [<ffffffff81053436>] ? kthread+0x96/0xa0
[ 543.041835] [<ffffffff813e1d14>] ? kernel_thread_helper+0x4/0x10
[ 543.041890] [<ffffffff810533a0>] ? kthread_worker_fn+0x120/0x120
[ 543.041944] [<ffffffff813e1d10>] ? gs_change+0xb/0xb
[ 543.041993] Padding 0xffff8803e4217f40: 5a 5a 5a 5a 5a 5a 5a 5a 5a
5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
[ 543.042718] Padding 0xffff8803e4217f50: 5a 5a 5a 5a 5a 5a 5a 5a 5a
5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
[ 543.043433] Padding 0xffff8803e4217f60: 5a 5a 5a 5a 5a 5a 5a 5a 5a
5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
[ 543.044155] Padding 0xffff8803e4217f70: 5a 5a 5a 5a 5a 5a 5a 5a 5a
5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
[ 543.044866] Padding 0xffff8803e4217f80: 5a 5a 5a 5a 5a 5a 5a 5a 5a
5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
[ 543.045590] Padding 0xffff8803e4217f90: 5a 5a 5a 5a 5a 5a 5a 5a 5a
5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
[ 543.046311] Padding 0xffff8803e4217fa0: 5a 5a 5a 5a 5a 5a 5a 5a 5a
5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
[ 543.047034] Padding 0xffff8803e4217fb0: 5a 5a 5a 5a 5a 5a 5a 5a 5a
5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
[ 543.047755] Padding 0xffff8803e4217fc0: 5a 5a 5a 5a 5a 5a 5a 5a 5a
5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
[ 543.048474] Padding 0xffff8803e4217fd0: 5a 5a 5a 5a 5a 5a 5a 5a 5a
5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
[ 543.049203] Padding 0xffff8803e4217fe0: 5a 5a 5a 5a 5a 5a 5a 5a 5a
5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
[ 543.049909] Padding 0xffff8803e4217ff0: 5a 5a 5a 5a 5a 5a 5a 5a 5a
5a 5a 5a 5a 5a 00 00 ZZZZZZZZZZZZZZ..
[ 543.050021] FIX ip_dst_cache: Restoring
0xffff8803e4217f40-0xffff8803e4217fff=0x5a
[ 543.050021]
Dropped -mm, Hugh and Andrea from CC as this does not appear to be mm or
ksm related.
I'll pare down the firewall and see if I can make it break easier with a
smaller test set.
^ permalink raw reply [flat|nested] 111+ messages in thread
* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
2011-06-07 3:33 ` Brad Campbell
@ 2011-06-07 13:30 ` Patrick McHardy
-1 siblings, 0 replies; 111+ messages in thread
From: Patrick McHardy @ 2011-06-07 13:30 UTC (permalink / raw)
To: Brad Campbell
Cc: Bart De Schuymer, kvm, linux-mm, linux-kernel, netdev, netfilter-devel
On 07.06.2011 05:33, Brad Campbell wrote:
> On 07/06/11 04:10, Bart De Schuymer wrote:
>> Hi Brad,
>>
>> This has probably nothing to do with ebtables, so please rmmod in case
>> it's loaded.
>> A few questions I didn't directly see an answer to in the threads I
>> scanned...
>> I'm assuming you actually use the bridging firewall functionality. So,
>> what iptables modules do you use? Can you reduce your iptables rules to
>> a core that triggers the bug?
>> Or does it get triggered even with an empty set of firewall rules?
>> Are you using a stock .35 kernel or is it patched?
>> Is this something I can trigger on a poor guy's laptop or does it
>> require specialized hardware (I'm catching up on qemu/kvm...)?
>
> Not specialised hardware as such, I've just not been able to reproduce
> it outside of this specific operating scenario.
The last similar problem we've had was related to the 32/64 bit compat
code. Are you running 32 bit userspace on a 64 bit kernel?
> I can't trigger it with empty firewall rules as it relies on a DNAT to
> occur. If I try it directly to the internal IP address (as I have to
> without netfilter loaded) then of course nothing fails.
>
> It's a pain in the bum as a fault, but it's one I can easily reproduce
> as long as I use the same set of circumstances.
>
> I'll try using 3.0-rc2 (current git) tonight, and if I can reproduce it
> on that then I'll attempt to pare down the IPTABLES rules to a bare
> minimum.
>
> It is nothing to do with ebtables as I don't compile it. I'm not really
> sure about "bridging firewall" functionality. I just use a couple of
> hand coded bash scripts to set the tables up.
>From one of your previous mails:
> # CONFIG_BRIDGE_NF_EBTABLES is not set
How about CONFIG_BRIDGE_NETFILTER?
^ permalink raw reply [flat|nested] 111+ messages in thread
* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
@ 2011-06-07 13:30 ` Patrick McHardy
0 siblings, 0 replies; 111+ messages in thread
From: Patrick McHardy @ 2011-06-07 13:30 UTC (permalink / raw)
To: Brad Campbell
Cc: Bart De Schuymer, kvm, linux-mm, linux-kernel, netdev, netfilter-devel
On 07.06.2011 05:33, Brad Campbell wrote:
> On 07/06/11 04:10, Bart De Schuymer wrote:
>> Hi Brad,
>>
>> This has probably nothing to do with ebtables, so please rmmod in case
>> it's loaded.
>> A few questions I didn't directly see an answer to in the threads I
>> scanned...
>> I'm assuming you actually use the bridging firewall functionality. So,
>> what iptables modules do you use? Can you reduce your iptables rules to
>> a core that triggers the bug?
>> Or does it get triggered even with an empty set of firewall rules?
>> Are you using a stock .35 kernel or is it patched?
>> Is this something I can trigger on a poor guy's laptop or does it
>> require specialized hardware (I'm catching up on qemu/kvm...)?
>
> Not specialised hardware as such, I've just not been able to reproduce
> it outside of this specific operating scenario.
The last similar problem we've had was related to the 32/64 bit compat
code. Are you running 32 bit userspace on a 64 bit kernel?
> I can't trigger it with empty firewall rules as it relies on a DNAT to
> occur. If I try it directly to the internal IP address (as I have to
> without netfilter loaded) then of course nothing fails.
>
> It's a pain in the bum as a fault, but it's one I can easily reproduce
> as long as I use the same set of circumstances.
>
> I'll try using 3.0-rc2 (current git) tonight, and if I can reproduce it
> on that then I'll attempt to pare down the IPTABLES rules to a bare
> minimum.
>
> It is nothing to do with ebtables as I don't compile it. I'm not really
> sure about "bridging firewall" functionality. I just use a couple of
> hand coded bash scripts to set the tables up.
>From one of your previous mails:
> # CONFIG_BRIDGE_NF_EBTABLES is not set
How about CONFIG_BRIDGE_NETFILTER?
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 111+ messages in thread
* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
2011-06-07 13:27 ` Brad Campbell
@ 2011-06-07 13:37 ` Eric Dumazet
2011-06-07 15:15 ` Brad Campbell
2011-08-20 13:16 ` Brad Campbell
0 siblings, 2 replies; 111+ messages in thread
From: Eric Dumazet @ 2011-06-07 13:37 UTC (permalink / raw)
To: Brad Campbell; +Cc: Avi Kivity, CaT, Borislav Petkov, linux-kernel, kvm, netdev
Le mardi 07 juin 2011 à 21:27 +0800, Brad Campbell a écrit :
> On 07/06/11 04:22, Eric Dumazet wrote:
>
> > Could you please try latest linux-2.6 tree ?
> >
> > We fixed many networking bugs that could explain your crash.
> >
> >
> >
> >
>
> No good I'm afraid.
>
> [ 543.040056]
> =============================================================================
> [ 543.040136] BUG ip_dst_cache: Padding overwritten.
> 0xffff8803e4217ffe-0xffff8803e4217fff
> [ 543.040194]
Thats pretty strange : These are the last two bytes of a page, set to
0x0000 (a 16 bit value)
There is no way a dst field could actually sit on this location (its a
padding), since a dst is a bit less than 256 bytes (0xe8), and each
entry is aligned on a 64byte address.
grep dst /proc/slabinfo
ip_dst_cache 32823 62944 256 32 2 : tunables 0 0
0 : slabdata 1967 1967 0
sizeof(struct rtable)=0xe8
> -----------------------------------------------------------------------------
> [ 543.040198]
> [ 543.040298] INFO: Slab 0xffffea000d9e74d0 objects=25 used=25 fp=0x
> (null) flags=0x8000000000004081
> [ 543.040364] Pid: 4576, comm: kworker/1:2 Not tainted 3.0.0-rc2 #1
> [ 543.040415] Call Trace:
> [ 543.040472] [<ffffffff810b9c1d>] ? slab_err+0xad/0xd0
> [ 543.040528] [<ffffffff8102e034>] ? check_preempt_wakeup+0xa4/0x160
> [ 543.040595] [<ffffffff810ba206>] ? slab_pad_check+0x126/0x170
> [ 543.040650] [<ffffffff8133045b>] ? dst_destroy+0x8b/0x110
> [ 543.040701] [<ffffffff810ba29a>] ? check_slab+0x4a/0xc0
> [ 543.040753] [<ffffffff810baf2d>] ? free_debug_processing+0x2d/0x250
> [ 543.040808] [<ffffffff810bb27b>] ? __slab_free+0x12b/0x140
> [ 543.040862] [<ffffffff810bbe99>] ? kmem_cache_free+0x99/0xa0
> [ 543.040915] [<ffffffff8133045b>] ? dst_destroy+0x8b/0x110
> [ 543.040967] [<ffffffff813307f6>] ? dst_gc_task+0x196/0x1f0
> [ 543.041021] [<ffffffff8104e954>] ? queue_delayed_work_on+0x154/0x160
> [ 543.041081] [<ffffffff813066fe>] ? do_dbs_timer+0x20e/0x3d0
> [ 543.041133] [<ffffffff81330660>] ? dst_alloc+0x180/0x180
> [ 543.041187] [<ffffffff8104f28b>] ? process_one_work+0xfb/0x3b0
> [ 543.041242] [<ffffffff8104f964>] ? worker_thread+0x144/0x3d0
> [ 543.041296] [<ffffffff8102cc10>] ? __wake_up_common+0x50/0x80
> [ 543.041678] [<ffffffff8104f820>] ? rescuer_thread+0x2e0/0x2e0
> [ 543.041729] [<ffffffff8104f820>] ? rescuer_thread+0x2e0/0x2e0
> [ 543.041782] [<ffffffff81053436>] ? kthread+0x96/0xa0
> [ 543.041835] [<ffffffff813e1d14>] ? kernel_thread_helper+0x4/0x10
> [ 543.041890] [<ffffffff810533a0>] ? kthread_worker_fn+0x120/0x120
> [ 543.041944] [<ffffffff813e1d10>] ? gs_change+0xb/0xb
> [ 543.041993] Padding 0xffff8803e4217f40: 5a 5a 5a 5a 5a 5a 5a 5a 5a
> 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
> [ 543.042718] Padding 0xffff8803e4217f50: 5a 5a 5a 5a 5a 5a 5a 5a 5a
> 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
> [ 543.043433] Padding 0xffff8803e4217f60: 5a 5a 5a 5a 5a 5a 5a 5a 5a
> 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
> [ 543.044155] Padding 0xffff8803e4217f70: 5a 5a 5a 5a 5a 5a 5a 5a 5a
> 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
> [ 543.044866] Padding 0xffff8803e4217f80: 5a 5a 5a 5a 5a 5a 5a 5a 5a
> 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
> [ 543.045590] Padding 0xffff8803e4217f90: 5a 5a 5a 5a 5a 5a 5a 5a 5a
> 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
> [ 543.046311] Padding 0xffff8803e4217fa0: 5a 5a 5a 5a 5a 5a 5a 5a 5a
> 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
> [ 543.047034] Padding 0xffff8803e4217fb0: 5a 5a 5a 5a 5a 5a 5a 5a 5a
> 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
> [ 543.047755] Padding 0xffff8803e4217fc0: 5a 5a 5a 5a 5a 5a 5a 5a 5a
> 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
> [ 543.048474] Padding 0xffff8803e4217fd0: 5a 5a 5a 5a 5a 5a 5a 5a 5a
> 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
> [ 543.049203] Padding 0xffff8803e4217fe0: 5a 5a 5a 5a 5a 5a 5a 5a 5a
> 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
> [ 543.049909] Padding 0xffff8803e4217ff0: 5a 5a 5a 5a 5a 5a 5a 5a 5a
> 5a 5a 5a 5a 5a 00 00 ZZZZZZZZZZZZZZ..
> [ 543.050021] FIX ip_dst_cache: Restoring
> 0xffff8803e4217f40-0xffff8803e4217fff=0x5a
> [ 543.050021]
>
> Dropped -mm, Hugh and Andrea from CC as this does not appear to be mm or
> ksm related.
>
> I'll pare down the firewall and see if I can make it break easier with a
> smaller test set.
Hmm, not sure now :(
Could you reproduce another bug please ?
^ permalink raw reply [flat|nested] 111+ messages in thread
* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
2011-06-07 13:30 ` Patrick McHardy
@ 2011-06-07 14:40 ` Brad Campbell
-1 siblings, 0 replies; 111+ messages in thread
From: Brad Campbell @ 2011-06-07 14:40 UTC (permalink / raw)
To: Patrick McHardy
Cc: Bart De Schuymer, kvm, linux-mm, linux-kernel, netdev, netfilter-devel
On 07/06/11 21:30, Patrick McHardy wrote:
> On 07.06.2011 05:33, Brad Campbell wrote:
>> On 07/06/11 04:10, Bart De Schuymer wrote:
>>> Hi Brad,
>>>
>>> This has probably nothing to do with ebtables, so please rmmod in case
>>> it's loaded.
>>> A few questions I didn't directly see an answer to in the threads I
>>> scanned...
>>> I'm assuming you actually use the bridging firewall functionality. So,
>>> what iptables modules do you use? Can you reduce your iptables rules to
>>> a core that triggers the bug?
>>> Or does it get triggered even with an empty set of firewall rules?
>>> Are you using a stock .35 kernel or is it patched?
>>> Is this something I can trigger on a poor guy's laptop or does it
>>> require specialized hardware (I'm catching up on qemu/kvm...)?
>>
>> Not specialised hardware as such, I've just not been able to reproduce
>> it outside of this specific operating scenario.
>
> The last similar problem we've had was related to the 32/64 bit compat
> code. Are you running 32 bit userspace on a 64 bit kernel?
No, 32 bit Guest OS, but a completely 64 bit userspace on a 64 bit kernel.
Userspace is current Debian Stable. Kernel is Vanilla and qemu-kvm is
current git
>> I can't trigger it with empty firewall rules as it relies on a DNAT to
>> occur. If I try it directly to the internal IP address (as I have to
>> without netfilter loaded) then of course nothing fails.
>>
>> It's a pain in the bum as a fault, but it's one I can easily reproduce
>> as long as I use the same set of circumstances.
>>
>> I'll try using 3.0-rc2 (current git) tonight, and if I can reproduce it
>> on that then I'll attempt to pare down the IPTABLES rules to a bare
>> minimum.
>>
>> It is nothing to do with ebtables as I don't compile it. I'm not really
>> sure about "bridging firewall" functionality. I just use a couple of
>> hand coded bash scripts to set the tables up.
>
> From one of your previous mails:
>
>> # CONFIG_BRIDGE_NF_EBTABLES is not set
>
> How about CONFIG_BRIDGE_NETFILTER?
>
It was compiled in.
With the following table set I was able to reproduce the problem on
3.0-rc2. Replaced my IP with xxx.xxx.xxx.xxx, but otherwise unmodified
root@srv:~# iptables-save
# Generated by iptables-save v1.4.10 on Tue Jun 7 22:11:30 2011
*filter
:INPUT ACCEPT [978:107619]
:FORWARD ACCEPT [142:7068]
:OUTPUT ACCEPT [1659:291870]
-A INPUT -i ppp0 -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT ! -i ppp0 -m state --state NEW -j ACCEPT
-A INPUT -i ppp0 -j DROP
COMMIT
# Completed on Tue Jun 7 22:11:30 2011
# Generated by iptables-save v1.4.10 on Tue Jun 7 22:11:30 2011
*nat
:PREROUTING ACCEPT [813:49170]
:INPUT ACCEPT [91:7090]
:OUTPUT ACCEPT [267:20731]
:POSTROUTING ACCEPT [296:22281]
-A PREROUTING -d xxx.xxx.xxx.xxx/32 ! -i ppp0 -p tcp -m tcp --dport 443
-j DNAT --to-destination 192.168.253.198
COMMIT
# Completed on Tue Jun 7 22:11:30 2011
# Generated by iptables-save v1.4.10 on Tue Jun 7 22:11:30 2011
*mangle
:PREROUTING ACCEPT [2729:274392]
:INPUT ACCEPT [2508:262976]
:FORWARD ACCEPT [142:7068]
:OUTPUT ACCEPT [1674:293701]
:POSTROUTING ACCEPT [2131:346411]
-A FORWARD -o ppp0 -p tcp -m tcp --tcp-flags SYN,RST SYN -m tcpmss --mss
1400:1536 -j TCPMSS --clamp-mss-to-pmtu
COMMIT
# Completed on Tue Jun 7 22:11:30 2011
I've just compiled out CONFIG_BRIDGE_NETFILTER and can no longer access
the address the way I was doing it, so that's a no-go for me.
^ permalink raw reply [flat|nested] 111+ messages in thread
* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
@ 2011-06-07 14:40 ` Brad Campbell
0 siblings, 0 replies; 111+ messages in thread
From: Brad Campbell @ 2011-06-07 14:40 UTC (permalink / raw)
To: Patrick McHardy
Cc: Bart De Schuymer, kvm, linux-mm, linux-kernel, netdev, netfilter-devel
On 07/06/11 21:30, Patrick McHardy wrote:
> On 07.06.2011 05:33, Brad Campbell wrote:
>> On 07/06/11 04:10, Bart De Schuymer wrote:
>>> Hi Brad,
>>>
>>> This has probably nothing to do with ebtables, so please rmmod in case
>>> it's loaded.
>>> A few questions I didn't directly see an answer to in the threads I
>>> scanned...
>>> I'm assuming you actually use the bridging firewall functionality. So,
>>> what iptables modules do you use? Can you reduce your iptables rules to
>>> a core that triggers the bug?
>>> Or does it get triggered even with an empty set of firewall rules?
>>> Are you using a stock .35 kernel or is it patched?
>>> Is this something I can trigger on a poor guy's laptop or does it
>>> require specialized hardware (I'm catching up on qemu/kvm...)?
>>
>> Not specialised hardware as such, I've just not been able to reproduce
>> it outside of this specific operating scenario.
>
> The last similar problem we've had was related to the 32/64 bit compat
> code. Are you running 32 bit userspace on a 64 bit kernel?
No, 32 bit Guest OS, but a completely 64 bit userspace on a 64 bit kernel.
Userspace is current Debian Stable. Kernel is Vanilla and qemu-kvm is
current git
>> I can't trigger it with empty firewall rules as it relies on a DNAT to
>> occur. If I try it directly to the internal IP address (as I have to
>> without netfilter loaded) then of course nothing fails.
>>
>> It's a pain in the bum as a fault, but it's one I can easily reproduce
>> as long as I use the same set of circumstances.
>>
>> I'll try using 3.0-rc2 (current git) tonight, and if I can reproduce it
>> on that then I'll attempt to pare down the IPTABLES rules to a bare
>> minimum.
>>
>> It is nothing to do with ebtables as I don't compile it. I'm not really
>> sure about "bridging firewall" functionality. I just use a couple of
>> hand coded bash scripts to set the tables up.
>
> From one of your previous mails:
>
>> # CONFIG_BRIDGE_NF_EBTABLES is not set
>
> How about CONFIG_BRIDGE_NETFILTER?
>
It was compiled in.
With the following table set I was able to reproduce the problem on
3.0-rc2. Replaced my IP with xxx.xxx.xxx.xxx, but otherwise unmodified
root@srv:~# iptables-save
# Generated by iptables-save v1.4.10 on Tue Jun 7 22:11:30 2011
*filter
:INPUT ACCEPT [978:107619]
:FORWARD ACCEPT [142:7068]
:OUTPUT ACCEPT [1659:291870]
-A INPUT -i ppp0 -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT ! -i ppp0 -m state --state NEW -j ACCEPT
-A INPUT -i ppp0 -j DROP
COMMIT
# Completed on Tue Jun 7 22:11:30 2011
# Generated by iptables-save v1.4.10 on Tue Jun 7 22:11:30 2011
*nat
:PREROUTING ACCEPT [813:49170]
:INPUT ACCEPT [91:7090]
:OUTPUT ACCEPT [267:20731]
:POSTROUTING ACCEPT [296:22281]
-A PREROUTING -d xxx.xxx.xxx.xxx/32 ! -i ppp0 -p tcp -m tcp --dport 443
-j DNAT --to-destination 192.168.253.198
COMMIT
# Completed on Tue Jun 7 22:11:30 2011
# Generated by iptables-save v1.4.10 on Tue Jun 7 22:11:30 2011
*mangle
:PREROUTING ACCEPT [2729:274392]
:INPUT ACCEPT [2508:262976]
:FORWARD ACCEPT [142:7068]
:OUTPUT ACCEPT [1674:293701]
:POSTROUTING ACCEPT [2131:346411]
-A FORWARD -o ppp0 -p tcp -m tcp --tcp-flags SYN,RST SYN -m tcpmss --mss
1400:1536 -j TCPMSS --clamp-mss-to-pmtu
COMMIT
# Completed on Tue Jun 7 22:11:30 2011
I've just compiled out CONFIG_BRIDGE_NETFILTER and can no longer access
the address the way I was doing it, so that's a no-go for me.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 111+ messages in thread
* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
2011-06-07 13:37 ` Eric Dumazet
@ 2011-06-07 15:15 ` Brad Campbell
2011-08-20 13:16 ` Brad Campbell
1 sibling, 0 replies; 111+ messages in thread
From: Brad Campbell @ 2011-06-07 15:15 UTC (permalink / raw)
To: Eric Dumazet; +Cc: Avi Kivity, CaT, Borislav Petkov, linux-kernel, kvm, netdev
On 07/06/11 21:37, Eric Dumazet wrote:
> Le mardi 07 juin 2011 à 21:27 +0800, Brad Campbell a écrit :
>> On 07/06/11 04:22, Eric Dumazet wrote:
>>
>>> Could you please try latest linux-2.6 tree ?
>>>
>>> We fixed many networking bugs that could explain your crash.
>>>
>>>
>>>
>>>
>>
>> No good I'm afraid.
>>
>> [ 543.040056]
>> =============================================================================
>> [ 543.040136] BUG ip_dst_cache: Padding overwritten.
>> 0xffff8803e4217ffe-0xffff8803e4217fff
>> [ 543.040194]
>
> Thats pretty strange : These are the last two bytes of a page, set to
> 0x0000 (a 16 bit value)
>
> There is no way a dst field could actually sit on this location (its a
> padding), since a dst is a bit less than 256 bytes (0xe8), and each
> entry is aligned on a 64byte address.
>
> grep dst /proc/slabinfo
>
> ip_dst_cache 32823 62944 256 32 2 : tunables 0 0
> 0 : slabdata 1967 1967 0
>
> sizeof(struct rtable)=0xe8
>
>
>> -----------------------------------------------------------------------------
>> [ 543.040198]
>> [ 543.040298] INFO: Slab 0xffffea000d9e74d0 objects=25 used=25 fp=0x
>> (null) flags=0x8000000000004081
>> [ 543.040364] Pid: 4576, comm: kworker/1:2 Not tainted 3.0.0-rc2 #1
>> [ 543.040415] Call Trace:
>> [ 543.040472] [<ffffffff810b9c1d>] ? slab_err+0xad/0xd0
>> [ 543.040528] [<ffffffff8102e034>] ? check_preempt_wakeup+0xa4/0x160
>> [ 543.040595] [<ffffffff810ba206>] ? slab_pad_check+0x126/0x170
>> [ 543.040650] [<ffffffff8133045b>] ? dst_destroy+0x8b/0x110
>> [ 543.040701] [<ffffffff810ba29a>] ? check_slab+0x4a/0xc0
>> [ 543.040753] [<ffffffff810baf2d>] ? free_debug_processing+0x2d/0x250
>> [ 543.040808] [<ffffffff810bb27b>] ? __slab_free+0x12b/0x140
>> [ 543.040862] [<ffffffff810bbe99>] ? kmem_cache_free+0x99/0xa0
>> [ 543.040915] [<ffffffff8133045b>] ? dst_destroy+0x8b/0x110
>> [ 543.040967] [<ffffffff813307f6>] ? dst_gc_task+0x196/0x1f0
>> [ 543.041021] [<ffffffff8104e954>] ? queue_delayed_work_on+0x154/0x160
>> [ 543.041081] [<ffffffff813066fe>] ? do_dbs_timer+0x20e/0x3d0
>> [ 543.041133] [<ffffffff81330660>] ? dst_alloc+0x180/0x180
>> [ 543.041187] [<ffffffff8104f28b>] ? process_one_work+0xfb/0x3b0
>> [ 543.041242] [<ffffffff8104f964>] ? worker_thread+0x144/0x3d0
>> [ 543.041296] [<ffffffff8102cc10>] ? __wake_up_common+0x50/0x80
>> [ 543.041678] [<ffffffff8104f820>] ? rescuer_thread+0x2e0/0x2e0
>> [ 543.041729] [<ffffffff8104f820>] ? rescuer_thread+0x2e0/0x2e0
>> [ 543.041782] [<ffffffff81053436>] ? kthread+0x96/0xa0
>> [ 543.041835] [<ffffffff813e1d14>] ? kernel_thread_helper+0x4/0x10
>> [ 543.041890] [<ffffffff810533a0>] ? kthread_worker_fn+0x120/0x120
>> [ 543.041944] [<ffffffff813e1d10>] ? gs_change+0xb/0xb
>> [ 543.041993] Padding 0xffff8803e4217f40: 5a 5a 5a 5a 5a 5a 5a 5a 5a
>> 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
>> [ 543.042718] Padding 0xffff8803e4217f50: 5a 5a 5a 5a 5a 5a 5a 5a 5a
>> 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
>> [ 543.043433] Padding 0xffff8803e4217f60: 5a 5a 5a 5a 5a 5a 5a 5a 5a
>> 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
>> [ 543.044155] Padding 0xffff8803e4217f70: 5a 5a 5a 5a 5a 5a 5a 5a 5a
>> 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
>> [ 543.044866] Padding 0xffff8803e4217f80: 5a 5a 5a 5a 5a 5a 5a 5a 5a
>> 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
>> [ 543.045590] Padding 0xffff8803e4217f90: 5a 5a 5a 5a 5a 5a 5a 5a 5a
>> 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
>> [ 543.046311] Padding 0xffff8803e4217fa0: 5a 5a 5a 5a 5a 5a 5a 5a 5a
>> 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
>> [ 543.047034] Padding 0xffff8803e4217fb0: 5a 5a 5a 5a 5a 5a 5a 5a 5a
>> 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
>> [ 543.047755] Padding 0xffff8803e4217fc0: 5a 5a 5a 5a 5a 5a 5a 5a 5a
>> 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
>> [ 543.048474] Padding 0xffff8803e4217fd0: 5a 5a 5a 5a 5a 5a 5a 5a 5a
>> 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
>> [ 543.049203] Padding 0xffff8803e4217fe0: 5a 5a 5a 5a 5a 5a 5a 5a 5a
>> 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
>> [ 543.049909] Padding 0xffff8803e4217ff0: 5a 5a 5a 5a 5a 5a 5a 5a 5a
>> 5a 5a 5a 5a 5a 00 00 ZZZZZZZZZZZZZZ..
>> [ 543.050021] FIX ip_dst_cache: Restoring
>> 0xffff8803e4217f40-0xffff8803e4217fff=0x5a
>> [ 543.050021]
>>
>> Dropped -mm, Hugh and Andrea from CC as this does not appear to be mm or
>> ksm related.
>>
>> I'll pare down the firewall and see if I can make it break easier with a
>> smaller test set.
>
> Hmm, not sure now :(
>
> Could you reproduce another bug please ?
Here is the remainder of the log from that bug that I managed to scrape
off the disk. Odd as usually the netconsole dies and so does the disk
subsystem. In this case it managed to get the log to disk.
[ 543.040056]
=============================================================================
[ 543.040136] BUG ip_dst_cache: Padding overwritten.
0xffff8803e4217ffe-0xffff8803e4217fff
[ 543.040194]
-----------------------------------------------------------------------------
[ 543.040198]
[ 543.040298] INFO: Slab 0xffffea000d9e74d0 objects=25 used=25 fp=0x
(null) flags=0x8000000000004081
[ 543.040364] Pid: 4576, comm: kworker/1:2 Not tainted 3.0.0-rc2 #1
[ 543.040415] Call Trace:
[ 543.040472] [<ffffffff810b9c1d>] ? slab_err+0xad/0xd0
[ 543.040528] [<ffffffff8102e034>] ? check_preempt_wakeup+0xa4/0x160
[ 543.040595] [<ffffffff810ba206>] ? slab_pad_check+0x126/0x170
[ 543.040650] [<ffffffff8133045b>] ? dst_destroy+0x8b/0x110
[ 543.040701] [<ffffffff810ba29a>] ? check_slab+0x4a/0xc0
[ 543.040753] [<ffffffff810baf2d>] ? free_debug_processing+0x2d/0x250
[ 543.040808] [<ffffffff810bb27b>] ? __slab_free+0x12b/0x140
[ 543.040862] [<ffffffff810bbe99>] ? kmem_cache_free+0x99/0xa0
[ 543.040915] [<ffffffff8133045b>] ? dst_destroy+0x8b/0x110
[ 543.040967] [<ffffffff813307f6>] ? dst_gc_task+0x196/0x1f0
[ 543.041021] [<ffffffff8104e954>] ? queue_delayed_work_on+0x154/0x160
[ 543.041081] [<ffffffff813066fe>] ? do_dbs_timer+0x20e/0x3d0
[ 543.041133] [<ffffffff81330660>] ? dst_alloc+0x180/0x180
[ 543.041187] [<ffffffff8104f28b>] ? process_one_work+0xfb/0x3b0
[ 543.041242] [<ffffffff8104f964>] ? worker_thread+0x144/0x3d0
[ 543.041296] [<ffffffff8102cc10>] ? __wake_up_common+0x50/0x80
[ 543.041678] [<ffffffff8104f820>] ? rescuer_thread+0x2e0/0x2e0
[ 543.041729] [<ffffffff8104f820>] ? rescuer_thread+0x2e0/0x2e0
[ 543.041782] [<ffffffff81053436>] ? kthread+0x96/0xa0
[ 543.041835] [<ffffffff813e1d14>] ? kernel_thread_helper+0x4/0x10
[ 543.041890] [<ffffffff810533a0>] ? kthread_worker_fn+0x120/0x120
[ 543.041944] [<ffffffff813e1d10>] ? gs_change+0xb/0xb
[ 543.041993] Padding 0xffff8803e4217f40: 5a 5a 5a 5a 5a 5a 5a 5a 5a
5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
[ 543.042718] Padding 0xffff8803e4217f50: 5a 5a 5a 5a 5a 5a 5a 5a 5a
5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
[ 543.043433] Padding 0xffff8803e4217f60: 5a 5a 5a 5a 5a 5a 5a 5a 5a
5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
[ 543.044155] Padding 0xffff8803e4217f70: 5a 5a 5a 5a 5a 5a 5a 5a 5a
5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
[ 543.044866] Padding 0xffff8803e4217f80: 5a 5a 5a 5a 5a 5a 5a 5a 5a
5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
[ 543.045590] Padding 0xffff8803e4217f90: 5a 5a 5a 5a 5a 5a 5a 5a 5a
5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
[ 543.046311] Padding 0xffff8803e4217fa0: 5a 5a 5a 5a 5a 5a 5a 5a 5a
5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
[ 543.047034] Padding 0xffff8803e4217fb0: 5a 5a 5a 5a 5a 5a 5a 5a 5a
5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
[ 543.047755] Padding 0xffff8803e4217fc0: 5a 5a 5a 5a 5a 5a 5a 5a 5a
5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
[ 543.048474] Padding 0xffff8803e4217fd0: 5a 5a 5a 5a 5a 5a 5a 5a 5a
5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
[ 543.049203] Padding 0xffff8803e4217fe0: 5a 5a 5a 5a 5a 5a 5a 5a 5a
5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
[ 543.049909] Padding 0xffff8803e4217ff0: 5a 5a 5a 5a 5a 5a 5a 5a 5a
5a 5a 5a 5a 5a 00 00 ZZZZZZZZZZZZZZ..
[ 543.050021] FIX ip_dst_cache: Restoring
0xffff8803e4217f40-0xffff8803e4217fff=0x5a
[ 543.050021]
[ 885.504318] br1: port 4(vnet2) entering forwarding state
[ 885.504797] device vnet2 left promiscuous mode
[ 885.504868] br1: port 4(vnet2) entering disabled state
[ 885.847186] br1: port 3(vnet1) entering forwarding state
[ 885.847667] device vnet1 left promiscuous mode
[ 885.847740] br1: port 3(vnet1) entering disabled state
[ 885.940042]
=============================================================================
[ 885.940122] BUG kmalloc-512: Object padding overwritten
[ 885.940171]
-----------------------------------------------------------------------------
[ 885.940175]
[ 885.940273] INFO: 0xffff8803dc09322e-0xffff8803dc09322f. First byte
0x0 instead of 0x5a
[ 885.940344] INFO: Allocated in qdisc_alloc+0x59/0x110 age=66496 cpu=2
pid=4948
[ 885.940404] INFO: Freed in qdisc_alloc+0x49/0x110 age=66496 cpu=2
pid=4948
[ 885.940457] INFO: Slab 0xffffea000d821f80 objects=28 used=21
fp=0xffff8803dc090b68 flags=0x8000000000004081
[ 885.940519] INFO: Object 0xffff8803dc092fe8 @offset=12264
fp=0xffff8803dc092238
[ 885.940523]
[ 885.940618] Bytes b4 0xffff8803dc092fd8: d3 e0 ff ff 00 00 00 00 5a
5a 5a 5a 5a 5a 5a 5a Óàÿÿ....ZZZZZZZZ
[ 885.941373] Object 0xffff8803dc092fe8: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 885.942120] Object 0xffff8803dc092ff8: 00 00 00 00 00 00 00 00 70
0a 34 81 ff ff ff ff ........p.4.ÿÿÿÿ
[ 885.942847] Object 0xffff8803dc093008: b0 08 34 81 ff ff ff ff 04
00 00 00 18 00 00 00 °.4.ÿÿÿÿ........
[ 885.943585] Object 0xffff8803dc093018: 60 ac 5d 81 ff ff ff ff 00
00 00 00 00 00 00 00 `¬].ÿÿÿÿ........
[ 885.944324] Object 0xffff8803dc093028: 28 30 09 dc 03 88 ff ff 28
30 09 dc 03 88 ff ff (0.Ü..ÿÿ(0.Ü..ÿÿ
[ 885.945071] Object 0xffff8803dc093038: 00 00 00 00 ff ff ff ff 00
00 00 00 00 00 00 00 ....ÿÿÿÿ........
[ 885.945794] Object 0xffff8803dc093048: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 885.946530] Object 0xffff8803dc093058: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 885.947281] Object 0xffff8803dc093068: 98 f3 f9 0e 04 88 ff ff 00
00 00 00 00 00 00 00 .óù...ÿÿ........
[ 885.948011] Object 0xffff8803dc093078: 00 00 00 00 00 00 00 00 02
00 00 00 00 00 00 00 ................
[ 885.948735] Object 0xffff8803dc093088: 88 30 09 dc 03 88 ff ff 88
30 09 dc 03 88 ff ff .0.Ü..ÿÿ.0.Ü..ÿÿ
[ 885.949483] Object 0xffff8803dc093098: 00 00 00 00 bf bf 00 00 ca
29 01 00 00 00 00 00 ....¿¿..Ê)......
[ 885.950007] Object 0xffff8803dc0930a8: de 01 00 00 00 00 00 00 00
00 00 00 00 00 00 00 Þ...............
[ 885.950007] Object 0xffff8803dc0930b8: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 885.950007] Object 0xffff8803dc0930c8: 10 cd 13 e7 03 88 ff ff a0
13 34 81 ff ff ff ff .Í.ç..ÿÿ..4.ÿÿÿÿ
[ 885.950007] Object 0xffff8803dc0930d8: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 885.950007] Object 0xffff8803dc0930e8: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 885.950007] Object 0xffff8803dc0930f8: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 885.950007] Object 0xffff8803dc093108: 08 31 09 dc 03 88 ff ff 08
31 09 dc 03 88 ff ff .1.Ü..ÿÿ.1.Ü..ÿÿ
[ 885.950007] Object 0xffff8803dc093118: 00 00 00 00 00 00 00 00 20
31 09 dc 03 88 ff ff .........1.Ü..ÿÿ
[ 885.950007] Object 0xffff8803dc093128: 20 31 09 dc 03 88 ff ff 00
00 00 00 00 00 00 00 .1.Ü..ÿÿ........
[ 885.950007] Object 0xffff8803dc093138: 38 31 09 dc 03 88 ff ff 38
31 09 dc 03 88 ff ff 81.Ü..ÿÿ81.Ü..ÿÿ
[ 885.950007] Object 0xffff8803dc093148: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 885.950007] Object 0xffff8803dc093158: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 885.950007] Object 0xffff8803dc093168: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 885.950007] Object 0xffff8803dc093178: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 885.950007] Object 0xffff8803dc093188: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 885.950007] Object 0xffff8803dc093198: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 885.950007] Object 0xffff8803dc0931a8: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 885.950007] Object 0xffff8803dc0931b8: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 885.950007] Object 0xffff8803dc0931c8: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 885.950007] Object 0xffff8803dc0931d8: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 885.950007] Redzone 0xffff8803dc0931e8: cc cc cc cc cc cc cc cc
ÌÌÌÌÌÌÌÌ
[ 885.950007] Padding 0xffff8803dc093228: 5a 5a 5a 5a 5a 5a 00 00
ZZZZZZ..
[ 885.950007] Pid: 11, comm: rcuc1 Not tainted 3.0.0-rc2 #1
[ 885.950007] Call Trace:
[ 885.950007] [<ffffffff810ba41d>] ? check_bytes_and_report+0x10d/0x150
[ 885.950007] [<ffffffff810ba509>] ? check_object+0xa9/0x260
[ 885.950007] [<ffffffff81075a49>] ? __rcu_process_callbacks+0x119/0x340
[ 885.950007] [<ffffffff810bb033>] ? free_debug_processing+0x133/0x250
[ 885.950007] [<ffffffff810bb27b>] ? __slab_free+0x12b/0x140
[ 885.950007] [<ffffffff81075a49>] ? __rcu_process_callbacks+0x119/0x340
[ 885.950007] [<ffffffff8107666b>] ? rcu_cpu_kthread+0x27b/0x3a0
[ 885.950007] [<ffffffff8102d5b9>] ? ttwu_do_wakeup+0x19/0xa0
[ 885.950007] [<ffffffff81074a80>] ? invoke_rcu_cpu_kthread+0x30/0x30
[ 885.950007] [<ffffffff810763f0>] ? rcu_check_callbacks+0x100/0x100
[ 885.950007] [<ffffffff810763f0>] ? rcu_check_callbacks+0x100/0x100
[ 885.950007] [<ffffffff81053436>] ? kthread+0x96/0xa0
[ 885.950007] [<ffffffff813e1d14>] ? kernel_thread_helper+0x4/0x10
[ 885.950007] [<ffffffff810533a0>] ? kthread_worker_fn+0x120/0x120
[ 885.950007] [<ffffffff813e1d10>] ? gs_change+0xb/0xb
[ 885.950007] FIX kmalloc-512: Restoring
0xffff8803dc09322e-0xffff8803dc09322f=0x5a
[ 885.950007]
[ 886.050119]
=============================================================================
[ 886.050198] BUG kmalloc-512: Object padding overwritten
[ 886.050247]
-----------------------------------------------------------------------------
[ 886.050251]
[ 886.050353] INFO: 0xffff8803dc091916-0xffff8803dc091917. First byte
0x0 instead of 0x5a
[ 886.050451] INFO: Allocated in kvm_ioapic_init+0x3c/0xd0 [kvm]
age=66256 cpu=0 pid=5122
[ 886.050517] INFO: Freed in __kfree_skb+0x11/0x90 age=66311 cpu=4 pid=5054
[ 886.050568] INFO: Slab 0xffffea000d821f80 objects=28 used=17
fp=0xffff8803dc091488 flags=0x8000000000004081
[ 886.050630] INFO: Object 0xffff8803dc0916d0 @offset=5840 fp=0x
(null)
[ 886.050634]
[ 886.050726] Bytes b4 0xffff8803dc0916c0: ea e4 00 00 01 00 00 00 5a
5a 5a 5a 5a 5a 5a 5a êä......ZZZZZZZZ
[ 886.051451] Object 0xffff8803dc0916d0: 00 00 c0 fe 00 00 00 00 3f
00 00 00 00 00 00 00 ..Àþ....?.......
[ 886.052183] Object 0xffff8803dc0916e0: 00 08 00 00 00 00 00 00 00
00 01 00 00 00 00 00 ................
[ 886.052898] Object 0xffff8803dc0916f0: 00 00 01 00 00 00 00 00 00
00 01 00 00 00 00 00 ................
[ 886.053613] Object 0xffff8803dc091700: 00 00 01 00 00 00 00 00 00
00 01 00 00 00 00 00 ................
[ 886.054325] Object 0xffff8803dc091710: 00 00 01 00 00 00 00 00 00
00 01 00 00 00 00 00 ................
[ 886.055032] Object 0xffff8803dc091720: 00 00 01 00 00 00 00 00 00
00 01 00 00 00 00 00 ................
[ 886.055743] Object 0xffff8803dc091730: 00 00 01 00 00 00 00 00 00
00 01 00 00 00 00 00 ................
[ 886.056462] Object 0xffff8803dc091740: 00 00 01 00 00 00 00 00 00
00 01 00 00 00 00 00 ................
[ 886.057177] Object 0xffff8803dc091750: 00 00 01 00 00 00 00 00 00
00 01 00 00 00 00 00 ................
[ 886.058210] Object 0xffff8803dc091760: 00 00 01 00 00 00 00 00 00
00 01 00 00 00 00 00 ................
[ 886.058957] Object 0xffff8803dc091770: 00 00 01 00 00 00 00 00 00
00 01 00 00 00 00 00 ................
[ 886.059667] Object 0xffff8803dc091780: 00 00 01 00 00 00 00 00 00
00 01 00 00 00 00 00 ................
[ 886.060104] Object 0xffff8803dc091790: 00 00 01 00 00 00 00 00 00
00 01 00 00 00 00 00 ................
[ 886.060104] Object 0xffff8803dc0917a0: 00 00 01 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 886.060104] Object 0xffff8803dc0917b0: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 886.060104] Object 0xffff8803dc0917c0: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 886.060104] Object 0xffff8803dc0917d0: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 886.060104] Object 0xffff8803dc0917e0: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 886.060104] Object 0xffff8803dc0917f0: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 886.060104] Object 0xffff8803dc091800: 01 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 886.060104] Object 0xffff8803dc091810: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 886.060104] Object 0xffff8803dc091820: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 886.060104] Object 0xffff8803dc091830: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 886.060104] Object 0xffff8803dc091840: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 886.060104] Object 0xffff8803dc091850: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 886.060104] Object 0xffff8803dc091860: 00 00 00 00 00 00 00 00 b0
80 10 a0 ff ff ff ff ........°...ÿÿÿÿ
[ 886.060104] Object 0xffff8803dc091870: 00 00 df f5 03 88 ff ff 00
00 00 00 00 00 00 00 ..ßõ..ÿÿ........
[ 886.060104] Object 0xffff8803dc091880: 43 43 00 00 00 00 00 00 01
00 00 00 00 00 00 00 CC..............
[ 886.060104] Object 0xffff8803dc091890: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 886.060104] Object 0xffff8803dc0918a0: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 886.060104] Object 0xffff8803dc0918b0: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 886.060104] Object 0xffff8803dc0918c0: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 886.060104] Redzone 0xffff8803dc0918d0: cc cc cc cc cc cc cc cc
ÌÌÌÌÌÌÌÌ
[ 886.060104] Padding 0xffff8803dc091910: 5a 5a 5a 5a 5a 5a 00 00
ZZZZZZ..
[ 886.060104] Pid: 7359, comm: qemu Not tainted 3.0.0-rc2 #1
[ 886.060104] Call Trace:
[ 886.060104] [<ffffffff810ba41d>] ? check_bytes_and_report+0x10d/0x150
[ 886.060104] [<ffffffff810ba509>] ? check_object+0xa9/0x260
[ 886.060104] [<ffffffffa00eb8bf>] ? kvm_arch_destroy_vm+0x2f/0x150 [kvm]
[ 886.060104] [<ffffffff810bb033>] ? free_debug_processing+0x133/0x250
[ 886.060104] [<ffffffff810bb27b>] ? __slab_free+0x12b/0x140
[ 886.060104] [<ffffffffa00eb8bf>] ? kvm_arch_destroy_vm+0x2f/0x150 [kvm]
[ 886.060104] [<ffffffffa00e2dd9>] ? kvm_put_kvm+0xe9/0x130 [kvm]
[ 886.060104] [<ffffffffa00e3763>] ? kvm_vcpu_release+0x13/0x20 [kvm]
[ 886.060104] [<ffffffff810c65fd>] ? fput+0xdd/0x270
[ 886.060104] [<ffffffff810c2e1c>] ? filp_close+0x5c/0x90
[ 886.060104] [<ffffffff8103ae1d>] ? put_files_struct+0xcd/0x100
[ 886.060104] [<ffffffff8103ccc7>] ? do_exit+0x6d7/0x7e0
[ 886.060104] [<ffffffff8103ce20>] ? do_group_exit+0x50/0xc0
[ 886.060104] [<ffffffff81048914>] ? get_signal_to_deliver+0x1a4/0x3a0
[ 886.060104] [<ffffffff8100188a>] ? do_signal+0x8a/0x7d0
[ 886.060104] [<ffffffff81049148>] ? kill_pid_info+0x38/0x60
[ 886.060104] [<ffffffff810493e4>] ? sys_kill+0x94/0x1b0
[ 886.060104] [<ffffffff8100201d>] ? do_notify_resume+0x4d/0x70
[ 886.060104] [<ffffffff810c59c4>] ? sys_pread64+0x74/0xa0
[ 886.060104] [<ffffffff813e11db>] ? int_signal+0x12/0x17
[ 886.060104] FIX kmalloc-512: Restoring
0xffff8803dc091916-0xffff8803dc091917=0x5a
[ 886.060104]
[ 926.148014] br1: port 6(vnet4) entering forwarding state
[ 926.148385] device vnet4 left promiscuous mode
[ 926.148457] br1: port 6(vnet4) entering disabled state
[ 945.659523] br1: port 2(vnet0) entering forwarding state
[ 945.660041] device vnet0 left promiscuous mode
[ 945.660116] br1: port 2(vnet0) entering disabled state
[ 1121.085267] br1: port 5(vnet3) entering forwarding state
[ 1121.085783] device vnet3 left promiscuous mode
[ 1121.085860] br1: port 5(vnet3) entering disabled state
[ 1176.307099] nfsd: last server has exited, flushing export cache
[ 1176.430113]
=============================================================================
[ 1176.430194] BUG kmalloc-512: Object padding overwritten
[ 1176.430199]
-----------------------------------------------------------------------------
[ 1176.430204]
[ 1176.430210] INFO: 0xffff8803f5df5b5e-0xffff8803f5df5b5f. First byte
0x0 instead of 0x5a
[ 1176.430225] INFO: Allocated in sock_alloc_send_pskb+0x1d0/0x320
age=93370 cpu=0 pid=5126
[ 1176.430234] INFO: Freed in __kfree_skb+0x11/0x90 age=93370 cpu=0 pid=4495
[ 1176.430240] INFO: Slab 0xffffea000ddc8d60 objects=28 used=0
fp=0xffff8803f5df76c0 flags=0x8000000000004080
[ 1176.430246] INFO: Object 0xffff8803f5df5918 @offset=6424
fp=0xffff8803f5df7908
[ 1176.430249]
[ 1176.430253] Bytes b4 0xffff8803f5df5908: ab f0 ff ff 00 00 00 00 5a
5a 5a 5a 5a 5a 5a 5a «ðÿÿ....ZZZZZZZZ
[ 1176.430270] Object 0xffff8803f5df5918: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.430285] Object 0xffff8803f5df5928: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.430300] Object 0xffff8803f5df5938: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.430314] Object 0xffff8803f5df5948: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.430328] Object 0xffff8803f5df5958: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.430343] Object 0xffff8803f5df5968: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.430357] Object 0xffff8803f5df5978: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.430371] Object 0xffff8803f5df5988: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.430386] Object 0xffff8803f5df5998: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.430400] Object 0xffff8803f5df59a8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.430414] Object 0xffff8803f5df59b8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.430429] Object 0xffff8803f5df59c8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.430443] Object 0xffff8803f5df59d8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.430457] Object 0xffff8803f5df59e8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.430471] Object 0xffff8803f5df59f8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.430485] Object 0xffff8803f5df5a08: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.430500] Object 0xffff8803f5df5a18: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.430514] Object 0xffff8803f5df5a28: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.430528] Object 0xffff8803f5df5a38: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.430542] Object 0xffff8803f5df5a48: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.430557] Object 0xffff8803f5df5a58: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.430571] Object 0xffff8803f5df5a68: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.430585] Object 0xffff8803f5df5a78: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.430599] Object 0xffff8803f5df5a88: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.430614] Object 0xffff8803f5df5a98: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.430628] Object 0xffff8803f5df5aa8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.430642] Object 0xffff8803f5df5ab8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.430656] Object 0xffff8803f5df5ac8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.430671] Object 0xffff8803f5df5ad8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.430685] Object 0xffff8803f5df5ae8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.430699] Object 0xffff8803f5df5af8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.430714] Object 0xffff8803f5df5b08: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b a5 kkkkkkkkkkkkkkk¥
[ 1176.430729] Redzone 0xffff8803f5df5b18: bb bb bb bb bb bb bb bb
»»»»»»»»
[ 1176.430742] Padding 0xffff8803f5df5b58: 5a 5a 5a 5a 5a 5a 00 00
ZZZZZZ..
[ 1176.430758] Pid: 24, comm: rcuc4 Not tainted 3.0.0-rc2 #1
[ 1176.430762] Call Trace:
[ 1176.430774] [<ffffffff810ba41d>] ? check_bytes_and_report+0x10d/0x150
[ 1176.430782] [<ffffffff810ba509>] ? check_object+0xa9/0x260
[ 1176.430789] [<ffffffff810bac15>] ? init_object+0x85/0xa0
[ 1176.430795] [<ffffffff810ba816>] ? __free_slab+0x156/0x180
[ 1176.430803] [<ffffffff81075a49>] ? __rcu_process_callbacks+0x119/0x340
[ 1176.430810] [<ffffffff8107666b>] ? rcu_cpu_kthread+0x27b/0x3a0
[ 1176.430818] [<ffffffff8102d5b9>] ? ttwu_do_wakeup+0x19/0xa0
[ 1176.430825] [<ffffffff81074a80>] ? invoke_rcu_cpu_kthread+0x30/0x30
[ 1176.430832] [<ffffffff810763f0>] ? rcu_check_callbacks+0x100/0x100
[ 1176.430838] [<ffffffff810763f0>] ? rcu_check_callbacks+0x100/0x100
[ 1176.430846] [<ffffffff81053436>] ? kthread+0x96/0xa0
[ 1176.430855] [<ffffffff813e1d14>] ? kernel_thread_helper+0x4/0x10
[ 1176.430863] [<ffffffff810533a0>] ? kthread_worker_fn+0x120/0x120
[ 1176.430870] [<ffffffff813e1d10>] ? gs_change+0xb/0xb
[ 1176.430876] FIX kmalloc-512: Restoring
0xffff8803f5df5b5e-0xffff8803f5df5b5f=0x5a
[ 1176.430879]
[ 1176.430897]
=============================================================================
[ 1176.430902] BUG kmalloc-512: Object padding overwritten
[ 1176.430905]
-----------------------------------------------------------------------------
[ 1176.430909]
[ 1176.430912] INFO: 0xffff8803f5df6fe6-0xffff8803f5df6fe7. First byte
0x0 instead of 0x5a
[ 1176.430921] INFO: Allocated in sock_alloc_send_pskb+0x1d0/0x320
age=93370 cpu=0 pid=5126
[ 1176.430928] INFO: Freed in __kfree_skb+0x11/0x90 age=93370 cpu=0 pid=4495
[ 1176.430935] INFO: Slab 0xffffea000ddc8d60 objects=28 used=0
fp=0xffff8803f5df76c0 flags=0x8000000000004080
[ 1176.430940] INFO: Object 0xffff8803f5df6da0 @offset=11680
fp=0xffff8803f5df5488
[ 1176.430943]
[ 1176.430946] Bytes b4 0xffff8803f5df6d90: cd 7a 00 00 01 00 00 00 5a
5a 5a 5a 5a 5a 5a 5a Íz......ZZZZZZZZ
[ 1176.430962] Object 0xffff8803f5df6da0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.430977] Object 0xffff8803f5df6db0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.430992] Object 0xffff8803f5df6dc0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.431006] Object 0xffff8803f5df6dd0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.431021] Object 0xffff8803f5df6de0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.431036] Object 0xffff8803f5df6df0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.431050] Object 0xffff8803f5df6e00: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.431065] Object 0xffff8803f5df6e10: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.431080] Object 0xffff8803f5df6e20: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.431094] Object 0xffff8803f5df6e30: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.431109] Object 0xffff8803f5df6e40: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.431124] Object 0xffff8803f5df6e50: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.431138] Object 0xffff8803f5df6e60: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.431153] Object 0xffff8803f5df6e70: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.431168] Object 0xffff8803f5df6e80: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.431182] Object 0xffff8803f5df6e90: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.431197] Object 0xffff8803f5df6ea0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.431213] Object 0xffff8803f5df6eb0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.431228] Object 0xffff8803f5df6ec0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.431243] Object 0xffff8803f5df6ed0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.431258] Object 0xffff8803f5df6ee0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.431273] Object 0xffff8803f5df6ef0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.431287] Object 0xffff8803f5df6f00: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.431303] Object 0xffff8803f5df6f10: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.431317] Object 0xffff8803f5df6f20: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.431332] Object 0xffff8803f5df6f30: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.431346] Object 0xffff8803f5df6f40: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.431361] Object 0xffff8803f5df6f50: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.431376] Object 0xffff8803f5df6f60: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.431391] Object 0xffff8803f5df6f70: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.431405] Object 0xffff8803f5df6f80: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.431420] Object 0xffff8803f5df6f90: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b a5 kkkkkkkkkkkkkkk¥
[ 1176.431435] Redzone 0xffff8803f5df6fa0: bb bb bb bb bb bb bb bb
»»»»»»»»
[ 1176.431448] Padding 0xffff8803f5df6fe0: 5a 5a 5a 5a 5a 5a 00 00
ZZZZZZ..
[ 1176.431463] Pid: 24, comm: rcuc4 Not tainted 3.0.0-rc2 #1
[ 1176.431467] Call Trace:
[ 1176.431474] [<ffffffff810ba41d>] ? check_bytes_and_report+0x10d/0x150
[ 1176.431481] [<ffffffff810ba509>] ? check_object+0xa9/0x260
[ 1176.431487] [<ffffffff810bac15>] ? init_object+0x85/0xa0
[ 1176.431494] [<ffffffff810ba816>] ? __free_slab+0x156/0x180
[ 1176.431501] [<ffffffff81075a49>] ? __rcu_process_callbacks+0x119/0x340
[ 1176.431508] [<ffffffff8107666b>] ? rcu_cpu_kthread+0x27b/0x3a0
[ 1176.431515] [<ffffffff8102d5b9>] ? ttwu_do_wakeup+0x19/0xa0
[ 1176.431522] [<ffffffff81074a80>] ? invoke_rcu_cpu_kthread+0x30/0x30
[ 1176.431529] [<ffffffff810763f0>] ? rcu_check_callbacks+0x100/0x100
[ 1176.431535] [<ffffffff810763f0>] ? rcu_check_callbacks+0x100/0x100
[ 1176.431543] [<ffffffff81053436>] ? kthread+0x96/0xa0
[ 1176.431550] [<ffffffff813e1d14>] ? kernel_thread_helper+0x4/0x10
[ 1176.431558] [<ffffffff810533a0>] ? kthread_worker_fn+0x120/0x120
[ 1176.431565] [<ffffffff813e1d10>] ? gs_change+0xb/0xb
[ 1176.431571] FIX kmalloc-512: Restoring
0xffff8803f5df6fe6-0xffff8803f5df6fe7=0x5a
[ 1176.431574]
[ 1179.773660]
=============================================================================
[ 1179.773742] BUG kmalloc-1024: Object padding overwritten
[ 1179.773792]
-----------------------------------------------------------------------------
[ 1179.773796]
[ 1179.773901] INFO: 0xffff8803e430e6be-0xffff8803e430e6bf. First byte
0x0 instead of 0x5a
[ 1179.773972] INFO: Allocated in tty_buffer_request_room+0xcc/0x1a0
age=93277 cpu=1 pid=5109
[ 1179.774042] INFO: Freed in __blockdev_direct_IO+0xa16/0xa90 age=93277
cpu=5 pid=5146
[ 1179.774102] INFO: Slab 0xffffea000d9ea9c0 objects=29 used=14
fp=0xffff8803e430d9e8 flags=0x8000000000004081
[ 1179.774163] INFO: Object 0xffff8803e430e278 @offset=25208
fp=0xffff8803e4309df8
[ 1179.774166]
[ 1179.774260] Bytes b4 0xffff8803e430e268: a5 57 01 00 01 00 00 00 5a
5a 5a 5a 5a 5a 5a 5a ¥W......ZZZZZZZZ
[ 1179.775297] Object 0xffff8803e430e278: 08 eb 30 e4 03 88 ff ff a0
e2 30 e4 03 88 ff ff .ë0ä..ÿÿ.â0ä..ÿÿ
[ 1179.776003] Object 0xffff8803e430e288: a0 e3 30 e4 03 88 ff ff be
00 00 00 00 01 00 00 .ã0ä..ÿÿŸ.......
[ 1179.776974] Object 0xffff8803e430e298: be 00 00 00 be 00 00 00 5b
20 20 20 20 32 2e 34 Ÿ...Ÿ...[....2.4
[ 1179.777656] Object 0xffff8803e430e2a8: 31 30 32 34 36 5d 20 70 63
69 20 30 30 30 30 3a 10246].pci.0000:
[ 1179.778609] Object 0xffff8803e430e2b8: 30 30 3a 31 34 2e 34 3a 20
20 20 62 72 69 64 67 00:14.4:...bridg
[ 1179.779289] Object 0xffff8803e430e2c8: 65 20 77 69 6e 64 6f 77 20
5b 69 6f 20 20 30 78 e.window.[io..0x
[ 1179.779976] Object 0xffff8803e430e2d8: 30 64 30 30 2d 30 78 66 66
66 66 5d 20 28 73 75 0d00-0xffff].(su
[ 1179.780327] Object 0xffff8803e430e2e8: 62 74 72 61 63 74 69 76 65
20 64 65 63 6f 64 65 btractive.decode
[ 1179.780327] Object 0xffff8803e430e2f8: 29 0d 0a 5b 20 20 20 20 32
2e 34 31 30 32 34 38 )..[....2.410248
[ 1179.780327] Object 0xffff8803e430e308: 5d 20 70 63 69 20 30 30 30
30 3a 30 30 3a 31 34 ].pci.0000:00:14
[ 1179.780327] Object 0xffff8803e430e318: 2e 34 3a 20 20 20 62 72 69
64 67 65 20 77 69 6e .4:...bridge.win
[ 1179.780327] Object 0xffff8803e430e328: 64 6f 77 20 5b 6d 65 6d 20
30 78 30 30 30 61 30 dow.[mem.0x000a0
[ 1179.780327] Object 0xffff8803e430e338: 30 30 30 2d 30 78 30 30 30
62 66 66 66 66 5d 20 000-0x000bffff].
[ 1179.780327] Object 0xffff8803e430e348: 28 73 75 62 74 72 61 63 74
69 76 65 20 64 65 63 (subtractive.dec
[ 1179.780327] Object 0xffff8803e430e358: 6f 64 65 29 0d 0a 32 30 5d
20 78 48 43 49 20 78 ode)..20].xHCI.x
[ 1179.780327] Object 0xffff8803e430e368: 68 63 69 5f 63 68 65 63 6b
5f 62 61 6e 64 77 69 hci_check_bandwi
[ 1179.780327] Object 0xffff8803e430e378: 64 74 68 20 63 61 6c 6c 65
64 20 66 6f 72 20 72 dth.called.for.r
[ 1179.780327] Object 0xffff8803e430e388: 6f 6f 74 20 68 75 62 0d 0a
0d 0a 0d 0a 0d 0a 0d oot.hub.........
[ 1179.780327] Object 0xffff8803e430e398: 0a 30 30 0d 0a 0d 0a 0a 00
00 00 00 00 00 00 00 .00.............
[ 1179.780327] Object 0xffff8803e430e3a8: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 1179.780327] Object 0xffff8803e430e3b8: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 1179.780327] Object 0xffff8803e430e3c8: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 1179.780327] Object 0xffff8803e430e3d8: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 1179.780327] Object 0xffff8803e430e3e8: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 1179.780327] Object 0xffff8803e430e3f8: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 1179.780327] Object 0xffff8803e430e408: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 1179.780327] Object 0xffff8803e430e418: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 1179.780327] Object 0xffff8803e430e428: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 1179.780327] Object 0xffff8803e430e438: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 1179.780327] Object 0xffff8803e430e448: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 1179.780327] Object 0xffff8803e430e458: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 1179.780327] Object 0xffff8803e430e468: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 1179.780327] Object 0xffff8803e430e478: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 1179.780327] Object 0xffff8803e430e488: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 1179.780327] Object 0xffff8803e430e498: 00 00 00 00 00 00 00 00 6b
6b 6b 6b 6b 6b 6b 6b ........kkkkkkkk
[ 1179.780327] Object 0xffff8803e430e4a8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1179.780327] Object 0xffff8803e430e4b8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1179.780327] Object 0xffff8803e430e4c8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1179.780327] Object 0xffff8803e430e4d8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1179.780327] Object 0xffff8803e430e4e8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1179.780327] Object 0xffff8803e430e4f8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1179.780327] Object 0xffff8803e430e508: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1179.780327] Object 0xffff8803e430e518: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1179.780327] Object 0xffff8803e430e528: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1179.780327] Object 0xffff8803e430e538: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1179.780327] Object 0xffff8803e430e548: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1179.780327] Object 0xffff8803e430e558: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1179.780327] Object 0xffff8803e430e568: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1179.780327] Object 0xffff8803e430e578: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1179.780327] Object 0xffff8803e430e588: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1179.780327] Object 0xffff8803e430e598: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1179.780327] Object 0xffff8803e430e5a8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1179.780327] Object 0xffff8803e430e5b8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1179.780327] Object 0xffff8803e430e5c8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1179.780327] Object 0xffff8803e430e5d8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1179.780327] Object 0xffff8803e430e5e8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1179.780327] Object 0xffff8803e430e5f8: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1179.780327] Object 0xffff8803e430e608: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1179.780327] Object 0xffff8803e430e618: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1179.780327] Object 0xffff8803e430e628: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1179.780327] Object 0xffff8803e430e638: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1179.780327] Object 0xffff8803e430e648: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1179.780327] Object 0xffff8803e430e658: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1179.780327] Object 0xffff8803e430e668: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b a5 kkkkkkkkkkkkkkk¥
[ 1179.780327] Redzone 0xffff8803e430e678: cc cc cc cc cc cc cc cc
ÌÌÌÌÌÌÌÌ
[ 1179.780327] Padding 0xffff8803e430e6b8: 5a 5a 5a 5a 5a 5a 00 00
ZZZZZZ..
[ 1179.780327] Pid: 2212, comm: kworker/5:2 Not tainted 3.0.0-rc2 #1
[ 1179.780327] Call Trace:
[ 1179.780327] [<ffffffff810ba41d>] ? check_bytes_and_report+0x10d/0x150
[ 1179.780327] [<ffffffff810ba509>] ? check_object+0xa9/0x260
[ 1179.780327] [<ffffffff8125acf7>] ? tty_buffer_free_all+0x47/0x70
[ 1179.780327] [<ffffffff810bb033>] ? free_debug_processing+0x133/0x250
[ 1179.780327] [<ffffffff810bb27b>] ? __slab_free+0x12b/0x140
[ 1179.780327] [<ffffffff81254c70>] ? free_tty_struct+0x30/0x30
[ 1179.780327] [<ffffffff8125acf7>] ? tty_buffer_free_all+0x47/0x70
[ 1179.780327] [<ffffffff81254c66>] ? free_tty_struct+0x26/0x30
[ 1179.780327] [<ffffffff8104f28b>] ? process_one_work+0xfb/0x3b0
[ 1179.780327] [<ffffffff8104f964>] ? worker_thread+0x144/0x3d0
[ 1179.780327] [<ffffffff8102cc10>] ? __wake_up_common+0x50/0x80
[ 1179.780327] [<ffffffff8104f820>] ? rescuer_thread+0x2e0/0x2e0
[ 1179.780327] [<ffffffff8104f820>] ? rescuer_thread+0x2e0/0x2e0
[ 1179.780327] [<ffffffff81053436>] ? kthread+0x96/0xa0
[ 1179.780327] [<ffffffff813e1d14>] ? kernel_thread_helper+0x4/0x10
[ 1179.780327] [<ffffffff810533a0>] ? kthread_worker_fn+0x120/0x120
[ 1179.780327] [<ffffffff813e1d10>] ? gs_change+0xb/0xb
[ 1179.780327] FIX kmalloc-1024: Restoring
0xffff8803e430e6be-0xffff8803e430e6bf=0x5a
[ 1179.780327]
[ 1180.826212]
=============================================================================
[ 1180.826289] BUG kmalloc-1024: Object padding overwritten
[ 1180.826338]
-----------------------------------------------------------------------------
[ 1180.826341]
[ 1180.826439] INFO: 0xffff8804097a9566-0xffff8804097a9567. First byte
0x0 instead of 0x5a
[ 1180.826505] INFO: Allocated in tty_buffer_request_room+0xcc/0x1a0
age=69291 cpu=0 pid=5978
[ 1180.826567] INFO: Freed in __blockdev_direct_IO+0xa16/0xa90 age=80371
cpu=5 pid=5713
[ 1180.826626] INFO: Slab 0xffffea000e212cc0 objects=29 used=3
fp=0xffff8804097a8890 flags=0x8000000000004081
[ 1180.826686] INFO: Object 0xffff8804097a9120 @offset=4384
fp=0xffff8804097a8890
[ 1180.826689]
[ 1180.826781] Bytes b4 0xffff8804097a9110: 91 48 01 00 01 00 00 00 5a
5a 5a 5a 5a 5a 5a 5a .H......ZZZZZZZZ
[ 1180.827478] Object 0xffff8804097a9120: 30 de 70 fc 03 88 ff ff 48
91 7a 09 04 88 ff ff 0Þpü..ÿÿH.z...ÿÿ
[ 1180.828179] Object 0xffff8804097a9130: 48 92 7a 09 04 88 ff ff ce
00 00 00 00 01 00 00 H.z...ÿÿÎ.......
[ 1180.829149] Object 0xffff8804097a9140: ce 00 00 00 ce 00 00 00 5b
20 20 20 20 32 2e 33 Î...Î...[....2.3
[ 1180.830162] Object 0xffff8804097a9150: 32 31 31 35 36 5d 20 70 63
69 20 30 30 30 30 3a 21156].pci.0000:
[ 1180.831118] Object 0xffff8804097a9160: 30 30 3a 31 33 2e 30 3a 20
5b 31 30 30 32 3a 34 00:13.0:.[1002:4
[ 1180.831796] Object 0xffff8804097a9170: 33 39 37 5d 20 74 79 70 65
20 30 20 63 6c 61 73 397].type.0.clas
[ 1180.832492] Object 0xffff8804097a9180: 73 20 30 78 30 30 30 63 30
33 0d 0a 5b 20 20 20 s.0x000c03..[...
[ 1180.833482] Object 0xffff8804097a9190: 20 32 2e 33 32 31 31 36 36
5d 20 70 63 69 20 30 .2.321166].pci.0
[ 1180.834173] Object 0xffff8804097a91a0: 30 30 30 3a 30 30 3a 31 33
2e 30 3a 20 72 65 67 000:00:13.0:.reg
[ 1180.835134] Object 0xffff8804097a91b0: 20 31 30 3a 20 5b 6d 65 6d
20 30 78 66 65 31 66 .10:.[mem.0xfe1f
[ 1180.835240] Object 0xffff8804097a91c0: 64 30 30 30 2d 30 78 66 65
31 66 64 66 66 66 5d d000-0xfe1fdfff]
[ 1180.835240] Object 0xffff8804097a91d0: 0d 0a 5b 20 20 20 20 32 2e
33 32 31 32 31 38 5d ..[....2.321218]
[ 1180.835240] Object 0xffff8804097a91e0: 20 70 63 69 20 30 30 30 30
3a 30 30 3a 31 33 2e .pci.0000:00:13.
[ 1180.835240] Object 0xffff8804097a91f0: 32 3a 20 5b 31 30 30 32 3a
34 33 39 36 5d 20 74 2:.[1002:4396].t
[ 1180.835240] Object 0xffff8804097a9200: 79 70 65 20 30 20 63 6c 61
73 73 20 30 78 30 30 ype.0.class.0x00
[ 1180.835240] Object 0xffff8804097a9210: 30 63 30 33 0d 0a 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b 0c03..kkkkkkkkkk
[ 1180.835240] Object 0xffff8804097a9220: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.835240] Object 0xffff8804097a9230: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.835240] Object 0xffff8804097a9240: 6b 6b 6b 6b 6b 6b 6b 6b 00
00 00 00 00 00 00 00 kkkkkkkk........
[ 1180.835240] Object 0xffff8804097a9250: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 1180.835240] Object 0xffff8804097a9260: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 1180.835240] Object 0xffff8804097a9270: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 1180.835240] Object 0xffff8804097a9280: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 1180.835240] Object 0xffff8804097a9290: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 1180.835240] Object 0xffff8804097a92a0: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 1180.835240] Object 0xffff8804097a92b0: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 1180.835240] Object 0xffff8804097a92c0: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 1180.835240] Object 0xffff8804097a92d0: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 1180.835240] Object 0xffff8804097a92e0: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 1180.835240] Object 0xffff8804097a92f0: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 1180.835240] Object 0xffff8804097a9300: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 1180.835240] Object 0xffff8804097a9310: 00 00 00 00 00 00 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b ......kkkkkkkkkk
[ 1180.835240] Object 0xffff8804097a9320: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.835240] Object 0xffff8804097a9330: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.835240] Object 0xffff8804097a9340: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.835240] Object 0xffff8804097a9350: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.835240] Object 0xffff8804097a9360: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.835240] Object 0xffff8804097a9370: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.835240] Object 0xffff8804097a9380: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.835240] Object 0xffff8804097a9390: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.835240] Object 0xffff8804097a93a0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.835240] Object 0xffff8804097a93b0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.835240] Object 0xffff8804097a93c0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.835240] Object 0xffff8804097a93d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.835240] Object 0xffff8804097a93e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.835240] Object 0xffff8804097a93f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.835240] Object 0xffff8804097a9400: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.835240] Object 0xffff8804097a9410: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.835240] Object 0xffff8804097a9420: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.835240] Object 0xffff8804097a9430: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.835240] Object 0xffff8804097a9440: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.835240] Object 0xffff8804097a9450: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.835240] Object 0xffff8804097a9460: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.835240] Object 0xffff8804097a9470: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.835240] Object 0xffff8804097a9480: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.835240] Object 0xffff8804097a9490: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.835240] Object 0xffff8804097a94a0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.835240] Object 0xffff8804097a94b0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.835240] Object 0xffff8804097a94c0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.835240] Object 0xffff8804097a94d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.835240] Object 0xffff8804097a94e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.835240] Object 0xffff8804097a94f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.835240] Object 0xffff8804097a9500: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.835240] Object 0xffff8804097a9510: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b a5 kkkkkkkkkkkkkkk¥
[ 1180.835240] Redzone 0xffff8804097a9520: cc cc cc cc cc cc cc cc
ÌÌÌÌÌÌÌÌ
[ 1180.835240] Padding 0xffff8804097a9560: 5a 5a 5a 5a 5a 5a 00 00
ZZZZZZ..
[ 1180.835240] Pid: 8295, comm: kworker/5:0 Not tainted 3.0.0-rc2 #1
[ 1180.835240] Call Trace:
[ 1180.835240] [<ffffffff810ba41d>] ? check_bytes_and_report+0x10d/0x150
[ 1180.835240] [<ffffffff810ba509>] ? check_object+0xa9/0x260
[ 1180.835240] [<ffffffff8125acf7>] ? tty_buffer_free_all+0x47/0x70
[ 1180.835240] [<ffffffff810bb033>] ? free_debug_processing+0x133/0x250
[ 1180.835240] [<ffffffff810bb27b>] ? __slab_free+0x12b/0x140
[ 1180.835240] [<ffffffff81254c70>] ? free_tty_struct+0x30/0x30
[ 1180.835240] [<ffffffff8125acf7>] ? tty_buffer_free_all+0x47/0x70
[ 1180.835240] [<ffffffff81254c66>] ? free_tty_struct+0x26/0x30
[ 1180.835240] [<ffffffff8104f28b>] ? process_one_work+0xfb/0x3b0
[ 1180.835240] [<ffffffff8104f964>] ? worker_thread+0x144/0x3d0
[ 1180.835240] [<ffffffff8102cc10>] ? __wake_up_common+0x50/0x80
[ 1180.835240] [<ffffffff8104f820>] ? rescuer_thread+0x2e0/0x2e0
[ 1180.835240] [<ffffffff8104f820>] ? rescuer_thread+0x2e0/0x2e0
[ 1180.835240] [<ffffffff81053436>] ? kthread+0x96/0xa0
[ 1180.835240] [<ffffffff813e1d14>] ? kernel_thread_helper+0x4/0x10
[ 1180.835240] [<ffffffff810533a0>] ? kthread_worker_fn+0x120/0x120
[ 1180.835240] [<ffffffff813e1d10>] ? gs_change+0xb/0xb
[ 1180.835240] FIX kmalloc-1024: Restoring
0xffff8804097a9566-0xffff8804097a9567=0x5a
[ 1180.835240]
[ 1180.881652]
=============================================================================
[ 1180.881713] BUG kmalloc-1024: Object padding overwritten
[ 1180.881760]
-----------------------------------------------------------------------------
[ 1180.881763]
[ 1180.881859] INFO: 0xffff8803fc70eb06-0xffff8803fc70eb07. First byte
0x0 instead of 0x5a
[ 1180.881920] INFO: Allocated in tty_buffer_request_room+0xcc/0x1a0
age=69297 cpu=0 pid=5978
[ 1180.881981] INFO: Freed in __blockdev_direct_IO+0xa16/0xa90 age=69384
cpu=5 pid=5146
[ 1180.882040] INFO: Slab 0xffffea000df389c0 objects=29 used=3
fp=0xffff8803fc70de30 flags=0x8000000000004081
[ 1180.882099] INFO: Object 0xffff8803fc70e6c0 @offset=26304
fp=0xffff8803fc70de30
[ 1180.882102]
[ 1180.882192] Bytes b4 0xffff8803fc70e6b0: aa 40 01 00 01 00 00 00 5a
5a 5a 5a 5a 5a 5a 5a ª@......ZZZZZZZZ
[ 1180.882884] Object 0xffff8803fc70e6c0: 68 95 6c fc 03 88 ff ff e8
e6 70 fc 03 88 ff ff h.lü..ÿÿèæpü..ÿÿ
[ 1180.883583] Object 0xffff8803fc70e6d0: e8 e7 70 fc 03 88 ff ff 00
01 00 00 00 01 00 00 èçpü..ÿÿ........
[ 1180.884542] Object 0xffff8803fc70e6e0: 00 01 00 00 00 01 00 00 5b
20 20 20 20 32 2e 33 ........[....2.3
[ 1180.885223] Object 0xffff8803fc70e6f0: 32 30 38 31 38 5d 20 70 63
69 20 30 30 30 30 3a 20818].pci.0000:
[ 1180.886169] Object 0xffff8803fc70e700: 30 30 3a 30 34 2e 30 3a 20
50 4d 45 23 20 73 75 00:04.0:.PME#.su
[ 1180.886850] Object 0xffff8803fc70e710: 70 70 6f 72 74 65 64 20 66
72 6f 6d 20 44 30 20 pported.from.D0.
[ 1180.887535] Object 0xffff8803fc70e720: 44 33 68 6f 74 20 44 33 63
6f 6c 64 0d 0a 5b 20 D3hot.D3cold..[.
[ 1180.888848] Object 0xffff8803fc70e730: 20 20 20 32 2e 33 32 30 38
32 30 5d 20 70 63 69 ...2.320820].pci
[ 1180.889537] Object 0xffff8803fc70e740: 20 30 30 30 30 3a 30 30 3a
30 34 2e 30 3a 20 50 .0000:00:04.0:.P
[ 1180.890481] Object 0xffff8803fc70e750: 4d 45 23 20 64 69 73 61 62
6c 65 64 0d 0a 5b 20 ME#.disabled..[.
[ 1180.891161] Object 0xffff8803fc70e760: 20 20 20 32 2e 33 32 30 38
33 36 5d 20 70 63 69 ...2.320836].pci
[ 1180.891365] Object 0xffff8803fc70e770: 20 30 30 30 30 3a 30 30 3a
30 39 2e 30 3a 20 5b .0000:00:09.0:.[
[ 1180.891365] Object 0xffff8803fc70e780: 31 30 32 32 3a 39 36 30 38
5d 20 74 79 70 65 20 1022:9608].type.
[ 1180.891365] Object 0xffff8803fc70e790: 31 20 63 6c 61 73 73 20 30
78 30 30 30 36 30 34 1.class.0x000604
[ 1180.891365] Object 0xffff8803fc70e7a0: 0d 0a 5b 20 20 20 20 32 2e
33 32 30 38 35 35 5d ..[....2.320855]
[ 1180.891365] Object 0xffff8803fc70e7b0: 20 70 63 69 20 30 30 30 30
3a 30 30 3a 30 39 2e .pci.0000:00:09.
[ 1180.891365] Object 0xffff8803fc70e7c0: 30 3a 20 50 4d 45 23 20 73
75 70 70 6f 72 74 65 0:.PME#.supporte
[ 1180.891365] Object 0xffff8803fc70e7d0: 64 20 66 72 6f 6d 20 44 30
20 44 33 68 6f 74 20 d.from.D0.D3hot.
[ 1180.891365] Object 0xffff8803fc70e7e0: 44 33 63 6f 6c 64 0d 0a 00
00 00 00 00 00 00 00 D3cold..........
[ 1180.891365] Object 0xffff8803fc70e7f0: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 1180.891365] Object 0xffff8803fc70e800: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 1180.891365] Object 0xffff8803fc70e810: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 1180.891365] Object 0xffff8803fc70e820: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 1180.891365] Object 0xffff8803fc70e830: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 1180.891365] Object 0xffff8803fc70e840: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 1180.891365] Object 0xffff8803fc70e850: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 1180.891365] Object 0xffff8803fc70e860: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 1180.891365] Object 0xffff8803fc70e870: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 1180.891365] Object 0xffff8803fc70e880: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 1180.891365] Object 0xffff8803fc70e890: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 1180.891365] Object 0xffff8803fc70e8a0: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 1180.891365] Object 0xffff8803fc70e8b0: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 1180.891365] Object 0xffff8803fc70e8c0: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 1180.891365] Object 0xffff8803fc70e8d0: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 ................
[ 1180.891365] Object 0xffff8803fc70e8e0: 00 00 00 00 00 00 00 00 6b
6b 6b 6b 6b 6b 6b 6b ........kkkkkkkk
[ 1180.891365] Object 0xffff8803fc70e8f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.891365] Object 0xffff8803fc70e900: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.891365] Object 0xffff8803fc70e910: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.891365] Object 0xffff8803fc70e920: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.891365] Object 0xffff8803fc70e930: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.891365] Object 0xffff8803fc70e940: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.891365] Object 0xffff8803fc70e950: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.891365] Object 0xffff8803fc70e960: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.891365] Object 0xffff8803fc70e970: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.891365] Object 0xffff8803fc70e980: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.891365] Object 0xffff8803fc70e990: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.891365] Object 0xffff8803fc70e9a0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.891365] Object 0xffff8803fc70e9b0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.891365] Object 0xffff8803fc70e9c0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.891365] Object 0xffff8803fc70e9d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.891365] Object 0xffff8803fc70e9e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.891365] Object 0xffff8803fc70e9f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.891365] Object 0xffff8803fc70ea00: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.891365] Object 0xffff8803fc70ea10: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.891365] Object 0xffff8803fc70ea20: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.891365] Object 0xffff8803fc70ea30: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.891365] Object 0xffff8803fc70ea40: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.891365] Object 0xffff8803fc70ea50: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.891365] Object 0xffff8803fc70ea60: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.891365] Object 0xffff8803fc70ea70: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.891365] Object 0xffff8803fc70ea80: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.891365] Object 0xffff8803fc70ea90: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.891365] Object 0xffff8803fc70eaa0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.891365] Object 0xffff8803fc70eab0: 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b 6b 6b 6b 6b 6b a5 kkkkkkkkkkkkkkk¥
[ 1180.891365] Redzone 0xffff8803fc70eac0: cc cc cc cc cc cc cc cc
ÌÌÌÌÌÌÌÌ
[ 1180.891365] Padding 0xffff8803fc70eb00: 5a 5a 5a 5a 5a 5a 00 00
ZZZZZZ..
[ 1180.891365] Pid: 8295, comm: kworker/5:0 Not tainted 3.0.0-rc2 #1
[ 1180.891365] Call Trace:
[ 1180.891365] [<ffffffff810ba41d>] ? check_bytes_and_report+0x10d/0x150
[ 1180.891365] [<ffffffff810ba509>] ? check_object+0xa9/0x260
[ 1180.891365] [<ffffffff8125acf7>] ? tty_buffer_free_all+0x47/0x70
[ 1180.891365] [<ffffffff810bb033>] ? free_debug_processing+0x133/0x250
[ 1180.891365] [<ffffffff810bb27b>] ? __slab_free+0x12b/0x140
[ 1180.891365] [<ffffffff81254c70>] ? free_tty_struct+0x30/0x30
[ 1180.891365] [<ffffffff8125acf7>] ? tty_buffer_free_all+0x47/0x70
[ 1180.891365] [<ffffffff81254c66>] ? free_tty_struct+0x26/0x30
[ 1180.891365] [<ffffffff8104f28b>] ? process_one_work+0xfb/0x3b0
[ 1180.891365] [<ffffffff8104f964>] ? worker_thread+0x144/0x3d0
[ 1180.891365] [<ffffffff8102cc10>] ? __wake_up_common+0x50/0x80
[ 1180.891365] [<ffffffff8104f820>] ? rescuer_thread+0x2e0/0x2e0
[ 1180.891365] [<ffffffff8104f820>] ? rescuer_thread+0x2e0/0x2e0
[ 1180.891365] [<ffffffff81053436>] ? kthread+0x96/0xa0
[ 1180.891365] [<ffffffff813e1d14>] ? kernel_thread_helper+0x4/0x10
[ 1180.891365] [<ffffffff810533a0>] ? kthread_worker_fn+0x120/0x120
[ 1180.891365] [<ffffffff813e1d10>] ? gs_change+0xb/0xb
[ 1180.891365] FIX kmalloc-1024: Restoring
0xffff8803fc70eb06-0xffff8803fc70eb07=0x5a
[ 1180.891365]
^ permalink raw reply [flat|nested] 111+ messages in thread
* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
2011-06-07 14:40 ` Brad Campbell
@ 2011-06-07 15:35 ` Patrick McHardy
-1 siblings, 0 replies; 111+ messages in thread
From: Patrick McHardy @ 2011-06-07 15:35 UTC (permalink / raw)
To: Brad Campbell
Cc: Bart De Schuymer, kvm, linux-mm, linux-kernel, netdev, netfilter-devel
On 07.06.2011 16:40, Brad Campbell wrote:
> On 07/06/11 21:30, Patrick McHardy wrote:
>> On 07.06.2011 05:33, Brad Campbell wrote:
>>> On 07/06/11 04:10, Bart De Schuymer wrote:
>>>> Hi Brad,
>>>>
>>>> This has probably nothing to do with ebtables, so please rmmod in case
>>>> it's loaded.
>>>> A few questions I didn't directly see an answer to in the threads I
>>>> scanned...
>>>> I'm assuming you actually use the bridging firewall functionality. So,
>>>> what iptables modules do you use? Can you reduce your iptables rules to
>>>> a core that triggers the bug?
>>>> Or does it get triggered even with an empty set of firewall rules?
>>>> Are you using a stock .35 kernel or is it patched?
>>>> Is this something I can trigger on a poor guy's laptop or does it
>>>> require specialized hardware (I'm catching up on qemu/kvm...)?
>>>
>>> Not specialised hardware as such, I've just not been able to reproduce
>>> it outside of this specific operating scenario.
>>
>> The last similar problem we've had was related to the 32/64 bit compat
>> code. Are you running 32 bit userspace on a 64 bit kernel?
>
> No, 32 bit Guest OS, but a completely 64 bit userspace on a 64 bit kernel.
>
> Userspace is current Debian Stable. Kernel is Vanilla and qemu-kvm is
> current git
>
>
>>> I can't trigger it with empty firewall rules as it relies on a DNAT to
>>> occur. If I try it directly to the internal IP address (as I have to
>>> without netfilter loaded) then of course nothing fails.
>>>
>>> It's a pain in the bum as a fault, but it's one I can easily reproduce
>>> as long as I use the same set of circumstances.
>>>
>>> I'll try using 3.0-rc2 (current git) tonight, and if I can reproduce it
>>> on that then I'll attempt to pare down the IPTABLES rules to a bare
>>> minimum.
>>>
>>> It is nothing to do with ebtables as I don't compile it. I'm not really
>>> sure about "bridging firewall" functionality. I just use a couple of
>>> hand coded bash scripts to set the tables up.
>>
>> From one of your previous mails:
>>
>>> # CONFIG_BRIDGE_NF_EBTABLES is not set
>>
>> How about CONFIG_BRIDGE_NETFILTER?
>>
>
> It was compiled in.
>
> With the following table set I was able to reproduce the problem on
> 3.0-rc2. Replaced my IP with xxx.xxx.xxx.xxx, but otherwise unmodified
Which kernel was the last version without this problem?
> root@srv:~# iptables-save
> # Generated by iptables-save v1.4.10 on Tue Jun 7 22:11:30 2011
> *filter
> :INPUT ACCEPT [978:107619]
> :FORWARD ACCEPT [142:7068]
> :OUTPUT ACCEPT [1659:291870]
> -A INPUT -i ppp0 -m state --state RELATED,ESTABLISHED -j ACCEPT
> -A INPUT ! -i ppp0 -m state --state NEW -j ACCEPT
> -A INPUT -i ppp0 -j DROP
> COMMIT
> # Completed on Tue Jun 7 22:11:30 2011
> # Generated by iptables-save v1.4.10 on Tue Jun 7 22:11:30 2011
> *nat
> :PREROUTING ACCEPT [813:49170]
> :INPUT ACCEPT [91:7090]
> :OUTPUT ACCEPT [267:20731]
> :POSTROUTING ACCEPT [296:22281]
> -A PREROUTING -d xxx.xxx.xxx.xxx/32 ! -i ppp0 -p tcp -m tcp --dport 443
> -j DNAT --to-destination 192.168.253.198
> COMMIT
> # Completed on Tue Jun 7 22:11:30 2011
> # Generated by iptables-save v1.4.10 on Tue Jun 7 22:11:30 2011
> *mangle
> :PREROUTING ACCEPT [2729:274392]
> :INPUT ACCEPT [2508:262976]
> :FORWARD ACCEPT [142:7068]
> :OUTPUT ACCEPT [1674:293701]
> :POSTROUTING ACCEPT [2131:346411]
> -A FORWARD -o ppp0 -p tcp -m tcp --tcp-flags SYN,RST SYN -m tcpmss --mss
> 1400:1536 -j TCPMSS --clamp-mss-to-pmtu
> COMMIT
> # Completed on Tue Jun 7 22:11:30 2011
The main suspects would be NAT and TCPMSS. Did you also try whether
the crash occurs with only one of these these rules?
> I've just compiled out CONFIG_BRIDGE_NETFILTER and can no longer access
> the address the way I was doing it, so that's a no-go for me.
That's really weird since you're apparently not using any bridge
netfilter features. It shouldn't have any effect besides changing
at which point ip_tables is invoked. How are your network devices
configured (specifically any bridges)?
^ permalink raw reply [flat|nested] 111+ messages in thread
* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
@ 2011-06-07 15:35 ` Patrick McHardy
0 siblings, 0 replies; 111+ messages in thread
From: Patrick McHardy @ 2011-06-07 15:35 UTC (permalink / raw)
To: Brad Campbell
Cc: Bart De Schuymer, kvm, linux-mm, linux-kernel, netdev, netfilter-devel
On 07.06.2011 16:40, Brad Campbell wrote:
> On 07/06/11 21:30, Patrick McHardy wrote:
>> On 07.06.2011 05:33, Brad Campbell wrote:
>>> On 07/06/11 04:10, Bart De Schuymer wrote:
>>>> Hi Brad,
>>>>
>>>> This has probably nothing to do with ebtables, so please rmmod in case
>>>> it's loaded.
>>>> A few questions I didn't directly see an answer to in the threads I
>>>> scanned...
>>>> I'm assuming you actually use the bridging firewall functionality. So,
>>>> what iptables modules do you use? Can you reduce your iptables rules to
>>>> a core that triggers the bug?
>>>> Or does it get triggered even with an empty set of firewall rules?
>>>> Are you using a stock .35 kernel or is it patched?
>>>> Is this something I can trigger on a poor guy's laptop or does it
>>>> require specialized hardware (I'm catching up on qemu/kvm...)?
>>>
>>> Not specialised hardware as such, I've just not been able to reproduce
>>> it outside of this specific operating scenario.
>>
>> The last similar problem we've had was related to the 32/64 bit compat
>> code. Are you running 32 bit userspace on a 64 bit kernel?
>
> No, 32 bit Guest OS, but a completely 64 bit userspace on a 64 bit kernel.
>
> Userspace is current Debian Stable. Kernel is Vanilla and qemu-kvm is
> current git
>
>
>>> I can't trigger it with empty firewall rules as it relies on a DNAT to
>>> occur. If I try it directly to the internal IP address (as I have to
>>> without netfilter loaded) then of course nothing fails.
>>>
>>> It's a pain in the bum as a fault, but it's one I can easily reproduce
>>> as long as I use the same set of circumstances.
>>>
>>> I'll try using 3.0-rc2 (current git) tonight, and if I can reproduce it
>>> on that then I'll attempt to pare down the IPTABLES rules to a bare
>>> minimum.
>>>
>>> It is nothing to do with ebtables as I don't compile it. I'm not really
>>> sure about "bridging firewall" functionality. I just use a couple of
>>> hand coded bash scripts to set the tables up.
>>
>> From one of your previous mails:
>>
>>> # CONFIG_BRIDGE_NF_EBTABLES is not set
>>
>> How about CONFIG_BRIDGE_NETFILTER?
>>
>
> It was compiled in.
>
> With the following table set I was able to reproduce the problem on
> 3.0-rc2. Replaced my IP with xxx.xxx.xxx.xxx, but otherwise unmodified
Which kernel was the last version without this problem?
> root@srv:~# iptables-save
> # Generated by iptables-save v1.4.10 on Tue Jun 7 22:11:30 2011
> *filter
> :INPUT ACCEPT [978:107619]
> :FORWARD ACCEPT [142:7068]
> :OUTPUT ACCEPT [1659:291870]
> -A INPUT -i ppp0 -m state --state RELATED,ESTABLISHED -j ACCEPT
> -A INPUT ! -i ppp0 -m state --state NEW -j ACCEPT
> -A INPUT -i ppp0 -j DROP
> COMMIT
> # Completed on Tue Jun 7 22:11:30 2011
> # Generated by iptables-save v1.4.10 on Tue Jun 7 22:11:30 2011
> *nat
> :PREROUTING ACCEPT [813:49170]
> :INPUT ACCEPT [91:7090]
> :OUTPUT ACCEPT [267:20731]
> :POSTROUTING ACCEPT [296:22281]
> -A PREROUTING -d xxx.xxx.xxx.xxx/32 ! -i ppp0 -p tcp -m tcp --dport 443
> -j DNAT --to-destination 192.168.253.198
> COMMIT
> # Completed on Tue Jun 7 22:11:30 2011
> # Generated by iptables-save v1.4.10 on Tue Jun 7 22:11:30 2011
> *mangle
> :PREROUTING ACCEPT [2729:274392]
> :INPUT ACCEPT [2508:262976]
> :FORWARD ACCEPT [142:7068]
> :OUTPUT ACCEPT [1674:293701]
> :POSTROUTING ACCEPT [2131:346411]
> -A FORWARD -o ppp0 -p tcp -m tcp --tcp-flags SYN,RST SYN -m tcpmss --mss
> 1400:1536 -j TCPMSS --clamp-mss-to-pmtu
> COMMIT
> # Completed on Tue Jun 7 22:11:30 2011
The main suspects would be NAT and TCPMSS. Did you also try whether
the crash occurs with only one of these these rules?
> I've just compiled out CONFIG_BRIDGE_NETFILTER and can no longer access
> the address the way I was doing it, so that's a no-go for me.
That's really weird since you're apparently not using any bridge
netfilter features. It shouldn't have any effect besides changing
at which point ip_tables is invoked. How are your network devices
configured (specifically any bridges)?
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 111+ messages in thread
* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
2011-06-07 14:40 ` Brad Campbell
@ 2011-06-07 18:04 ` Bart De Schuymer
-1 siblings, 0 replies; 111+ messages in thread
From: Bart De Schuymer @ 2011-06-07 18:04 UTC (permalink / raw)
To: Brad Campbell
Cc: Patrick McHardy, kvm, linux-mm, linux-kernel, netdev, netfilter-devel
Op 7/06/2011 16:40, Brad Campbell schreef:
> On 07/06/11 21:30, Patrick McHardy wrote:
>> On 07.06.2011 05:33, Brad Campbell wrote:
>>> On 07/06/11 04:10, Bart De Schuymer wrote:
>>>> Hi Brad,
>>>>
>>>> This has probably nothing to do with ebtables, so please rmmod in case
>>>> it's loaded.
>>>> A few questions I didn't directly see an answer to in the threads I
>>>> scanned...
>>>> I'm assuming you actually use the bridging firewall functionality. So,
>>>> what iptables modules do you use? Can you reduce your iptables
>>>> rules to
>>>> a core that triggers the bug?
>>>> Or does it get triggered even with an empty set of firewall rules?
>>>> Are you using a stock .35 kernel or is it patched?
>>>> Is this something I can trigger on a poor guy's laptop or does it
>>>> require specialized hardware (I'm catching up on qemu/kvm...)?
>>>
>>> Not specialised hardware as such, I've just not been able to reproduce
>>> it outside of this specific operating scenario.
>>
>> The last similar problem we've had was related to the 32/64 bit compat
>> code. Are you running 32 bit userspace on a 64 bit kernel?
>
> No, 32 bit Guest OS, but a completely 64 bit userspace on a 64 bit
> kernel.
>
> Userspace is current Debian Stable. Kernel is Vanilla and qemu-kvm is
> current git
>
If the bug is easily triggered with your guest os, then you could try to
capture the traffic with wireshark (or something else) in a
configuration that doesn't crash your system. Save the traffic in a pcap
file. Then you can see if resending that traffic in the vulnerable
configuration triggers the bug (I don't know if something in Windows
exists, but tcpreplay should work for Linux). Once you have such a
capture , chances are the bug is even easily reproducible by us (unless
it's hardware-specific). Success isn't guaranteed, but I think it's
worth a shot...
cheers,
Bart
--
Bart De Schuymer
www.artinalgorithms.be
^ permalink raw reply [flat|nested] 111+ messages in thread
* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
@ 2011-06-07 18:04 ` Bart De Schuymer
0 siblings, 0 replies; 111+ messages in thread
From: Bart De Schuymer @ 2011-06-07 18:04 UTC (permalink / raw)
To: Brad Campbell
Cc: Patrick McHardy, kvm, linux-mm, linux-kernel, netdev, netfilter-devel
Op 7/06/2011 16:40, Brad Campbell schreef:
> On 07/06/11 21:30, Patrick McHardy wrote:
>> On 07.06.2011 05:33, Brad Campbell wrote:
>>> On 07/06/11 04:10, Bart De Schuymer wrote:
>>>> Hi Brad,
>>>>
>>>> This has probably nothing to do with ebtables, so please rmmod in case
>>>> it's loaded.
>>>> A few questions I didn't directly see an answer to in the threads I
>>>> scanned...
>>>> I'm assuming you actually use the bridging firewall functionality. So,
>>>> what iptables modules do you use? Can you reduce your iptables
>>>> rules to
>>>> a core that triggers the bug?
>>>> Or does it get triggered even with an empty set of firewall rules?
>>>> Are you using a stock .35 kernel or is it patched?
>>>> Is this something I can trigger on a poor guy's laptop or does it
>>>> require specialized hardware (I'm catching up on qemu/kvm...)?
>>>
>>> Not specialised hardware as such, I've just not been able to reproduce
>>> it outside of this specific operating scenario.
>>
>> The last similar problem we've had was related to the 32/64 bit compat
>> code. Are you running 32 bit userspace on a 64 bit kernel?
>
> No, 32 bit Guest OS, but a completely 64 bit userspace on a 64 bit
> kernel.
>
> Userspace is current Debian Stable. Kernel is Vanilla and qemu-kvm is
> current git
>
If the bug is easily triggered with your guest os, then you could try to
capture the traffic with wireshark (or something else) in a
configuration that doesn't crash your system. Save the traffic in a pcap
file. Then you can see if resending that traffic in the vulnerable
configuration triggers the bug (I don't know if something in Windows
exists, but tcpreplay should work for Linux). Once you have such a
capture , chances are the bug is even easily reproducible by us (unless
it's hardware-specific). Success isn't guaranteed, but I think it's
worth a shot...
cheers,
Bart
--
Bart De Schuymer
www.artinalgorithms.be
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 111+ messages in thread
* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
2011-06-07 15:35 ` Patrick McHardy
(?)
@ 2011-06-07 18:31 ` Eric Dumazet
-1 siblings, 0 replies; 111+ messages in thread
From: Eric Dumazet @ 2011-06-07 18:31 UTC (permalink / raw)
To: Patrick McHardy
Cc: Brad Campbell, Bart De Schuymer, kvm, linux-mm, linux-kernel,
netdev, netfilter-devel
Le mardi 07 juin 2011 à 17:35 +0200, Patrick McHardy a écrit :
> The main suspects would be NAT and TCPMSS. Did you also try whether
> the crash occurs with only one of these these rules?
>
> > I've just compiled out CONFIG_BRIDGE_NETFILTER and can no longer access
> > the address the way I was doing it, so that's a no-go for me.
>
> That's really weird since you're apparently not using any bridge
> netfilter features. It shouldn't have any effect besides changing
> at which point ip_tables is invoked. How are your network devices
> configured (specifically any bridges)?
Something in the kernel does
u16 *ptr = addr (given by kmalloc())
ptr[-1] = 0;
Could be an off-one error in a memmove()/memcopy() or loop...
I cant see a network issue here.
I checked arch/x86/lib/memmove_64.S and it seems fine.
^ permalink raw reply [flat|nested] 111+ messages in thread
* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
@ 2011-06-07 18:31 ` Eric Dumazet
0 siblings, 0 replies; 111+ messages in thread
From: Eric Dumazet @ 2011-06-07 18:31 UTC (permalink / raw)
To: Patrick McHardy
Cc: Brad Campbell, Bart De Schuymer, kvm, linux-mm, linux-kernel,
netdev, netfilter-devel
Le mardi 07 juin 2011 à 17:35 +0200, Patrick McHardy a écrit :
> The main suspects would be NAT and TCPMSS. Did you also try whether
> the crash occurs with only one of these these rules?
>
> > I've just compiled out CONFIG_BRIDGE_NETFILTER and can no longer access
> > the address the way I was doing it, so that's a no-go for me.
>
> That's really weird since you're apparently not using any bridge
> netfilter features. It shouldn't have any effect besides changing
> at which point ip_tables is invoked. How are your network devices
> configured (specifically any bridges)?
Something in the kernel does
u16 *ptr = addr (given by kmalloc())
ptr[-1] = 0;
Could be an off-one error in a memmove()/memcopy() or loop...
I cant see a network issue here.
I checked arch/x86/lib/memmove_64.S and it seems fine.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 111+ messages in thread
* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
@ 2011-06-07 18:31 ` Eric Dumazet
0 siblings, 0 replies; 111+ messages in thread
From: Eric Dumazet @ 2011-06-07 18:31 UTC (permalink / raw)
To: Patrick McHardy
Cc: Brad Campbell, Bart De Schuymer, kvm, linux-mm, linux-kernel,
netdev, netfilter-devel
Le mardi 07 juin 2011 A 17:35 +0200, Patrick McHardy a A(C)crit :
> The main suspects would be NAT and TCPMSS. Did you also try whether
> the crash occurs with only one of these these rules?
>
> > I've just compiled out CONFIG_BRIDGE_NETFILTER and can no longer access
> > the address the way I was doing it, so that's a no-go for me.
>
> That's really weird since you're apparently not using any bridge
> netfilter features. It shouldn't have any effect besides changing
> at which point ip_tables is invoked. How are your network devices
> configured (specifically any bridges)?
Something in the kernel does
u16 *ptr = addr (given by kmalloc())
ptr[-1] = 0;
Could be an off-one error in a memmove()/memcopy() or loop...
I cant see a network issue here.
I checked arch/x86/lib/memmove_64.S and it seems fine.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 111+ messages in thread
* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
2011-06-07 18:31 ` Eric Dumazet
(?)
@ 2011-06-07 22:57 ` Patrick McHardy
-1 siblings, 0 replies; 111+ messages in thread
From: Patrick McHardy @ 2011-06-07 22:57 UTC (permalink / raw)
To: Eric Dumazet
Cc: Brad Campbell, Bart De Schuymer, kvm, linux-mm, linux-kernel,
netdev, netfilter-devel
On 07.06.2011 20:31, Eric Dumazet wrote:
> Le mardi 07 juin 2011 à 17:35 +0200, Patrick McHardy a écrit :
>
>> The main suspects would be NAT and TCPMSS. Did you also try whether
>> the crash occurs with only one of these these rules?
>>
>>> I've just compiled out CONFIG_BRIDGE_NETFILTER and can no longer access
>>> the address the way I was doing it, so that's a no-go for me.
>>
>> That's really weird since you're apparently not using any bridge
>> netfilter features. It shouldn't have any effect besides changing
>> at which point ip_tables is invoked. How are your network devices
>> configured (specifically any bridges)?
>
> Something in the kernel does
>
> u16 *ptr = addr (given by kmalloc())
>
> ptr[-1] = 0;
>
> Could be an off-one error in a memmove()/memcopy() or loop...
>
> I cant see a network issue here.
So far me neither, but netfilter appears to trigger the bug.
> I checked arch/x86/lib/memmove_64.S and it seems fine.
I was thinking it might be a missing skb_make_writable() combined
with vhost_net specifics in the netfilter code (TCPMSS and NAT are
both suspect), but was unable to find something. I also went
through the dst_metrics() conversion to see whether anything could
cause problems with the bridge fake_rttable, but also nothing
so far.
^ permalink raw reply [flat|nested] 111+ messages in thread
* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
@ 2011-06-07 22:57 ` Patrick McHardy
0 siblings, 0 replies; 111+ messages in thread
From: Patrick McHardy @ 2011-06-07 22:57 UTC (permalink / raw)
To: Eric Dumazet
Cc: Brad Campbell, Bart De Schuymer, kvm, linux-mm, linux-kernel,
netdev, netfilter-devel
On 07.06.2011 20:31, Eric Dumazet wrote:
> Le mardi 07 juin 2011 à 17:35 +0200, Patrick McHardy a écrit :
>
>> The main suspects would be NAT and TCPMSS. Did you also try whether
>> the crash occurs with only one of these these rules?
>>
>>> I've just compiled out CONFIG_BRIDGE_NETFILTER and can no longer access
>>> the address the way I was doing it, so that's a no-go for me.
>>
>> That's really weird since you're apparently not using any bridge
>> netfilter features. It shouldn't have any effect besides changing
>> at which point ip_tables is invoked. How are your network devices
>> configured (specifically any bridges)?
>
> Something in the kernel does
>
> u16 *ptr = addr (given by kmalloc())
>
> ptr[-1] = 0;
>
> Could be an off-one error in a memmove()/memcopy() or loop...
>
> I cant see a network issue here.
So far me neither, but netfilter appears to trigger the bug.
> I checked arch/x86/lib/memmove_64.S and it seems fine.
I was thinking it might be a missing skb_make_writable() combined
with vhost_net specifics in the netfilter code (TCPMSS and NAT are
both suspect), but was unable to find something. I also went
through the dst_metrics() conversion to see whether anything could
cause problems with the bridge fake_rttable, but also nothing
so far.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 111+ messages in thread
* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
@ 2011-06-07 22:57 ` Patrick McHardy
0 siblings, 0 replies; 111+ messages in thread
From: Patrick McHardy @ 2011-06-07 22:57 UTC (permalink / raw)
To: Eric Dumazet
Cc: Brad Campbell, Bart De Schuymer, kvm, linux-mm, linux-kernel,
netdev, netfilter-devel
On 07.06.2011 20:31, Eric Dumazet wrote:
> Le mardi 07 juin 2011 a 17:35 +0200, Patrick McHardy a ecrit :
>
>> The main suspects would be NAT and TCPMSS. Did you also try whether
>> the crash occurs with only one of these these rules?
>>
>>> I've just compiled out CONFIG_BRIDGE_NETFILTER and can no longer access
>>> the address the way I was doing it, so that's a no-go for me.
>>
>> That's really weird since you're apparently not using any bridge
>> netfilter features. It shouldn't have any effect besides changing
>> at which point ip_tables is invoked. How are your network devices
>> configured (specifically any bridges)?
>
> Something in the kernel does
>
> u16 *ptr = addr (given by kmalloc())
>
> ptr[-1] = 0;
>
> Could be an off-one error in a memmove()/memcopy() or loop...
>
> I cant see a network issue here.
So far me neither, but netfilter appears to trigger the bug.
> I checked arch/x86/lib/memmove_64.S and it seems fine.
I was thinking it might be a missing skb_make_writable() combined
with vhost_net specifics in the netfilter code (TCPMSS and NAT are
both suspect), but was unable to find something. I also went
through the dst_metrics() conversion to see whether anything could
cause problems with the bridge fake_rttable, but also nothing
so far.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 111+ messages in thread
* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
2011-06-07 15:35 ` Patrick McHardy
@ 2011-06-07 23:43 ` Brad Campbell
-1 siblings, 0 replies; 111+ messages in thread
From: Brad Campbell @ 2011-06-07 23:43 UTC (permalink / raw)
To: Patrick McHardy
Cc: Bart De Schuymer, kvm, linux-mm, linux-kernel, netdev, netfilter-devel
On 07/06/11 23:35, Patrick McHardy wrote:
> The main suspects would be NAT and TCPMSS. Did you also try whether
> the crash occurs with only one of these these rules?
To be honest I'm actually having trouble finding where TCPMSS is
actually set in that ruleset. This is a production machine so I can only
take it down after about 9PM at night. I'll have another crack at it
tonight.
>> I've just compiled out CONFIG_BRIDGE_NETFILTER and can no longer access
>> the address the way I was doing it, so that's a no-go for me.
>
> That's really weird since you're apparently not using any bridge
> netfilter features. It shouldn't have any effect besides changing
> at which point ip_tables is invoked. How are your network devices
> configured (specifically any bridges)?
>
I have one bridge with all my virtual machines on it.
In this particular instance the packets leave VM A destined for the IP
address of ppp0 (the external interface). This is intercepted by the
DNAT PREROUTING rule above and shunted back to VM B.
The VM's are on br1 and the external address is ppp0. Without
CONFIG_BRIDGE_NETFILTER compiled in I can see the traffic entering and
leaving VM B with tcpdump, but the packets never seem to get back to VM A.
VM A is XP 32 bit, VM B is Linux. I have some other Linux VM's, so I'll
do some more testing tonight between those to see where the packets are
going without CONFIG_BRIDGE_NETFILTER set.
^ permalink raw reply [flat|nested] 111+ messages in thread
* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
@ 2011-06-07 23:43 ` Brad Campbell
0 siblings, 0 replies; 111+ messages in thread
From: Brad Campbell @ 2011-06-07 23:43 UTC (permalink / raw)
To: Patrick McHardy
Cc: Bart De Schuymer, kvm, linux-mm, linux-kernel, netdev, netfilter-devel
On 07/06/11 23:35, Patrick McHardy wrote:
> The main suspects would be NAT and TCPMSS. Did you also try whether
> the crash occurs with only one of these these rules?
To be honest I'm actually having trouble finding where TCPMSS is
actually set in that ruleset. This is a production machine so I can only
take it down after about 9PM at night. I'll have another crack at it
tonight.
>> I've just compiled out CONFIG_BRIDGE_NETFILTER and can no longer access
>> the address the way I was doing it, so that's a no-go for me.
>
> That's really weird since you're apparently not using any bridge
> netfilter features. It shouldn't have any effect besides changing
> at which point ip_tables is invoked. How are your network devices
> configured (specifically any bridges)?
>
I have one bridge with all my virtual machines on it.
In this particular instance the packets leave VM A destined for the IP
address of ppp0 (the external interface). This is intercepted by the
DNAT PREROUTING rule above and shunted back to VM B.
The VM's are on br1 and the external address is ppp0. Without
CONFIG_BRIDGE_NETFILTER compiled in I can see the traffic entering and
leaving VM B with tcpdump, but the packets never seem to get back to VM A.
VM A is XP 32 bit, VM B is Linux. I have some other Linux VM's, so I'll
do some more testing tonight between those to see where the packets are
going without CONFIG_BRIDGE_NETFILTER set.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 111+ messages in thread
* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
2011-06-07 18:04 ` Bart De Schuymer
@ 2011-06-08 0:15 ` Brad Campbell
-1 siblings, 0 replies; 111+ messages in thread
From: Brad Campbell @ 2011-06-08 0:15 UTC (permalink / raw)
To: Bart De Schuymer
Cc: Patrick McHardy, kvm, linux-mm, linux-kernel, netdev, netfilter-devel
On 08/06/11 02:04, Bart De Schuymer wrote:
> If the bug is easily triggered with your guest os, then you could try to
> capture the traffic with wireshark (or something else) in a
> configuration that doesn't crash your system. Save the traffic in a pcap
> file. Then you can see if resending that traffic in the vulnerable
> configuration triggers the bug (I don't know if something in Windows
> exists, but tcpreplay should work for Linux). Once you have such a
> capture , chances are the bug is even easily reproducible by us (unless
> it's hardware-specific). Success isn't guaranteed, but I think it's
> worth a shot...
The issue with this is I don't have a configuration that does not crash
the system. This only happens under the specific circumstance that
traffic from VM A is being DNAT'd to VM B. If I disable
CONFIG_BRIDGE_NETFILTER, or I leave out the DNAT then I can't replicate
the problem as I don't seem to be able to get the packets to go where I
want them to go.
Let me try and explain it a little more clearly with made up IP
addresses to illustrate the problem.
I have VM A (1.1.1.2) and VM B (1.1.1.3) on br1 (1.1.1.1)
I have public IP on ppp0 (2.2.2.2).
VM B can talk to VM A using its host address (1.1.1.2) and there is no
problem.
The DNAT says anything destined for PPP0 that is on port 443 and coming
from anywhere other than PPP0 (ie inside the network) is to be DNAT'd to
1.1.1.3.
So VM B (1.1.1.3) tries to connect to ppp0 (2.2.2.2) on port 443, and
this is redirected to VM B on 1.1.1.2.
Only under this specific circumstance does the problem occur. I can get
VM B (1.1.1.3) to talk directly to VM A (1.1.1.2) all day long and there
is no problem, it's only when VM B tries to talk to ppp0 that there is
an issue (and it happens within seconds of the initial connection).
All these tests have been performed with VM B being a Windows XP guest.
Tonight I'll try it with a Linux guest and see if I can make it happen.
If that works I might be able to come up with some reproducible test
case for you. I have a desktop machine that has Intel VT extensions, so
I'll work toward making a portable test case.
^ permalink raw reply [flat|nested] 111+ messages in thread
* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
@ 2011-06-08 0:15 ` Brad Campbell
0 siblings, 0 replies; 111+ messages in thread
From: Brad Campbell @ 2011-06-08 0:15 UTC (permalink / raw)
To: Bart De Schuymer
Cc: Patrick McHardy, kvm, linux-mm, linux-kernel, netdev, netfilter-devel
On 08/06/11 02:04, Bart De Schuymer wrote:
> If the bug is easily triggered with your guest os, then you could try to
> capture the traffic with wireshark (or something else) in a
> configuration that doesn't crash your system. Save the traffic in a pcap
> file. Then you can see if resending that traffic in the vulnerable
> configuration triggers the bug (I don't know if something in Windows
> exists, but tcpreplay should work for Linux). Once you have such a
> capture , chances are the bug is even easily reproducible by us (unless
> it's hardware-specific). Success isn't guaranteed, but I think it's
> worth a shot...
The issue with this is I don't have a configuration that does not crash
the system. This only happens under the specific circumstance that
traffic from VM A is being DNAT'd to VM B. If I disable
CONFIG_BRIDGE_NETFILTER, or I leave out the DNAT then I can't replicate
the problem as I don't seem to be able to get the packets to go where I
want them to go.
Let me try and explain it a little more clearly with made up IP
addresses to illustrate the problem.
I have VM A (1.1.1.2) and VM B (1.1.1.3) on br1 (1.1.1.1)
I have public IP on ppp0 (2.2.2.2).
VM B can talk to VM A using its host address (1.1.1.2) and there is no
problem.
The DNAT says anything destined for PPP0 that is on port 443 and coming
from anywhere other than PPP0 (ie inside the network) is to be DNAT'd to
1.1.1.3.
So VM B (1.1.1.3) tries to connect to ppp0 (2.2.2.2) on port 443, and
this is redirected to VM B on 1.1.1.2.
Only under this specific circumstance does the problem occur. I can get
VM B (1.1.1.3) to talk directly to VM A (1.1.1.2) all day long and there
is no problem, it's only when VM B tries to talk to ppp0 that there is
an issue (and it happens within seconds of the initial connection).
All these tests have been performed with VM B being a Windows XP guest.
Tonight I'll try it with a Linux guest and see if I can make it happen.
If that works I might be able to come up with some reproducible test
case for you. I have a desktop machine that has Intel VT extensions, so
I'll work toward making a portable test case.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 111+ messages in thread
* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
2011-06-07 22:57 ` Patrick McHardy
(?)
@ 2011-06-08 0:18 ` Brad Campbell
-1 siblings, 0 replies; 111+ messages in thread
From: Brad Campbell @ 2011-06-08 0:18 UTC (permalink / raw)
To: Patrick McHardy
Cc: Eric Dumazet, Bart De Schuymer, kvm, linux-mm, linux-kernel,
netdev, netfilter-devel
On 08/06/11 06:57, Patrick McHardy wrote:
> On 07.06.2011 20:31, Eric Dumazet wrote:
>> Le mardi 07 juin 2011 à 17:35 +0200, Patrick McHardy a écrit :
>>
>>> The main suspects would be NAT and TCPMSS. Did you also try whether
>>> the crash occurs with only one of these these rules?
>>>
>>>> I've just compiled out CONFIG_BRIDGE_NETFILTER and can no longer access
>>>> the address the way I was doing it, so that's a no-go for me.
>>>
>>> That's really weird since you're apparently not using any bridge
>>> netfilter features. It shouldn't have any effect besides changing
>>> at which point ip_tables is invoked. How are your network devices
>>> configured (specifically any bridges)?
>>
>> Something in the kernel does
>>
>> u16 *ptr = addr (given by kmalloc())
>>
>> ptr[-1] = 0;
>>
>> Could be an off-one error in a memmove()/memcopy() or loop...
>>
>> I cant see a network issue here.
>
> So far me neither, but netfilter appears to trigger the bug.
Would it help if I tried some older kernels? This issue only surfaced
for me recently as I only installed the VM's in question about 12 weeks
ago and have only just started really using them in anger. I could try
reproducing it on progressively older kernels to see if I can find one
that works and then bisect from there.
^ permalink raw reply [flat|nested] 111+ messages in thread
* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
@ 2011-06-08 0:18 ` Brad Campbell
0 siblings, 0 replies; 111+ messages in thread
From: Brad Campbell @ 2011-06-08 0:18 UTC (permalink / raw)
To: Patrick McHardy
Cc: Eric Dumazet, Bart De Schuymer, kvm, linux-mm, linux-kernel,
netdev, netfilter-devel
On 08/06/11 06:57, Patrick McHardy wrote:
> On 07.06.2011 20:31, Eric Dumazet wrote:
>> Le mardi 07 juin 2011 à 17:35 +0200, Patrick McHardy a écrit :
>>
>>> The main suspects would be NAT and TCPMSS. Did you also try whether
>>> the crash occurs with only one of these these rules?
>>>
>>>> I've just compiled out CONFIG_BRIDGE_NETFILTER and can no longer access
>>>> the address the way I was doing it, so that's a no-go for me.
>>>
>>> That's really weird since you're apparently not using any bridge
>>> netfilter features. It shouldn't have any effect besides changing
>>> at which point ip_tables is invoked. How are your network devices
>>> configured (specifically any bridges)?
>>
>> Something in the kernel does
>>
>> u16 *ptr = addr (given by kmalloc())
>>
>> ptr[-1] = 0;
>>
>> Could be an off-one error in a memmove()/memcopy() or loop...
>>
>> I cant see a network issue here.
>
> So far me neither, but netfilter appears to trigger the bug.
Would it help if I tried some older kernels? This issue only surfaced
for me recently as I only installed the VM's in question about 12 weeks
ago and have only just started really using them in anger. I could try
reproducing it on progressively older kernels to see if I can find one
that works and then bisect from there.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 111+ messages in thread
* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
@ 2011-06-08 0:18 ` Brad Campbell
0 siblings, 0 replies; 111+ messages in thread
From: Brad Campbell @ 2011-06-08 0:18 UTC (permalink / raw)
To: Patrick McHardy
Cc: Eric Dumazet, Bart De Schuymer, kvm, linux-mm, linux-kernel,
netdev, netfilter-devel
On 08/06/11 06:57, Patrick McHardy wrote:
> On 07.06.2011 20:31, Eric Dumazet wrote:
>> Le mardi 07 juin 2011 a 17:35 +0200, Patrick McHardy a ecrit :
>>
>>> The main suspects would be NAT and TCPMSS. Did you also try whether
>>> the crash occurs with only one of these these rules?
>>>
>>>> I've just compiled out CONFIG_BRIDGE_NETFILTER and can no longer access
>>>> the address the way I was doing it, so that's a no-go for me.
>>>
>>> That's really weird since you're apparently not using any bridge
>>> netfilter features. It shouldn't have any effect besides changing
>>> at which point ip_tables is invoked. How are your network devices
>>> configured (specifically any bridges)?
>>
>> Something in the kernel does
>>
>> u16 *ptr = addr (given by kmalloc())
>>
>> ptr[-1] = 0;
>>
>> Could be an off-one error in a memmove()/memcopy() or loop...
>>
>> I cant see a network issue here.
>
> So far me neither, but netfilter appears to trigger the bug.
Would it help if I tried some older kernels? This issue only surfaced
for me recently as I only installed the VM's in question about 12 weeks
ago and have only just started really using them in anger. I could try
reproducing it on progressively older kernels to see if I can find one
that works and then bisect from there.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 111+ messages in thread
* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
2011-06-08 0:18 ` Brad Campbell
(?)
@ 2011-06-08 3:59 ` Eric Dumazet
-1 siblings, 0 replies; 111+ messages in thread
From: Eric Dumazet @ 2011-06-08 3:59 UTC (permalink / raw)
To: Brad Campbell
Cc: Patrick McHardy, Bart De Schuymer, kvm, linux-mm, linux-kernel,
netdev, netfilter-devel
Le mercredi 08 juin 2011 à 08:18 +0800, Brad Campbell a écrit :
> On 08/06/11 06:57, Patrick McHardy wrote:
> > On 07.06.2011 20:31, Eric Dumazet wrote:
> >> Le mardi 07 juin 2011 à 17:35 +0200, Patrick McHardy a écrit :
> >>
> >>> The main suspects would be NAT and TCPMSS. Did you also try whether
> >>> the crash occurs with only one of these these rules?
> >>>
> >>>> I've just compiled out CONFIG_BRIDGE_NETFILTER and can no longer access
> >>>> the address the way I was doing it, so that's a no-go for me.
> >>>
> >>> That's really weird since you're apparently not using any bridge
> >>> netfilter features. It shouldn't have any effect besides changing
> >>> at which point ip_tables is invoked. How are your network devices
> >>> configured (specifically any bridges)?
> >>
> >> Something in the kernel does
> >>
> >> u16 *ptr = addr (given by kmalloc())
> >>
> >> ptr[-1] = 0;
> >>
> >> Could be an off-one error in a memmove()/memcopy() or loop...
> >>
> >> I cant see a network issue here.
> >
> > So far me neither, but netfilter appears to trigger the bug.
>
> Would it help if I tried some older kernels? This issue only surfaced
> for me recently as I only installed the VM's in question about 12 weeks
> ago and have only just started really using them in anger. I could try
> reproducing it on progressively older kernels to see if I can find one
> that works and then bisect from there.
Well, a bisection definitely should help, but needs a lot of time in
your case.
Could you try following patch, because this is the 'usual suspect' I had
yesterday :
diff --git a/net/core/skbuff.c b/net/core/skbuff.c
index 46cbd28..9f548f9 100644
--- a/net/core/skbuff.c
+++ b/net/core/skbuff.c
@@ -792,6 +792,7 @@ int pskb_expand_head(struct sk_buff *skb, int nhead, int ntail,
fastpath = atomic_read(&skb_shinfo(skb)->dataref) == delta;
}
+#if 0
if (fastpath &&
size + sizeof(struct skb_shared_info) <= ksize(skb->head)) {
memmove(skb->head + size, skb_shinfo(skb),
@@ -802,7 +803,7 @@ int pskb_expand_head(struct sk_buff *skb, int nhead, int ntail,
off = nhead;
goto adjust_others;
}
-
+#endif
data = kmalloc(size + sizeof(struct skb_shared_info), gfp_mask);
if (!data)
goto nodata;
^ permalink raw reply related [flat|nested] 111+ messages in thread
* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
@ 2011-06-08 3:59 ` Eric Dumazet
0 siblings, 0 replies; 111+ messages in thread
From: Eric Dumazet @ 2011-06-08 3:59 UTC (permalink / raw)
To: Brad Campbell
Cc: Patrick McHardy, Bart De Schuymer, kvm, linux-mm, linux-kernel,
netdev, netfilter-devel
Le mercredi 08 juin 2011 à 08:18 +0800, Brad Campbell a écrit :
> On 08/06/11 06:57, Patrick McHardy wrote:
> > On 07.06.2011 20:31, Eric Dumazet wrote:
> >> Le mardi 07 juin 2011 à 17:35 +0200, Patrick McHardy a écrit :
> >>
> >>> The main suspects would be NAT and TCPMSS. Did you also try whether
> >>> the crash occurs with only one of these these rules?
> >>>
> >>>> I've just compiled out CONFIG_BRIDGE_NETFILTER and can no longer access
> >>>> the address the way I was doing it, so that's a no-go for me.
> >>>
> >>> That's really weird since you're apparently not using any bridge
> >>> netfilter features. It shouldn't have any effect besides changing
> >>> at which point ip_tables is invoked. How are your network devices
> >>> configured (specifically any bridges)?
> >>
> >> Something in the kernel does
> >>
> >> u16 *ptr = addr (given by kmalloc())
> >>
> >> ptr[-1] = 0;
> >>
> >> Could be an off-one error in a memmove()/memcopy() or loop...
> >>
> >> I cant see a network issue here.
> >
> > So far me neither, but netfilter appears to trigger the bug.
>
> Would it help if I tried some older kernels? This issue only surfaced
> for me recently as I only installed the VM's in question about 12 weeks
> ago and have only just started really using them in anger. I could try
> reproducing it on progressively older kernels to see if I can find one
> that works and then bisect from there.
Well, a bisection definitely should help, but needs a lot of time in
your case.
Could you try following patch, because this is the 'usual suspect' I had
yesterday :
diff --git a/net/core/skbuff.c b/net/core/skbuff.c
index 46cbd28..9f548f9 100644
--- a/net/core/skbuff.c
+++ b/net/core/skbuff.c
@@ -792,6 +792,7 @@ int pskb_expand_head(struct sk_buff *skb, int nhead, int ntail,
fastpath = atomic_read(&skb_shinfo(skb)->dataref) == delta;
}
+#if 0
if (fastpath &&
size + sizeof(struct skb_shared_info) <= ksize(skb->head)) {
memmove(skb->head + size, skb_shinfo(skb),
@@ -802,7 +803,7 @@ int pskb_expand_head(struct sk_buff *skb, int nhead, int ntail,
off = nhead;
goto adjust_others;
}
-
+#endif
data = kmalloc(size + sizeof(struct skb_shared_info), gfp_mask);
if (!data)
goto nodata;
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related [flat|nested] 111+ messages in thread
* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
@ 2011-06-08 3:59 ` Eric Dumazet
0 siblings, 0 replies; 111+ messages in thread
From: Eric Dumazet @ 2011-06-08 3:59 UTC (permalink / raw)
To: Brad Campbell
Cc: Patrick McHardy, Bart De Schuymer, kvm, linux-mm, linux-kernel,
netdev, netfilter-devel
Le mercredi 08 juin 2011 A 08:18 +0800, Brad Campbell a A(C)crit :
> On 08/06/11 06:57, Patrick McHardy wrote:
> > On 07.06.2011 20:31, Eric Dumazet wrote:
> >> Le mardi 07 juin 2011 A 17:35 +0200, Patrick McHardy a A(C)crit :
> >>
> >>> The main suspects would be NAT and TCPMSS. Did you also try whether
> >>> the crash occurs with only one of these these rules?
> >>>
> >>>> I've just compiled out CONFIG_BRIDGE_NETFILTER and can no longer access
> >>>> the address the way I was doing it, so that's a no-go for me.
> >>>
> >>> That's really weird since you're apparently not using any bridge
> >>> netfilter features. It shouldn't have any effect besides changing
> >>> at which point ip_tables is invoked. How are your network devices
> >>> configured (specifically any bridges)?
> >>
> >> Something in the kernel does
> >>
> >> u16 *ptr = addr (given by kmalloc())
> >>
> >> ptr[-1] = 0;
> >>
> >> Could be an off-one error in a memmove()/memcopy() or loop...
> >>
> >> I cant see a network issue here.
> >
> > So far me neither, but netfilter appears to trigger the bug.
>
> Would it help if I tried some older kernels? This issue only surfaced
> for me recently as I only installed the VM's in question about 12 weeks
> ago and have only just started really using them in anger. I could try
> reproducing it on progressively older kernels to see if I can find one
> that works and then bisect from there.
Well, a bisection definitely should help, but needs a lot of time in
your case.
Could you try following patch, because this is the 'usual suspect' I had
yesterday :
diff --git a/net/core/skbuff.c b/net/core/skbuff.c
index 46cbd28..9f548f9 100644
--- a/net/core/skbuff.c
+++ b/net/core/skbuff.c
@@ -792,6 +792,7 @@ int pskb_expand_head(struct sk_buff *skb, int nhead, int ntail,
fastpath = atomic_read(&skb_shinfo(skb)->dataref) == delta;
}
+#if 0
if (fastpath &&
size + sizeof(struct skb_shared_info) <= ksize(skb->head)) {
memmove(skb->head + size, skb_shinfo(skb),
@@ -802,7 +803,7 @@ int pskb_expand_head(struct sk_buff *skb, int nhead, int ntail,
off = nhead;
goto adjust_others;
}
-
+#endif
data = kmalloc(size + sizeof(struct skb_shared_info), gfp_mask);
if (!data)
goto nodata;
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related [flat|nested] 111+ messages in thread
* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
2011-06-08 3:59 ` Eric Dumazet
(?)
@ 2011-06-08 17:02 ` Brad Campbell
-1 siblings, 0 replies; 111+ messages in thread
From: Brad Campbell @ 2011-06-08 17:02 UTC (permalink / raw)
To: Eric Dumazet
Cc: Patrick McHardy, Bart De Schuymer, kvm, linux-mm, linux-kernel,
netdev, netfilter-devel
On 08/06/11 11:59, Eric Dumazet wrote:
> Well, a bisection definitely should help, but needs a lot of time in
> your case.
Yes. compile, test, crash, walk out to the other building to press
reset, lather, rinse, repeat.
I need a reset button on the end of a 50M wire, or a hardware watchdog!
Actually it's not so bad. If I turn off slub debugging the kernel panics
and reboots itself.
This.. :
[ 2.913034] netconsole: remote ethernet address 00:16:cb:a7:dd:d1
[ 2.913066] netconsole: device eth0 not up yet, forcing it
[ 3.660062] Refined TSC clocksource calibration: 3213.422 MHz.
[ 3.660118] Switching to clocksource tsc
[ 63.200273] r8169 0000:03:00.0: eth0: unable to load firmware patch
rtl_nic/rtl8168e-1.fw (-2)
[ 63.223513] r8169 0000:03:00.0: eth0: link down
[ 63.223556] r8169 0000:03:00.0: eth0: link down
..is slowing down reboots considerably. 3.0-rc does _not_ like some
timing hardware in my machine. Having said that, at least it does not
randomly panic on SCSI like 2.6.39 does.
Ok, I've ruled out TCPMSS. Found out where it was being set and neutered
it. I've replicated it with only the single DNAT rule.
> Could you try following patch, because this is the 'usual suspect' I had
> yesterday :
>
> diff --git a/net/core/skbuff.c b/net/core/skbuff.c
> index 46cbd28..9f548f9 100644
> --- a/net/core/skbuff.c
> +++ b/net/core/skbuff.c
> @@ -792,6 +792,7 @@ int pskb_expand_head(struct sk_buff *skb, int nhead, int ntail,
> fastpath = atomic_read(&skb_shinfo(skb)->dataref) == delta;
> }
>
> +#if 0
> if (fastpath&&
> size + sizeof(struct skb_shared_info)<= ksize(skb->head)) {
> memmove(skb->head + size, skb_shinfo(skb),
> @@ -802,7 +803,7 @@ int pskb_expand_head(struct sk_buff *skb, int nhead, int ntail,
> off = nhead;
> goto adjust_others;
> }
> -
> +#endif
> data = kmalloc(size + sizeof(struct skb_shared_info), gfp_mask);
> if (!data)
> goto nodata;
>
>
>
Nope.. that's not it. <sigh> That might have changed the characteristic
of the fault slightly, but unfortunately I got caught with a couple of
fsck's, so I only got to test it 3 times tonight.
It's unfortunate that this is a production system, so I can only take it
down between about 9pm and 1am. That would normally be pretty
productive, except that an fsck of a 14TB ext4 can take 30 minutes if it
panics at the wrong time.
I'm out of time tonight, but I'll have a crack at some bisection
tomorrow night. Now I just have to go back far enough that it works, and
be near enough not to have to futz around with /proc /sys or drivers.
I really, really, really appreciate you guys helping me with this. It
has been driving me absolutely bonkers. If I'm ever in the same town as
any of you, dinner and drinks are on me.
^ permalink raw reply [flat|nested] 111+ messages in thread
* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
@ 2011-06-08 17:02 ` Brad Campbell
0 siblings, 0 replies; 111+ messages in thread
From: Brad Campbell @ 2011-06-08 17:02 UTC (permalink / raw)
To: Eric Dumazet
Cc: Patrick McHardy, Bart De Schuymer, kvm, linux-mm, linux-kernel,
netdev, netfilter-devel
On 08/06/11 11:59, Eric Dumazet wrote:
> Well, a bisection definitely should help, but needs a lot of time in
> your case.
Yes. compile, test, crash, walk out to the other building to press
reset, lather, rinse, repeat.
I need a reset button on the end of a 50M wire, or a hardware watchdog!
Actually it's not so bad. If I turn off slub debugging the kernel panics
and reboots itself.
This.. :
[ 2.913034] netconsole: remote ethernet address 00:16:cb:a7:dd:d1
[ 2.913066] netconsole: device eth0 not up yet, forcing it
[ 3.660062] Refined TSC clocksource calibration: 3213.422 MHz.
[ 3.660118] Switching to clocksource tsc
[ 63.200273] r8169 0000:03:00.0: eth0: unable to load firmware patch
rtl_nic/rtl8168e-1.fw (-2)
[ 63.223513] r8169 0000:03:00.0: eth0: link down
[ 63.223556] r8169 0000:03:00.0: eth0: link down
..is slowing down reboots considerably. 3.0-rc does _not_ like some
timing hardware in my machine. Having said that, at least it does not
randomly panic on SCSI like 2.6.39 does.
Ok, I've ruled out TCPMSS. Found out where it was being set and neutered
it. I've replicated it with only the single DNAT rule.
> Could you try following patch, because this is the 'usual suspect' I had
> yesterday :
>
> diff --git a/net/core/skbuff.c b/net/core/skbuff.c
> index 46cbd28..9f548f9 100644
> --- a/net/core/skbuff.c
> +++ b/net/core/skbuff.c
> @@ -792,6 +792,7 @@ int pskb_expand_head(struct sk_buff *skb, int nhead, int ntail,
> fastpath = atomic_read(&skb_shinfo(skb)->dataref) == delta;
> }
>
> +#if 0
> if (fastpath&&
> size + sizeof(struct skb_shared_info)<= ksize(skb->head)) {
> memmove(skb->head + size, skb_shinfo(skb),
> @@ -802,7 +803,7 @@ int pskb_expand_head(struct sk_buff *skb, int nhead, int ntail,
> off = nhead;
> goto adjust_others;
> }
> -
> +#endif
> data = kmalloc(size + sizeof(struct skb_shared_info), gfp_mask);
> if (!data)
> goto nodata;
>
>
>
Nope.. that's not it. <sigh> That might have changed the characteristic
of the fault slightly, but unfortunately I got caught with a couple of
fsck's, so I only got to test it 3 times tonight.
It's unfortunate that this is a production system, so I can only take it
down between about 9pm and 1am. That would normally be pretty
productive, except that an fsck of a 14TB ext4 can take 30 minutes if it
panics at the wrong time.
I'm out of time tonight, but I'll have a crack at some bisection
tomorrow night. Now I just have to go back far enough that it works, and
be near enough not to have to futz around with /proc /sys or drivers.
I really, really, really appreciate you guys helping me with this. It
has been driving me absolutely bonkers. If I'm ever in the same town as
any of you, dinner and drinks are on me.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 111+ messages in thread
* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
@ 2011-06-08 17:02 ` Brad Campbell
0 siblings, 0 replies; 111+ messages in thread
From: Brad Campbell @ 2011-06-08 17:02 UTC (permalink / raw)
To: Eric Dumazet
Cc: Patrick McHardy, Bart De Schuymer, kvm, linux-mm, linux-kernel,
netdev, netfilter-devel
On 08/06/11 11:59, Eric Dumazet wrote:
> Well, a bisection definitely should help, but needs a lot of time in
> your case.
Yes. compile, test, crash, walk out to the other building to press
reset, lather, rinse, repeat.
I need a reset button on the end of a 50M wire, or a hardware watchdog!
Actually it's not so bad. If I turn off slub debugging the kernel panics
and reboots itself.
This.. :
[ 2.913034] netconsole: remote ethernet address 00:16:cb:a7:dd:d1
[ 2.913066] netconsole: device eth0 not up yet, forcing it
[ 3.660062] Refined TSC clocksource calibration: 3213.422 MHz.
[ 3.660118] Switching to clocksource tsc
[ 63.200273] r8169 0000:03:00.0: eth0: unable to load firmware patch
rtl_nic/rtl8168e-1.fw (-2)
[ 63.223513] r8169 0000:03:00.0: eth0: link down
[ 63.223556] r8169 0000:03:00.0: eth0: link down
..is slowing down reboots considerably. 3.0-rc does _not_ like some
timing hardware in my machine. Having said that, at least it does not
randomly panic on SCSI like 2.6.39 does.
Ok, I've ruled out TCPMSS. Found out where it was being set and neutered
it. I've replicated it with only the single DNAT rule.
> Could you try following patch, because this is the 'usual suspect' I had
> yesterday :
>
> diff --git a/net/core/skbuff.c b/net/core/skbuff.c
> index 46cbd28..9f548f9 100644
> --- a/net/core/skbuff.c
> +++ b/net/core/skbuff.c
> @@ -792,6 +792,7 @@ int pskb_expand_head(struct sk_buff *skb, int nhead, int ntail,
> fastpath = atomic_read(&skb_shinfo(skb)->dataref) == delta;
> }
>
> +#if 0
> if (fastpath&&
> size + sizeof(struct skb_shared_info)<= ksize(skb->head)) {
> memmove(skb->head + size, skb_shinfo(skb),
> @@ -802,7 +803,7 @@ int pskb_expand_head(struct sk_buff *skb, int nhead, int ntail,
> off = nhead;
> goto adjust_others;
> }
> -
> +#endif
> data = kmalloc(size + sizeof(struct skb_shared_info), gfp_mask);
> if (!data)
> goto nodata;
>
>
>
Nope.. that's not it. <sigh> That might have changed the characteristic
of the fault slightly, but unfortunately I got caught with a couple of
fsck's, so I only got to test it 3 times tonight.
It's unfortunate that this is a production system, so I can only take it
down between about 9pm and 1am. That would normally be pretty
productive, except that an fsck of a 14TB ext4 can take 30 minutes if it
panics at the wrong time.
I'm out of time tonight, but I'll have a crack at some bisection
tomorrow night. Now I just have to go back far enough that it works, and
be near enough not to have to futz around with /proc /sys or drivers.
I really, really, really appreciate you guys helping me with this. It
has been driving me absolutely bonkers. If I'm ever in the same town as
any of you, dinner and drinks are on me.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 111+ messages in thread
* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
2011-06-08 17:02 ` Brad Campbell
(?)
@ 2011-06-08 21:22 ` Eric Dumazet
-1 siblings, 0 replies; 111+ messages in thread
From: Eric Dumazet @ 2011-06-08 21:22 UTC (permalink / raw)
To: Brad Campbell
Cc: Patrick McHardy, Bart De Schuymer, kvm, linux-mm, linux-kernel,
netdev, netfilter-devel
Le jeudi 09 juin 2011 à 01:02 +0800, Brad Campbell a écrit :
> On 08/06/11 11:59, Eric Dumazet wrote:
>
> > Well, a bisection definitely should help, but needs a lot of time in
> > your case.
>
> Yes. compile, test, crash, walk out to the other building to press
> reset, lather, rinse, repeat.
>
> I need a reset button on the end of a 50M wire, or a hardware watchdog!
>
> Actually it's not so bad. If I turn off slub debugging the kernel panics
> and reboots itself.
>
> This.. :
> [ 2.913034] netconsole: remote ethernet address 00:16:cb:a7:dd:d1
> [ 2.913066] netconsole: device eth0 not up yet, forcing it
> [ 3.660062] Refined TSC clocksource calibration: 3213.422 MHz.
> [ 3.660118] Switching to clocksource tsc
> [ 63.200273] r8169 0000:03:00.0: eth0: unable to load firmware patch
> rtl_nic/rtl8168e-1.fw (-2)
> [ 63.223513] r8169 0000:03:00.0: eth0: link down
> [ 63.223556] r8169 0000:03:00.0: eth0: link down
>
> ..is slowing down reboots considerably. 3.0-rc does _not_ like some
> timing hardware in my machine. Having said that, at least it does not
> randomly panic on SCSI like 2.6.39 does.
>
> Ok, I've ruled out TCPMSS. Found out where it was being set and neutered
> it. I've replicated it with only the single DNAT rule.
>
>
> > Could you try following patch, because this is the 'usual suspect' I had
> > yesterday :
> >
> > diff --git a/net/core/skbuff.c b/net/core/skbuff.c
> > index 46cbd28..9f548f9 100644
> > --- a/net/core/skbuff.c
> > +++ b/net/core/skbuff.c
> > @@ -792,6 +792,7 @@ int pskb_expand_head(struct sk_buff *skb, int nhead, int ntail,
> > fastpath = atomic_read(&skb_shinfo(skb)->dataref) == delta;
> > }
> >
> > +#if 0
> > if (fastpath&&
> > size + sizeof(struct skb_shared_info)<= ksize(skb->head)) {
> > memmove(skb->head + size, skb_shinfo(skb),
> > @@ -802,7 +803,7 @@ int pskb_expand_head(struct sk_buff *skb, int nhead, int ntail,
> > off = nhead;
> > goto adjust_others;
> > }
> > -
> > +#endif
> > data = kmalloc(size + sizeof(struct skb_shared_info), gfp_mask);
> > if (!data)
> > goto nodata;
> >
> >
> >
>
> Nope.. that's not it. <sigh> That might have changed the characteristic
> of the fault slightly, but unfortunately I got caught with a couple of
> fsck's, so I only got to test it 3 times tonight.
>
> It's unfortunate that this is a production system, so I can only take it
> down between about 9pm and 1am. That would normally be pretty
> productive, except that an fsck of a 14TB ext4 can take 30 minutes if it
> panics at the wrong time.
>
> I'm out of time tonight, but I'll have a crack at some bisection
> tomorrow night. Now I just have to go back far enough that it works, and
> be near enough not to have to futz around with /proc /sys or drivers.
>
> I really, really, really appreciate you guys helping me with this. It
> has been driving me absolutely bonkers. If I'm ever in the same town as
> any of you, dinner and drinks are on me.
Hmm, I wonder if kmemcheck could help you, but its slow as hell, so not
appropriate for production :(
^ permalink raw reply [flat|nested] 111+ messages in thread
* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
@ 2011-06-08 21:22 ` Eric Dumazet
0 siblings, 0 replies; 111+ messages in thread
From: Eric Dumazet @ 2011-06-08 21:22 UTC (permalink / raw)
To: Brad Campbell
Cc: Patrick McHardy, Bart De Schuymer, kvm, linux-mm, linux-kernel,
netdev, netfilter-devel
Le jeudi 09 juin 2011 à 01:02 +0800, Brad Campbell a écrit :
> On 08/06/11 11:59, Eric Dumazet wrote:
>
> > Well, a bisection definitely should help, but needs a lot of time in
> > your case.
>
> Yes. compile, test, crash, walk out to the other building to press
> reset, lather, rinse, repeat.
>
> I need a reset button on the end of a 50M wire, or a hardware watchdog!
>
> Actually it's not so bad. If I turn off slub debugging the kernel panics
> and reboots itself.
>
> This.. :
> [ 2.913034] netconsole: remote ethernet address 00:16:cb:a7:dd:d1
> [ 2.913066] netconsole: device eth0 not up yet, forcing it
> [ 3.660062] Refined TSC clocksource calibration: 3213.422 MHz.
> [ 3.660118] Switching to clocksource tsc
> [ 63.200273] r8169 0000:03:00.0: eth0: unable to load firmware patch
> rtl_nic/rtl8168e-1.fw (-2)
> [ 63.223513] r8169 0000:03:00.0: eth0: link down
> [ 63.223556] r8169 0000:03:00.0: eth0: link down
>
> ..is slowing down reboots considerably. 3.0-rc does _not_ like some
> timing hardware in my machine. Having said that, at least it does not
> randomly panic on SCSI like 2.6.39 does.
>
> Ok, I've ruled out TCPMSS. Found out where it was being set and neutered
> it. I've replicated it with only the single DNAT rule.
>
>
> > Could you try following patch, because this is the 'usual suspect' I had
> > yesterday :
> >
> > diff --git a/net/core/skbuff.c b/net/core/skbuff.c
> > index 46cbd28..9f548f9 100644
> > --- a/net/core/skbuff.c
> > +++ b/net/core/skbuff.c
> > @@ -792,6 +792,7 @@ int pskb_expand_head(struct sk_buff *skb, int nhead, int ntail,
> > fastpath = atomic_read(&skb_shinfo(skb)->dataref) == delta;
> > }
> >
> > +#if 0
> > if (fastpath&&
> > size + sizeof(struct skb_shared_info)<= ksize(skb->head)) {
> > memmove(skb->head + size, skb_shinfo(skb),
> > @@ -802,7 +803,7 @@ int pskb_expand_head(struct sk_buff *skb, int nhead, int ntail,
> > off = nhead;
> > goto adjust_others;
> > }
> > -
> > +#endif
> > data = kmalloc(size + sizeof(struct skb_shared_info), gfp_mask);
> > if (!data)
> > goto nodata;
> >
> >
> >
>
> Nope.. that's not it. <sigh> That might have changed the characteristic
> of the fault slightly, but unfortunately I got caught with a couple of
> fsck's, so I only got to test it 3 times tonight.
>
> It's unfortunate that this is a production system, so I can only take it
> down between about 9pm and 1am. That would normally be pretty
> productive, except that an fsck of a 14TB ext4 can take 30 minutes if it
> panics at the wrong time.
>
> I'm out of time tonight, but I'll have a crack at some bisection
> tomorrow night. Now I just have to go back far enough that it works, and
> be near enough not to have to futz around with /proc /sys or drivers.
>
> I really, really, really appreciate you guys helping me with this. It
> has been driving me absolutely bonkers. If I'm ever in the same town as
> any of you, dinner and drinks are on me.
Hmm, I wonder if kmemcheck could help you, but its slow as hell, so not
appropriate for production :(
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 111+ messages in thread
* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
@ 2011-06-08 21:22 ` Eric Dumazet
0 siblings, 0 replies; 111+ messages in thread
From: Eric Dumazet @ 2011-06-08 21:22 UTC (permalink / raw)
To: Brad Campbell
Cc: Patrick McHardy, Bart De Schuymer, kvm, linux-mm, linux-kernel,
netdev, netfilter-devel
Le jeudi 09 juin 2011 A 01:02 +0800, Brad Campbell a A(C)crit :
> On 08/06/11 11:59, Eric Dumazet wrote:
>
> > Well, a bisection definitely should help, but needs a lot of time in
> > your case.
>
> Yes. compile, test, crash, walk out to the other building to press
> reset, lather, rinse, repeat.
>
> I need a reset button on the end of a 50M wire, or a hardware watchdog!
>
> Actually it's not so bad. If I turn off slub debugging the kernel panics
> and reboots itself.
>
> This.. :
> [ 2.913034] netconsole: remote ethernet address 00:16:cb:a7:dd:d1
> [ 2.913066] netconsole: device eth0 not up yet, forcing it
> [ 3.660062] Refined TSC clocksource calibration: 3213.422 MHz.
> [ 3.660118] Switching to clocksource tsc
> [ 63.200273] r8169 0000:03:00.0: eth0: unable to load firmware patch
> rtl_nic/rtl8168e-1.fw (-2)
> [ 63.223513] r8169 0000:03:00.0: eth0: link down
> [ 63.223556] r8169 0000:03:00.0: eth0: link down
>
> ..is slowing down reboots considerably. 3.0-rc does _not_ like some
> timing hardware in my machine. Having said that, at least it does not
> randomly panic on SCSI like 2.6.39 does.
>
> Ok, I've ruled out TCPMSS. Found out where it was being set and neutered
> it. I've replicated it with only the single DNAT rule.
>
>
> > Could you try following patch, because this is the 'usual suspect' I had
> > yesterday :
> >
> > diff --git a/net/core/skbuff.c b/net/core/skbuff.c
> > index 46cbd28..9f548f9 100644
> > --- a/net/core/skbuff.c
> > +++ b/net/core/skbuff.c
> > @@ -792,6 +792,7 @@ int pskb_expand_head(struct sk_buff *skb, int nhead, int ntail,
> > fastpath = atomic_read(&skb_shinfo(skb)->dataref) == delta;
> > }
> >
> > +#if 0
> > if (fastpath&&
> > size + sizeof(struct skb_shared_info)<= ksize(skb->head)) {
> > memmove(skb->head + size, skb_shinfo(skb),
> > @@ -802,7 +803,7 @@ int pskb_expand_head(struct sk_buff *skb, int nhead, int ntail,
> > off = nhead;
> > goto adjust_others;
> > }
> > -
> > +#endif
> > data = kmalloc(size + sizeof(struct skb_shared_info), gfp_mask);
> > if (!data)
> > goto nodata;
> >
> >
> >
>
> Nope.. that's not it. <sigh> That might have changed the characteristic
> of the fault slightly, but unfortunately I got caught with a couple of
> fsck's, so I only got to test it 3 times tonight.
>
> It's unfortunate that this is a production system, so I can only take it
> down between about 9pm and 1am. That would normally be pretty
> productive, except that an fsck of a 14TB ext4 can take 30 minutes if it
> panics at the wrong time.
>
> I'm out of time tonight, but I'll have a crack at some bisection
> tomorrow night. Now I just have to go back far enough that it works, and
> be near enough not to have to futz around with /proc /sys or drivers.
>
> I really, really, really appreciate you guys helping me with this. It
> has been driving me absolutely bonkers. If I'm ever in the same town as
> any of you, dinner and drinks are on me.
Hmm, I wonder if kmemcheck could help you, but its slow as hell, so not
appropriate for production :(
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 111+ messages in thread
* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
2011-06-08 17:02 ` Brad Campbell
@ 2011-06-10 2:52 ` Simon Horman
-1 siblings, 0 replies; 111+ messages in thread
From: Simon Horman @ 2011-06-10 2:52 UTC (permalink / raw)
To: Brad Campbell
Cc: Eric Dumazet, Patrick McHardy, Bart De Schuymer, kvm, linux-mm,
linux-kernel, netdev, netfilter-devel
On Thu, Jun 09, 2011 at 01:02:13AM +0800, Brad Campbell wrote:
> On 08/06/11 11:59, Eric Dumazet wrote:
>
> >Well, a bisection definitely should help, but needs a lot of time in
> >your case.
>
> Yes. compile, test, crash, walk out to the other building to press
> reset, lather, rinse, repeat.
>
> I need a reset button on the end of a 50M wire, or a hardware watchdog!
Not strictly on-topic, but in situations where I have machines
that either don't have lights-out facilities or have broken ones
I find that network controlled power switches to be very useful.
At one point I would have need an 8000km long wire to the reset switch :-)
^ permalink raw reply [flat|nested] 111+ messages in thread
* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
@ 2011-06-10 2:52 ` Simon Horman
0 siblings, 0 replies; 111+ messages in thread
From: Simon Horman @ 2011-06-10 2:52 UTC (permalink / raw)
To: Brad Campbell
Cc: Eric Dumazet, Patrick McHardy, Bart De Schuymer, kvm, linux-mm,
linux-kernel, netdev, netfilter-devel
On Thu, Jun 09, 2011 at 01:02:13AM +0800, Brad Campbell wrote:
> On 08/06/11 11:59, Eric Dumazet wrote:
>
> >Well, a bisection definitely should help, but needs a lot of time in
> >your case.
>
> Yes. compile, test, crash, walk out to the other building to press
> reset, lather, rinse, repeat.
>
> I need a reset button on the end of a 50M wire, or a hardware watchdog!
Not strictly on-topic, but in situations where I have machines
that either don't have lights-out facilities or have broken ones
I find that network controlled power switches to be very useful.
At one point I would have need an 8000km long wire to the reset switch :-)
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 111+ messages in thread
* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
2011-06-10 2:52 ` Simon Horman
@ 2011-06-10 12:37 ` Mark Lord
-1 siblings, 0 replies; 111+ messages in thread
From: Mark Lord @ 2011-06-10 12:37 UTC (permalink / raw)
To: Simon Horman
Cc: Brad Campbell, Eric Dumazet, Patrick McHardy, Bart De Schuymer,
kvm, linux-mm, linux-kernel, netdev, netfilter-devel
On 11-06-09 10:52 PM, Simon Horman wrote:
> On Thu, Jun 09, 2011 at 01:02:13AM +0800, Brad Campbell wrote:
>> On 08/06/11 11:59, Eric Dumazet wrote:
>>
>>> Well, a bisection definitely should help, but needs a lot of time in
>>> your case.
>>
>> Yes. compile, test, crash, walk out to the other building to press
>> reset, lather, rinse, repeat.
>>
>> I need a reset button on the end of a 50M wire, or a hardware watchdog!
Something many of us don't realize is that nearly all Intel chipsets
have a built-in hardware watchdog timer. This includes chipset for
consumer desktop boards as well as the big iron server stuff.
It's the "i8xx_tco" driver in the kernel enables use of them:
modprobe i8xx_tco
Cheers
^ permalink raw reply [flat|nested] 111+ messages in thread
* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
@ 2011-06-10 12:37 ` Mark Lord
0 siblings, 0 replies; 111+ messages in thread
From: Mark Lord @ 2011-06-10 12:37 UTC (permalink / raw)
To: Simon Horman
Cc: Brad Campbell, Eric Dumazet, Patrick McHardy, Bart De Schuymer,
kvm, linux-mm, linux-kernel, netdev, netfilter-devel
On 11-06-09 10:52 PM, Simon Horman wrote:
> On Thu, Jun 09, 2011 at 01:02:13AM +0800, Brad Campbell wrote:
>> On 08/06/11 11:59, Eric Dumazet wrote:
>>
>>> Well, a bisection definitely should help, but needs a lot of time in
>>> your case.
>>
>> Yes. compile, test, crash, walk out to the other building to press
>> reset, lather, rinse, repeat.
>>
>> I need a reset button on the end of a 50M wire, or a hardware watchdog!
Something many of us don't realize is that nearly all Intel chipsets
have a built-in hardware watchdog timer. This includes chipset for
consumer desktop boards as well as the big iron server stuff.
It's the "i8xx_tco" driver in the kernel enables use of them:
modprobe i8xx_tco
Cheers
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 111+ messages in thread
* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
2011-06-10 12:37 ` Mark Lord
@ 2011-06-10 16:43 ` Henrique de Moraes Holschuh
-1 siblings, 0 replies; 111+ messages in thread
From: Henrique de Moraes Holschuh @ 2011-06-10 16:43 UTC (permalink / raw)
To: Mark Lord
Cc: Simon Horman, Brad Campbell, Eric Dumazet, Patrick McHardy,
Bart De Schuymer, kvm, linux-mm, linux-kernel, netdev,
netfilter-devel
On Fri, 10 Jun 2011, Mark Lord wrote:
> Something many of us don't realize is that nearly all Intel chipsets
> have a built-in hardware watchdog timer. This includes chipset for
> consumer desktop boards as well as the big iron server stuff.
>
> It's the "i8xx_tco" driver in the kernel enables use of them:
That's the old module name, but yes, it is very useful in desktops and
laptops (when it works). Server-class hardware will have a baseboard
management unit that can really power-cycle the system instead of just
rebooting.
And test it first before you depend on it triggering at a remote location,
as the firmware might cause the Intel chipset watchdog to actually hang the
box instead of causing a proper reboot (happens on the IBM thinkpad T43, for
example).
--
"One disk to rule them all, One disk to find them. One disk to bring
them all and in the darkness grind them. In the Land of Redmond
where the shadows lie." -- The Silicon Valley Tarot
Henrique Holschuh
^ permalink raw reply [flat|nested] 111+ messages in thread
* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
@ 2011-06-10 16:43 ` Henrique de Moraes Holschuh
0 siblings, 0 replies; 111+ messages in thread
From: Henrique de Moraes Holschuh @ 2011-06-10 16:43 UTC (permalink / raw)
To: Mark Lord
Cc: Simon Horman, Brad Campbell, Eric Dumazet, Patrick McHardy,
Bart De Schuymer, kvm, linux-mm, linux-kernel, netdev,
netfilter-devel
On Fri, 10 Jun 2011, Mark Lord wrote:
> Something many of us don't realize is that nearly all Intel chipsets
> have a built-in hardware watchdog timer. This includes chipset for
> consumer desktop boards as well as the big iron server stuff.
>
> It's the "i8xx_tco" driver in the kernel enables use of them:
That's the old module name, but yes, it is very useful in desktops and
laptops (when it works). Server-class hardware will have a baseboard
management unit that can really power-cycle the system instead of just
rebooting.
And test it first before you depend on it triggering at a remote location,
as the firmware might cause the Intel chipset watchdog to actually hang the
box instead of causing a proper reboot (happens on the IBM thinkpad T43, for
example).
--
"One disk to rule them all, One disk to find them. One disk to bring
them all and in the darkness grind them. In the Land of Redmond
where the shadows lie." -- The Silicon Valley Tarot
Henrique Holschuh
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 111+ messages in thread
* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
2011-06-10 2:52 ` Simon Horman
@ 2011-06-12 15:38 ` Avi Kivity
-1 siblings, 0 replies; 111+ messages in thread
From: Avi Kivity @ 2011-06-12 15:38 UTC (permalink / raw)
To: Simon Horman
Cc: Brad Campbell, Eric Dumazet, Patrick McHardy, Bart De Schuymer,
kvm, linux-mm, linux-kernel, netdev, netfilter-devel
On 06/10/2011 05:52 AM, Simon Horman wrote:
> At one point I would have need an 8000km long wire to the reset switch :-)
Even more off-topic, there has been a case when a 200,000,000 km long
wire to the reset button was needed. IIRC they got away with a watchdog.
--
error compiling committee.c: too many arguments to function
^ permalink raw reply [flat|nested] 111+ messages in thread
* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
@ 2011-06-12 15:38 ` Avi Kivity
0 siblings, 0 replies; 111+ messages in thread
From: Avi Kivity @ 2011-06-12 15:38 UTC (permalink / raw)
To: Simon Horman
Cc: Brad Campbell, Eric Dumazet, Patrick McHardy, Bart De Schuymer,
kvm, linux-mm, linux-kernel, netdev, netfilter-devel
On 06/10/2011 05:52 AM, Simon Horman wrote:
> At one point I would have need an 8000km long wire to the reset switch :-)
Even more off-topic, there has been a case when a 200,000,000 km long
wire to the reset button was needed. IIRC they got away with a watchdog.
--
error compiling committee.c: too many arguments to function
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 111+ messages in thread
* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
2011-06-07 13:37 ` Eric Dumazet
2011-06-07 15:15 ` Brad Campbell
@ 2011-08-20 13:16 ` Brad Campbell
2011-08-22 6:36 ` Avi Kivity
1 sibling, 1 reply; 111+ messages in thread
From: Brad Campbell @ 2011-08-20 13:16 UTC (permalink / raw)
To: Eric Dumazet; +Cc: Avi Kivity, CaT, Borislav Petkov, linux-kernel, kvm, netdev
On 07/06/11 21:37, Eric Dumazet wrote:
> Le mardi 07 juin 2011 à 21:27 +0800, Brad Campbell a écrit :
>> On 07/06/11 04:22, Eric Dumazet wrote:
>>
>>> Could you please try latest linux-2.6 tree ?
>>>
>>> We fixed many networking bugs that could explain your crash.
>>>
>>>
>>>
>>>
>>
>> No good I'm afraid.
>>
>> [ 543.040056]
>> =============================================================================
>> [ 543.040136] BUG ip_dst_cache: Padding overwritten.
>> 0xffff8803e4217ffe-0xffff8803e4217fff
>> [ 543.040194]
>
> Thats pretty strange : These are the last two bytes of a page, set to
> 0x0000 (a 16 bit value)
>
> There is no way a dst field could actually sit on this location (its a
> padding), since a dst is a bit less than 256 bytes (0xe8), and each
> entry is aligned on a 64byte address.
>
> grep dst /proc/slabinfo
>
> ip_dst_cache 32823 62944 256 32 2 : tunables 0 0
> 0 : slabdata 1967 1967 0
>
> sizeof(struct rtable)=0xe8
>
>
>> -----------------------------------------------------------------------------
>> [ 543.040198]
>> [ 543.040298] INFO: Slab 0xffffea000d9e74d0 objects=25 used=25 fp=0x
>> (null) flags=0x8000000000004081
>> [ 543.040364] Pid: 4576, comm: kworker/1:2 Not tainted 3.0.0-rc2 #1
>> [ 543.040415] Call Trace:
>> [ 543.040472] [<ffffffff810b9c1d>] ? slab_err+0xad/0xd0
>> [ 543.040528] [<ffffffff8102e034>] ? check_preempt_wakeup+0xa4/0x160
>> [ 543.040595] [<ffffffff810ba206>] ? slab_pad_check+0x126/0x170
>> [ 543.040650] [<ffffffff8133045b>] ? dst_destroy+0x8b/0x110
>> [ 543.040701] [<ffffffff810ba29a>] ? check_slab+0x4a/0xc0
>> [ 543.040753] [<ffffffff810baf2d>] ? free_debug_processing+0x2d/0x250
>> [ 543.040808] [<ffffffff810bb27b>] ? __slab_free+0x12b/0x140
>> [ 543.040862] [<ffffffff810bbe99>] ? kmem_cache_free+0x99/0xa0
>> [ 543.040915] [<ffffffff8133045b>] ? dst_destroy+0x8b/0x110
>> [ 543.040967] [<ffffffff813307f6>] ? dst_gc_task+0x196/0x1f0
>> [ 543.041021] [<ffffffff8104e954>] ? queue_delayed_work_on+0x154/0x160
>> [ 543.041081] [<ffffffff813066fe>] ? do_dbs_timer+0x20e/0x3d0
>> [ 543.041133] [<ffffffff81330660>] ? dst_alloc+0x180/0x180
>> [ 543.041187] [<ffffffff8104f28b>] ? process_one_work+0xfb/0x3b0
>> [ 543.041242] [<ffffffff8104f964>] ? worker_thread+0x144/0x3d0
>> [ 543.041296] [<ffffffff8102cc10>] ? __wake_up_common+0x50/0x80
>> [ 543.041678] [<ffffffff8104f820>] ? rescuer_thread+0x2e0/0x2e0
>> [ 543.041729] [<ffffffff8104f820>] ? rescuer_thread+0x2e0/0x2e0
>> [ 543.041782] [<ffffffff81053436>] ? kthread+0x96/0xa0
>> [ 543.041835] [<ffffffff813e1d14>] ? kernel_thread_helper+0x4/0x10
>> [ 543.041890] [<ffffffff810533a0>] ? kthread_worker_fn+0x120/0x120
>> [ 543.041944] [<ffffffff813e1d10>] ? gs_change+0xb/0xb
>> [ 543.041993] Padding 0xffff8803e4217f40: 5a 5a 5a 5a 5a 5a 5a 5a 5a
>> 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
>> [ 543.042718] Padding 0xffff8803e4217f50: 5a 5a 5a 5a 5a 5a 5a 5a 5a
>> 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
>> [ 543.043433] Padding 0xffff8803e4217f60: 5a 5a 5a 5a 5a 5a 5a 5a 5a
>> 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
>> [ 543.044155] Padding 0xffff8803e4217f70: 5a 5a 5a 5a 5a 5a 5a 5a 5a
>> 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
>> [ 543.044866] Padding 0xffff8803e4217f80: 5a 5a 5a 5a 5a 5a 5a 5a 5a
>> 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
>> [ 543.045590] Padding 0xffff8803e4217f90: 5a 5a 5a 5a 5a 5a 5a 5a 5a
>> 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
>> [ 543.046311] Padding 0xffff8803e4217fa0: 5a 5a 5a 5a 5a 5a 5a 5a 5a
>> 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
>> [ 543.047034] Padding 0xffff8803e4217fb0: 5a 5a 5a 5a 5a 5a 5a 5a 5a
>> 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
>> [ 543.047755] Padding 0xffff8803e4217fc0: 5a 5a 5a 5a 5a 5a 5a 5a 5a
>> 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
>> [ 543.048474] Padding 0xffff8803e4217fd0: 5a 5a 5a 5a 5a 5a 5a 5a 5a
>> 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
>> [ 543.049203] Padding 0xffff8803e4217fe0: 5a 5a 5a 5a 5a 5a 5a 5a 5a
>> 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
>> [ 543.049909] Padding 0xffff8803e4217ff0: 5a 5a 5a 5a 5a 5a 5a 5a 5a
>> 5a 5a 5a 5a 5a 00 00 ZZZZZZZZZZZZZZ..
>> [ 543.050021] FIX ip_dst_cache: Restoring
>> 0xffff8803e4217f40-0xffff8803e4217fff=0x5a
>> [ 543.050021]
>>
>> Dropped -mm, Hugh and Andrea from CC as this does not appear to be mm or
>> ksm related.
>>
>> I'll pare down the firewall and see if I can make it break easier with a
>> smaller test set.
>
> Hmm, not sure now :(
>
> Could you reproduce another bug please ?
I know this is an old one, but I recently purchased a second system to
allow me to test and bisect this off-line (the live system is too much
of a headache to bisect on).
brad@test:/raid10/src/linux-2.6$ git bisect log
git bisect start
# good: [9fe6206f400646a2322096b56c59891d530e8d51] Linux 2.6.35
git bisect good 9fe6206f400646a2322096b56c59891d530e8d51
# bad: [da5cabf80e2433131bf0ed8993abc0f7ea618c73] Linux 2.6.36-rc1
git bisect bad da5cabf80e2433131bf0ed8993abc0f7ea618c73
# bad: [0f477dd0851bdcee82923da66a7fc4a44cb1bc3d] Merge branch
'x86-cpu-for-linus' of
git://git.kernel.org/pub/scm/linux/kernel/git/tip/linux-2.6-tip
git bisect bad 0f477dd0851bdcee82923da66a7fc4a44cb1bc3d
# bad: [3ff1c25927e3af61c6bf0e4ed959504058ae4565] phy/marvell: add
88ec048 support
git bisect bad 3ff1c25927e3af61c6bf0e4ed959504058ae4565
# good: [05318bc905467237d4aa68a701f6e92a2b332218] Merge branch 'master'
of git://git.kernel.org/pub/scm/linux/kernel/git/linville/wireless-next-2.6
git bisect good 05318bc905467237d4aa68a701f6e92a2b332218
# bad: [2ba13ed678775195e8255b4e503c59d48b615bd8] Bluetooth: Remove
check for supported mode
git bisect bad 2ba13ed678775195e8255b4e503c59d48b615bd8
# bad: [1e2cfeef060fa0270f9a2d66b1218c12c05062e0] Revert "tc35815: fix
iomap leak"
git bisect bad 1e2cfeef060fa0270f9a2d66b1218c12c05062e0
# bad: [d9bed6bbd4f2a0120c93fed68605950651e1f225] isdn/gigaset: remove
EXPERIMENTAL tag from GIGASET_CAPI
git bisect bad d9bed6bbd4f2a0120c93fed68605950651e1f225
# bad: [d117b6665847084cfe8a44b870f771153e18991d] fealnx: Use the
instance of net_device_stats from net_device.
git bisect bad d117b6665847084cfe8a44b870f771153e18991d
# bad: [e490c1defec4236a6a131fe2d13bf7ba787c02f8] Merge branch 'master'
of git://git.kernel.org/pub/scm/linux/kernel/git/kaber/nf-next-2.6
git bisect bad e490c1defec4236a6a131fe2d13bf7ba787c02f8
# bad: [0a17d8c744e44617a3c22e7af68b4c5c9c1c5dba] ixgbe: use NETIF_F_LRO
git bisect bad 0a17d8c744e44617a3c22e7af68b4c5c9c1c5dba
# bad: [ede3ef0d940ef052466f42c849390b23c6859abc] igb: fix PHY config
access on 82580
git bisect bad ede3ef0d940ef052466f42c849390b23c6859abc
# good: [ee3cb6295144b0adfa75ccaca307643a6998b1e2] be2net: changes to
properly provide phy details
git bisect good ee3cb6295144b0adfa75ccaca307643a6998b1e2
# bad: [7475271004b66e9c22e1bb28f240a38c5d6fe76e] x86: Drop
CONFIG_MCORE2 check around setting of NET_IP_ALIGN
git bisect bad 7475271004b66e9c22e1bb28f240a38c5d6fe76e
brad@test:/raid10/src/linux-2.6$ git bisect log
git bisect start
# good: [9fe6206f400646a2322096b56c59891d530e8d51] Linux 2.6.35
git bisect good 9fe6206f400646a2322096b56c59891d530e8d51
# bad: [da5cabf80e2433131bf0ed8993abc0f7ea618c73] Linux 2.6.36-rc1
git bisect bad da5cabf80e2433131bf0ed8993abc0f7ea618c73
# bad: [0f477dd0851bdcee82923da66a7fc4a44cb1bc3d] Merge branch
'x86-cpu-for-linus' of
git://git.kernel.org/pub/scm/linux/kernel/git/tip/linux-2.6-tip
git bisect bad 0f477dd0851bdcee82923da66a7fc4a44cb1bc3d
# bad: [3ff1c25927e3af61c6bf0e4ed959504058ae4565] phy/marvell: add
88ec048 support
git bisect bad 3ff1c25927e3af61c6bf0e4ed959504058ae4565
# good: [05318bc905467237d4aa68a701f6e92a2b332218] Merge branch 'master'
of git://git.kernel.org/pub/scm/linux/kernel/git/linville/wireless-next-2.6
git bisect good 05318bc905467237d4aa68a701f6e92a2b332218
# bad: [2ba13ed678775195e8255b4e503c59d48b615bd8] Bluetooth: Remove
check for supported mode
git bisect bad 2ba13ed678775195e8255b4e503c59d48b615bd8
# bad: [1e2cfeef060fa0270f9a2d66b1218c12c05062e0] Revert "tc35815: fix
iomap leak"
git bisect bad 1e2cfeef060fa0270f9a2d66b1218c12c05062e0
# bad: [d9bed6bbd4f2a0120c93fed68605950651e1f225] isdn/gigaset: remove
EXPERIMENTAL tag from GIGASET_CAPI
git bisect bad d9bed6bbd4f2a0120c93fed68605950651e1f225
# bad: [d117b6665847084cfe8a44b870f771153e18991d] fealnx: Use the
instance of net_device_stats from net_device.
git bisect bad d117b6665847084cfe8a44b870f771153e18991d
# bad: [e490c1defec4236a6a131fe2d13bf7ba787c02f8] Merge branch 'master'
of git://git.kernel.org/pub/scm/linux/kernel/git/kaber/nf-next-2.6
git bisect bad e490c1defec4236a6a131fe2d13bf7ba787c02f8
# bad: [0a17d8c744e44617a3c22e7af68b4c5c9c1c5dba] ixgbe: use NETIF_F_LRO
git bisect bad 0a17d8c744e44617a3c22e7af68b4c5c9c1c5dba
# bad: [ede3ef0d940ef052466f42c849390b23c6859abc] igb: fix PHY config
access on 82580
git bisect bad ede3ef0d940ef052466f42c849390b23c6859abc
# good: [ee3cb6295144b0adfa75ccaca307643a6998b1e2] be2net: changes to
properly provide phy details
git bisect good ee3cb6295144b0adfa75ccaca307643a6998b1e2
# bad: [7475271004b66e9c22e1bb28f240a38c5d6fe76e] x86: Drop
CONFIG_MCORE2 check around setting of NET_IP_ALIGN
git bisect bad 7475271004b66e9c22e1bb28f240a38c5d6fe76e
brad@test:/raid10/src/linux-2.6$ git bisect good
7475271004b66e9c22e1bb28f240a38c5d6fe76e is the first bad commit
commit 7475271004b66e9c22e1bb28f240a38c5d6fe76e
Author: Alexander Duyck <alexander.h.duyck@intel.com>
Date: Thu Jul 1 13:28:27 2010 +0000
x86: Drop CONFIG_MCORE2 check around setting of NET_IP_ALIGN
This patch removes the CONFIG_MCORE2 check from around
NET_IP_ALIGN. It is
based on a suggestion from Andi Kleen. The assumption is that
there are
not any x86 cores where unaligned access is really slow, and this
change
would allow for a performance improvement to still exist on
configurations
that are not necessarily optimized for Core 2.
Cc: Andi Kleen <ak@linux.intel.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: "H. Peter Anvin" <hpa@zytor.com>
Cc: x86@kernel.org
Signed-off-by: Alexander Duyck <alexander.h.duyck@intel.com>
Signed-off-by: Jeff Kirsher <jeffrey.t.kirsher@intel.com>
Acked-by: H. Peter Anvin <hpa@zytor.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
:040000 040000 5a15867789080a2f67a74b17c4422f85b7a9fb4a
b98769348bd765731ca3ff03b33764257e23226c M arch
I can confirm this bug exists in the 3.0 kernel, however I'm unable to
reproduce it on todays git.
So anyone using netfilter, kvm and bridge on kernels between 2.6.36-rc1
and 3.0 may hit this bug, but it looks like it is fixed in the current
3.1-rc kernels.
^ permalink raw reply [flat|nested] 111+ messages in thread
* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
2011-08-20 13:16 ` Brad Campbell
@ 2011-08-22 6:36 ` Avi Kivity
2011-08-22 6:45 ` Eric Dumazet
0 siblings, 1 reply; 111+ messages in thread
From: Avi Kivity @ 2011-08-22 6:36 UTC (permalink / raw)
To: Brad Campbell
Cc: Eric Dumazet, CaT, Borislav Petkov, linux-kernel, kvm, netdev
On 08/20/2011 04:16 PM, Brad Campbell wrote:
> Author: Alexander Duyck <alexander.h.duyck@intel.com>
> Date: Thu Jul 1 13:28:27 2010 +0000
>
> x86: Drop CONFIG_MCORE2 check around setting of NET_IP_ALIGN
>
> This patch removes the CONFIG_MCORE2 check from around
> NET_IP_ALIGN. It is
> based on a suggestion from Andi Kleen. The assumption is that
> there are
> not any x86 cores where unaligned access is really slow, and this
> change
> would allow for a performance improvement to still exist on
> configurations
> that are not necessarily optimized for Core 2.
>
> Cc: Andi Kleen <ak@linux.intel.com>
> Cc: Thomas Gleixner <tglx@linutronix.de>
> Cc: Ingo Molnar <mingo@redhat.com>
> Cc: "H. Peter Anvin" <hpa@zytor.com>
> Cc: x86@kernel.org
> Signed-off-by: Alexander Duyck <alexander.h.duyck@intel.com>
> Signed-off-by: Jeff Kirsher <jeffrey.t.kirsher@intel.com>
> Acked-by: H. Peter Anvin <hpa@zytor.com>
> Signed-off-by: David S. Miller <davem@davemloft.net>
>
> :040000 040000 5a15867789080a2f67a74b17c4422f85b7a9fb4a
> b98769348bd765731ca3ff03b33764257e23226c M arch
>
> I can confirm this bug exists in the 3.0 kernel, however I'm unable to
> reproduce it on todays git.
>
> So anyone using netfilter, kvm and bridge on kernels between
> 2.6.36-rc1 and 3.0 may hit this bug, but it looks like it is fixed in
> the current 3.1-rc kernels.
>
Thanks for this effort. I don't think this patch is buggy in itself, it
merely exposed another bug which was fixed later on.
--
I have a truly marvellous patch that fixes the bug which this
signature is too narrow to contain.
^ permalink raw reply [flat|nested] 111+ messages in thread
* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
2011-08-22 6:36 ` Avi Kivity
@ 2011-08-22 6:45 ` Eric Dumazet
2011-08-22 11:45 ` Brad Campbell
0 siblings, 1 reply; 111+ messages in thread
From: Eric Dumazet @ 2011-08-22 6:45 UTC (permalink / raw)
To: Avi Kivity; +Cc: Brad Campbell, CaT, Borislav Petkov, linux-kernel, kvm, netdev
Le lundi 22 août 2011 à 09:36 +0300, Avi Kivity a écrit :
> On 08/20/2011 04:16 PM, Brad Campbell wrote:
> > Author: Alexander Duyck <alexander.h.duyck@intel.com>
> > Date: Thu Jul 1 13:28:27 2010 +0000
> >
> > x86: Drop CONFIG_MCORE2 check around setting of NET_IP_ALIGN
> >
> > This patch removes the CONFIG_MCORE2 check from around
> > NET_IP_ALIGN. It is
> > based on a suggestion from Andi Kleen. The assumption is that
> > there are
> > not any x86 cores where unaligned access is really slow, and this
> > change
> > would allow for a performance improvement to still exist on
> > configurations
> > that are not necessarily optimized for Core 2.
> >
> > Cc: Andi Kleen <ak@linux.intel.com>
> > Cc: Thomas Gleixner <tglx@linutronix.de>
> > Cc: Ingo Molnar <mingo@redhat.com>
> > Cc: "H. Peter Anvin" <hpa@zytor.com>
> > Cc: x86@kernel.org
> > Signed-off-by: Alexander Duyck <alexander.h.duyck@intel.com>
> > Signed-off-by: Jeff Kirsher <jeffrey.t.kirsher@intel.com>
> > Acked-by: H. Peter Anvin <hpa@zytor.com>
> > Signed-off-by: David S. Miller <davem@davemloft.net>
> >
> > :040000 040000 5a15867789080a2f67a74b17c4422f85b7a9fb4a
> > b98769348bd765731ca3ff03b33764257e23226c M arch
> >
> > I can confirm this bug exists in the 3.0 kernel, however I'm unable to
> > reproduce it on todays git.
> >
> > So anyone using netfilter, kvm and bridge on kernels between
> > 2.6.36-rc1 and 3.0 may hit this bug, but it looks like it is fixed in
> > the current 3.1-rc kernels.
> >
>
> Thanks for this effort. I don't think this patch is buggy in itself, it
> merely exposed another bug which was fixed later on.
>
Some piece of hardware has a 2-byte offset requirement, and driver
incorrectly assumed NET_IP_ALIGN was 2 on x86.
Brad, could you post your config (lsmod, dmesg) again ?
tg3.c code for example uses a private value, not related to NET_IP_ALIGN
#define TG3_RAW_IP_ALIGN 2
^ permalink raw reply [flat|nested] 111+ messages in thread
* Re: KVM induced panic on 2.6.38[2367] & 2.6.39
2011-08-22 6:45 ` Eric Dumazet
@ 2011-08-22 11:45 ` Brad Campbell
0 siblings, 0 replies; 111+ messages in thread
From: Brad Campbell @ 2011-08-22 11:45 UTC (permalink / raw)
To: Eric Dumazet; +Cc: Avi Kivity, CaT, Borislav Petkov, linux-kernel, kvm, netdev
On 22/08/11 14:45, Eric Dumazet wrote:
> Le lundi 22 août 2011 à 09:36 +0300, Avi Kivity a écrit :
>> On 08/20/2011 04:16 PM, Brad Campbell wrote:
>>> Author: Alexander Duyck<alexander.h.duyck@intel.com>
>>> Date: Thu Jul 1 13:28:27 2010 +0000
>>>
>>> x86: Drop CONFIG_MCORE2 check around setting of NET_IP_ALIGN
>>>
>>> This patch removes the CONFIG_MCORE2 check from around
>>> NET_IP_ALIGN. It is
>>> based on a suggestion from Andi Kleen. The assumption is that
>>> there are
>>> not any x86 cores where unaligned access is really slow, and this
>>> change
>>> would allow for a performance improvement to still exist on
>>> configurations
>>> that are not necessarily optimized for Core 2.
>>>
>>> Cc: Andi Kleen<ak@linux.intel.com>
>>> Cc: Thomas Gleixner<tglx@linutronix.de>
>>> Cc: Ingo Molnar<mingo@redhat.com>
>>> Cc: "H. Peter Anvin"<hpa@zytor.com>
>>> Cc: x86@kernel.org
>>> Signed-off-by: Alexander Duyck<alexander.h.duyck@intel.com>
>>> Signed-off-by: Jeff Kirsher<jeffrey.t.kirsher@intel.com>
>>> Acked-by: H. Peter Anvin<hpa@zytor.com>
>>> Signed-off-by: David S. Miller<davem@davemloft.net>
>>>
>>> :040000 040000 5a15867789080a2f67a74b17c4422f85b7a9fb4a
>>> b98769348bd765731ca3ff03b33764257e23226c M arch
>>>
>>> I can confirm this bug exists in the 3.0 kernel, however I'm unable to
>>> reproduce it on todays git.
>>>
>>> So anyone using netfilter, kvm and bridge on kernels between
>>> 2.6.36-rc1 and 3.0 may hit this bug, but it looks like it is fixed in
>>> the current 3.1-rc kernels.
>>>
>>
>> Thanks for this effort. I don't think this patch is buggy in itself, it
>> merely exposed another bug which was fixed later on.
>>
>
> Some piece of hardware has a 2-byte offset requirement, and driver
> incorrectly assumed NET_IP_ALIGN was 2 on x86.
>
> Brad, could you post your config (lsmod, dmesg) again ?
>
> tg3.c code for example uses a private value, not related to NET_IP_ALIGN
>
> #define TG3_RAW_IP_ALIGN 2
Both sets of hardware were using Marvell 88e1116 chips (the r8169 driver).
Interestingly on the hardware I just used to reproduce the fault the
network path is VM1 -> br1 -> eth0 -> DNAT -> br1 -> VM2
On the machine that originally exhibited the fault the network path was
VM1 -> br1 -> ppp0 -> DNAT -> br1 -> VM2
So on the original machine the packet never even goes near a physical
interface.
The problem only exhibited itself if I tried to use the address of eth0
or ppp0 to get to VM2 from VM1. If I went dirctly from VM1 to VM2 using
local addresses and never traversed netfilter it never paniced.
I also agree the patch I identified simply unmasked an underlying issue,
and of course on anything later than the linus 3.0 kernel I can't
reproduce it now anyway.
It's nice to know its fixed, but it'd be nice to know *what* fixed it
rather than it just went away after some code re-factoring.
I did have a perverse thought about doing a bisect between 3.0.0 and
3.1-rc2 to see precisely *what* fixed the problem, but the thought of
another 7 hours of crash and burn is not overly appealing.
[ 0.000000] Linux version 3.0.0 (brad@srv) (gcc version 4.4.5 (Debian
4.4.5-8) ) #5 SMP Mon Jul 25 22:43:34 WST 2011
[ 0.000000] Command line: BOOT_IMAGE=/bzImage
root=UUID=9162858d-df5e-43d6-8331-6d2b65082b4c ro panic=5
netconsole=6666@192.168.2.1/eth0,5050@192.168.2.87/00:16:cb:a7:dd:d1
[ 0.000000] BIOS-provided physical RAM map:
[ 0.000000] BIOS-e820: 0000000000000000 - 000000000009c400 (usable)
[ 0.000000] BIOS-e820: 000000000009c400 - 00000000000a0000 (reserved)
[ 0.000000] BIOS-e820: 00000000000e4000 - 0000000000100000 (reserved)
[ 0.000000] BIOS-e820: 0000000000100000 - 00000000dbf80000 (usable)
[ 0.000000] BIOS-e820: 00000000dbf80000 - 00000000dffa0000 (reserved)
[ 0.000000] BIOS-e820: 00000000dffa0000 - 00000000dffb0000 (ACPI data)
[ 0.000000] BIOS-e820: 00000000dffb0000 - 00000000dffe0000 (ACPI NVS)
[ 0.000000] BIOS-e820: 00000000dffe0000 - 00000000e0000000 (reserved)
[ 0.000000] BIOS-e820: 00000000fff00000 - 0000000100000000 (reserved)
[ 0.000000] BIOS-e820: 0000000100000000 - 0000000420000000 (usable)
[ 0.000000] NX (Execute Disable) protection: active
[ 0.000000] DMI present.
[ 0.000000] DMI: To Be Filled By O.E.M. To Be Filled By O.E.M./880G
Extreme3, BIOS P1.70 06/21/2010
[ 0.000000] e820 update range: 0000000000000000 - 0000000000010000
(usable) ==> (reserved)
[ 0.000000] e820 remove range: 00000000000a0000 - 0000000000100000
(usable)
[ 0.000000] No AGP bridge found
[ 0.000000] last_pfn = 0x420000 max_arch_pfn = 0x400000000
[ 0.000000] MTRR default type: uncachable
[ 0.000000] MTRR fixed ranges enabled:
[ 0.000000] 00000-9FFFF write-back
[ 0.000000] A0000-EFFFF uncachable
[ 0.000000] F0000-FFFFF write-protect
[ 0.000000] MTRR variable ranges enabled:
[ 0.000000] 0 base 000000000000 mask FFFF80000000 write-back
[ 0.000000] 1 base 000080000000 mask FFFFC0000000 write-back
[ 0.000000] 2 base 0000C0000000 mask FFFFE0000000 write-back
[ 0.000000] 3 disabled
[ 0.000000] 4 disabled
[ 0.000000] 5 disabled
[ 0.000000] 6 disabled
[ 0.000000] 7 disabled
[ 0.000000] TOM2: 0000000420000000 aka 16896M
[ 0.000000] x86 PAT enabled: cpu 0, old 0x7040600070406, new
0x7010600070106
[ 0.000000] e820 update range: 00000000e0000000 - 0000000100000000
(usable) ==> (reserved)
[ 0.000000] last_pfn = 0xdbf80 max_arch_pfn = 0x400000000
[ 0.000000] initial memory mapped : 0 - 20000000
[ 0.000000] Base memory trampoline at [ffff88000009a000] 9a000 size 8192
[ 0.000000] Using GB pages for direct mapping
[ 0.000000] init_memory_mapping: 0000000000000000-00000000dbf80000
[ 0.000000] 0000000000 - 00c0000000 page 1G
[ 0.000000] 00c0000000 - 00dbe00000 page 2M
[ 0.000000] 00dbe00000 - 00dbf80000 page 4k
[ 0.000000] kernel direct mapping tables up to dbf80000 @
dbf7d000-dbf80000
[ 0.000000] init_memory_mapping: 0000000100000000-0000000420000000
[ 0.000000] 0100000000 - 0400000000 page 1G
[ 0.000000] 0400000000 - 0420000000 page 2M
[ 0.000000] kernel direct mapping tables up to 420000000 @
41fffe000-420000000
[ 0.000000] RAMDISK: 37cd8000 - 37ff0000
[ 0.000000] ACPI: RSDP 00000000000fa940 00014 (v00 ACPIAM)
[ 0.000000] ACPI: RSDT 00000000dffa0000 00044 (v01 062110 RSDT2249
20100621 MSFT 00000097)
[ 0.000000] ACPI: FACP 00000000dffa0200 00084 (v01 A_M_I OEMFACP
12000601 MSFT 00000097)
[ 0.000000] ACPI: DSDT 00000000dffa0450 07D9F (v01 AS368 AS368164
00000164 INTL 20051117)
[ 0.000000] ACPI: FACS 00000000dffb0000 00040
[ 0.000000] ACPI: APIC 00000000dffa0390 0007C (v01 062110 APIC2249
20100621 MSFT 00000097)
[ 0.000000] ACPI: MCFG 00000000dffa0410 0003C (v01 062110 OEMMCFG
20100621 MSFT 00000097)
[ 0.000000] ACPI: OEMB 00000000dffb0040 00072 (v01 062110 OEMB2249
20100621 MSFT 00000097)
[ 0.000000] ACPI: SRAT 00000000dffaa450 00108 (v03 AMD FAM_F_10
00000002 AMD 00000001)
[ 0.000000] ACPI: AAFT 00000000dffaa560 00027 (v01 062110 OEMAAFT
20100621 MSFT 00000097)
[ 0.000000] ACPI: HPET 00000000dffaa590 00038 (v01 062110 OEMHPET
20100621 MSFT 00000097)
[ 0.000000] ACPI: SSDT 00000000dffaa5d0 00DA4 (v01 A M I POWERNOW
00000001 AMD 00000001)
[ 0.000000] ACPI: Local APIC address 0xfee00000
[ 0.000000] [ffffea0000000000-ffffea000e7fffff] PMD ->
[ffff88040f600000-ffff88041d7fffff] on node 0
[ 0.000000] Zone PFN ranges:
[ 0.000000] DMA 0x00000010 -> 0x00001000
[ 0.000000] DMA32 0x00001000 -> 0x00100000
[ 0.000000] Normal 0x00100000 -> 0x00420000
[ 0.000000] Movable zone start PFN for each node
[ 0.000000] early_node_map[3] active PFN ranges
[ 0.000000] 0: 0x00000010 -> 0x0000009c
[ 0.000000] 0: 0x00000100 -> 0x000dbf80
[ 0.000000] 0: 0x00100000 -> 0x00420000
[ 0.000000] On node 0 totalpages: 4177676
[ 0.000000] DMA zone: 56 pages used for memmap
[ 0.000000] DMA zone: 2 pages reserved
[ 0.000000] DMA zone: 3922 pages, LIFO batch:0
[ 0.000000] DMA32 zone: 14280 pages used for memmap
[ 0.000000] DMA32 zone: 882616 pages, LIFO batch:31
[ 0.000000] Normal zone: 44800 pages used for memmap
[ 0.000000] Normal zone: 3232000 pages, LIFO batch:31
[ 0.000000] ACPI: PM-Timer IO Port: 0x808
[ 0.000000] ACPI: Local APIC address 0xfee00000
[ 0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled)
[ 0.000000] ACPI: LAPIC (acpi_id[0x02] lapic_id[0x01] enabled)
[ 0.000000] ACPI: LAPIC (acpi_id[0x03] lapic_id[0x02] enabled)
[ 0.000000] ACPI: LAPIC (acpi_id[0x04] lapic_id[0x03] enabled)
[ 0.000000] ACPI: LAPIC (acpi_id[0x05] lapic_id[0x04] enabled)
[ 0.000000] ACPI: LAPIC (acpi_id[0x06] lapic_id[0x05] enabled)
[ 0.000000] ACPI: IOAPIC (id[0x06] address[0xfec00000] gsi_base[0])
[ 0.000000] IOAPIC[0]: apic_id 6, version 33, address 0xfec00000, GSI
0-23
[ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
[ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 low level)
[ 0.000000] ACPI: IRQ0 used by override.
[ 0.000000] ACPI: IRQ2 used by override.
[ 0.000000] ACPI: IRQ9 used by override.
[ 0.000000] Using ACPI (MADT) for SMP configuration information
[ 0.000000] ACPI: HPET id: 0x8300 base: 0xfed00000
[ 0.000000] SMP: Allowing 6 CPUs, 0 hotplug CPUs
[ 0.000000] nr_irqs_gsi: 40
[ 0.000000] Allocating PCI resources starting at e0000000 (gap:
e0000000:1ff00000)
[ 0.000000] setup_percpu: NR_CPUS:8 nr_cpumask_bits:8 nr_cpu_ids:6
nr_node_ids:1
[ 0.000000] PERCPU: Embedded 24 pages/cpu @ffff88041fc00000 s69248
r8192 d20864 u262144
[ 0.000000] pcpu-alloc: s69248 r8192 d20864 u262144 alloc=1*2097152
[ 0.000000] pcpu-alloc: [0] 0 1 2 3 4 5 - -
[ 0.000000] Built 1 zonelists in Zone order, mobility grouping on.
Total pages: 4118538
[ 0.000000] Kernel command line: BOOT_IMAGE=/bzImage
root=UUID=9162858d-df5e-43d6-8331-6d2b65082b4c ro panic=5
netconsole=6666@192.168.2.1/eth0,5050@192.168.2.87/00:16:cb:a7:dd:d1
[ 0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes)
[ 0.000000] Dentry cache hash table entries: 2097152 (order: 12,
16777216 bytes)
[ 0.000000] Inode-cache hash table entries: 1048576 (order: 11,
8388608 bytes)
[ 0.000000] Checking aperture...
[ 0.000000] No AGP bridge found
[ 0.000000] Node 0: aperture @ d21e000000 size 32 MB
[ 0.000000] Aperture beyond 4GB. Ignoring.
[ 0.000000] Your BIOS doesn't leave a aperture memory hole
[ 0.000000] Please enable the IOMMU option in the BIOS setup
[ 0.000000] This costs you 64 MB of RAM
[ 0.000000] Mapping aperture over 65536 KB of RAM @ d0000000
[ 0.000000] Memory: 16310976k/17301504k available (4036k kernel code,
590800k absent, 399728k reserved, 2027k data, 468k init)
[ 0.000000] SLUB: Genslabs=15, HWalign=64, Order=0-3, MinObjects=0,
CPUs=6, Nodes=1
[ 0.000000] Hierarchical RCU implementation.
[ 0.000000] NR_IRQS:512
[ 0.000000] spurious 8259A interrupt: IRQ7.
[ 0.000000] Console: colour VGA+ 80x25
[ 0.000000] console [tty0] enabled
[ 0.000000] hpet clockevent registered
[ 0.000000] Fast TSC calibration using PIT
[ 0.000000] Detected 3213.157 MHz processor.
[ 0.000003] Calibrating delay loop (skipped), value calculated using
timer frequency.. 6426.31 BogoMIPS (lpj=32131570)
[ 0.000064] pid_max: default: 32768 minimum: 301
[ 0.000126] Mount-cache hash table entries: 256
[ 0.000267] tseg: 0000000000
[ 0.000272] CPU: Physical Processor ID: 0
[ 0.000304] CPU: Processor Core ID: 0
[ 0.000334] mce: CPU supports 6 MCE banks
[ 0.000420] Freeing SMP alternatives: 20k freed
[ 0.000456] ACPI: Core revision 20110413
[ 0.012256] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
[ 0.112297] CPU0: AMD Phenom(tm) II X6 1090T Processor stepping 00
[ 0.120000] Performance Events: AMD PMU driver.
[ 0.120000] ... version: 0
[ 0.120000] ... bit width: 48
[ 0.120000] ... generic registers: 4
[ 0.120000] ... value mask: 0000ffffffffffff
[ 0.120000] ... max period: 00007fffffffffff
[ 0.120000] ... fixed-purpose events: 0
[ 0.120000] ... event mask: 000000000000000f
[ 0.120000] Booting Node 0, Processors #1
[ 0.120000] smpboot cpu 1: start_ip = 9a000
[ 0.280028] #2
[ 0.280056] smpboot cpu 2: start_ip = 9a000
[ 0.450026] #3
[ 0.450054] smpboot cpu 3: start_ip = 9a000
[ 0.620025] #4
[ 0.620066] smpboot cpu 4: start_ip = 9a000
[ 0.790027] #5 Ok.
[ 0.790057] smpboot cpu 5: start_ip = 9a000
[ 0.960003] Brought up 6 CPUs
[ 0.960033] Total of 6 processors activated (38560.60 BogoMIPS).
[ 0.963391] devtmpfs: initialized
[ 0.963391] xor: automatically using best checksumming function:
generic_sse
[ 1.009895] generic_sse: 14408.400 MB/sec
[ 1.009928] xor: using function: generic_sse (14408.400 MB/sec)
[ 1.010040] NET: Registered protocol family 16
[ 1.010079] node 0 link 0: io port [1000, ffffff]
[ 1.010079] TOM: 00000000e0000000 aka 3584M
[ 1.010079] Fam 10h mmconf [e0000000, efffffff]
[ 1.010079] node 0 link 0: mmio [a0000, bffff]
[ 1.010079] node 0 link 0: mmio [e0000000, efffffff] ==> none
[ 1.010079] node 0 link 0: mmio [f0000000, fe1fffff]
[ 1.010079] node 0 link 0: mmio [fe200000, fe3fffff]
[ 1.010079] node 0 link 0: mmio [fe400000, ffefffff]
[ 1.010079] TOM2: 0000000420000000 aka 16896M
[ 1.010079] bus: [00, 07] on node 0 link 0
[ 1.010079] bus: 00 index 0 [io 0x0000-0xffff]
[ 1.010080] bus: 00 index 1 [mem 0x000a0000-0x000bffff]
[ 1.010081] bus: 00 index 2 [mem 0xf0000000-0xffffffff]
[ 1.010082] bus: 00 index 3 [mem 0x420000000-0xfcffffffff]
[ 1.010089] Extended Config Space enabled on 1 nodes
[ 1.010151] ACPI: bus type pci registered
[ 1.010185] PCI: MMCONFIG for domain 0000 [bus 00-ff] at [mem
0xe0000000-0xefffffff] (base 0xe0000000)
[ 1.010185] PCI: not using MMCONFIG
[ 1.010185] PCI: Using configuration type 1 for base access
[ 1.010185] PCI: Using configuration type 1 for extended access
[ 1.010185] bio: create slab <bio-0> at 0
[ 1.179907] raid6: int64x1 3168 MB/s
[ 1.349907] raid6: int64x2 4039 MB/s
[ 1.519903] raid6: int64x4 3205 MB/s
[ 1.689900] raid6: int64x8 3017 MB/s
[ 1.859895] raid6: sse2x1 4891 MB/s
[ 2.029898] raid6: sse2x2 7971 MB/s
[ 2.199900] raid6: sse2x4 9353 MB/s
[ 2.199934] raid6: using algorithm sse2x4 (9353 MB/s)
[ 2.200432] ACPI: EC: Look up EC in DSDT
[ 2.200983] ACPI: Executed 4 blocks of module-level executable AML code
[ 2.230239] ACPI: Interpreter enabled
[ 2.230271] ACPI: (supports S0 S5)
[ 2.230358] ACPI: Using IOAPIC for interrupt routing
[ 2.230475] PCI: MMCONFIG for domain 0000 [bus 00-ff] at [mem
0xe0000000-0xefffffff] (base 0xe0000000)
[ 2.231052] PCI: MMCONFIG at [mem 0xe0000000-0xefffffff] reserved in
ACPI motherboard resources
[ 2.245403] ACPI: No dock devices found.
[ 2.245403] PCI: Using host bridge windows from ACPI; if necessary,
use "pci=nocrs" and report a bug
[ 2.245403] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])
[ 2.245403] pci_root PNP0A03:00: host bridge window [io 0x0000-0x0cf7]
[ 2.245403] pci_root PNP0A03:00: host bridge window [io 0x0d00-0xffff]
[ 2.245403] pci_root PNP0A03:00: host bridge window [mem
0x000a0000-0x000bffff]
[ 2.245403] pci_root PNP0A03:00: host bridge window [mem
0x000d0000-0x000dffff]
[ 2.245403] pci_root PNP0A03:00: host bridge window [mem
0xf0000000-0xfebfffff]
[ 2.245403] pci_root PNP0A03:00: address space collision: host bridge
window [mem 0x000d0000-0x000dffff] conflicts with Adapter ROM [mem
0x000cf000-0x000d53ff]
[ 2.245403] pci 0000:00:00.0: [1022:9601] type 0 class 0x000600
[ 2.245403] pci 0000:00:01.0: [1849:9602] type 1 class 0x000604
[ 2.245403] pci 0000:00:02.0: [1022:9603] type 1 class 0x000604
[ 2.245403] pci 0000:00:02.0: PME# supported from D0 D3hot D3cold
[ 2.245403] pci 0000:00:02.0: PME# disabled
[ 2.245403] pci 0000:00:03.0: [1022:960b] type 1 class 0x000604
[ 2.245403] pci 0000:00:03.0: PME# supported from D0 D3hot D3cold
[ 2.245403] pci 0000:00:03.0: PME# disabled
[ 2.245403] pci 0000:00:04.0: [1022:9604] type 1 class 0x000604
[ 2.245403] pci 0000:00:04.0: PME# supported from D0 D3hot D3cold
[ 2.245403] pci 0000:00:04.0: PME# disabled
[ 2.245403] pci 0000:00:09.0: [1022:9608] type 1 class 0x000604
[ 2.245403] pci 0000:00:09.0: PME# supported from D0 D3hot D3cold
[ 2.245403] pci 0000:00:09.0: PME# disabled
[ 2.245403] pci 0000:00:0a.0: [1022:9609] type 1 class 0x000604
[ 2.245403] pci 0000:00:0a.0: PME# supported from D0 D3hot D3cold
[ 2.245403] pci 0000:00:0a.0: PME# disabled
[ 2.245403] pci 0000:00:11.0: [1002:4391] type 0 class 0x000106
[ 2.245403] pci 0000:00:11.0: reg 10: [io 0x9000-0x9007]
[ 2.245403] pci 0000:00:11.0: reg 14: [io 0x8000-0x8003]
[ 2.245403] pci 0000:00:11.0: reg 18: [io 0x7000-0x7007]
[ 2.245403] pci 0000:00:11.0: reg 1c: [io 0x6000-0x6003]
[ 2.245403] pci 0000:00:11.0: reg 20: [io 0x5000-0x500f]
[ 2.245403] pci 0000:00:11.0: reg 24: [mem 0xfe1ffc00-0xfe1fffff]
[ 2.245403] pci 0000:00:12.0: [1002:4397] type 0 class 0x000c03
[ 2.245403] pci 0000:00:12.0: reg 10: [mem 0xfe1fe000-0xfe1fefff]
[ 2.245403] pci 0000:00:12.2: [1002:4396] type 0 class 0x000c03
[ 2.245403] pci 0000:00:12.2: reg 10: [mem 0xfe1ff800-0xfe1ff8ff]
[ 2.245403] pci 0000:00:12.2: supports D1 D2
[ 2.245403] pci 0000:00:12.2: PME# supported from D0 D1 D2 D3hot
[ 2.245403] pci 0000:00:12.2: PME# disabled
[ 2.245403] pci 0000:00:13.0: [1002:4397] type 0 class 0x000c03
[ 2.245403] pci 0000:00:13.0: reg 10: [mem 0xfe1fd000-0xfe1fdfff]
[ 2.245403] pci 0000:00:13.2: [1002:4396] type 0 class 0x000c03
[ 2.245403] pci 0000:00:13.2: reg 10: [mem 0xfe1ff400-0xfe1ff4ff]
[ 2.245403] pci 0000:00:13.2: supports D1 D2
[ 2.245403] pci 0000:00:13.2: PME# supported from D0 D1 D2 D3hot
[ 2.245403] pci 0000:00:13.2: PME# disabled
[ 2.245403] pci 0000:00:14.0: [1002:4385] type 0 class 0x000c05
[ 2.245403] pci 0000:00:14.2: [1002:4383] type 0 class 0x000403
[ 2.245403] pci 0000:00:14.2: reg 10: [mem 0xfe1f8000-0xfe1fbfff 64bit]
[ 2.245403] pci 0000:00:14.2: PME# supported from D0 D3hot D3cold
[ 2.245403] pci 0000:00:14.2: PME# disabled
[ 2.245403] pci 0000:00:14.3: [1002:439d] type 0 class 0x000601
[ 2.245403] pci 0000:00:14.4: [1002:4384] type 1 class 0x000604
[ 2.245403] pci 0000:00:14.5: [1002:4399] type 0 class 0x000c03
[ 2.245403] pci 0000:00:14.5: reg 10: [mem 0xfe1fc000-0xfe1fcfff]
[ 2.245403] pci 0000:00:16.0: [1002:4397] type 0 class 0x000c03
[ 2.245403] pci 0000:00:16.0: reg 10: [mem 0xfe1f7000-0xfe1f7fff]
[ 2.245403] pci 0000:00:16.2: [1002:4396] type 0 class 0x000c03
[ 2.245403] pci 0000:00:16.2: reg 10: [mem 0xfe1ff000-0xfe1ff0ff]
[ 2.245403] pci 0000:00:16.2: supports D1 D2
[ 2.245403] pci 0000:00:16.2: PME# supported from D0 D1 D2 D3hot
[ 2.245403] pci 0000:00:16.2: PME# disabled
[ 2.245403] pci 0000:00:18.0: [1022:1200] type 0 class 0x000600
[ 2.245403] pci 0000:00:18.1: [1022:1201] type 0 class 0x000600
[ 2.245403] pci 0000:00:18.2: [1022:1202] type 0 class 0x000600
[ 2.245403] pci 0000:00:18.3: [1022:1203] type 0 class 0x000600
[ 2.250003] pci 0000:00:18.4: [1022:1204] type 0 class 0x000600
[ 2.250040] pci 0000:01:05.0: [1002:9715] type 0 class 0x000300
[ 2.250046] pci 0000:01:05.0: reg 10: [mem 0xf0000000-0xf7ffffff pref]
[ 2.250050] pci 0000:01:05.0: reg 14: [io 0xa000-0xa0ff]
[ 2.250054] pci 0000:01:05.0: reg 18: [mem 0xfe3f0000-0xfe3fffff]
[ 2.250061] pci 0000:01:05.0: reg 24: [mem 0xfe200000-0xfe2fffff]
[ 2.250070] pci 0000:01:05.0: supports D1 D2
[ 2.250097] pci 0000:00:01.0: PCI bridge to [bus 01-01]
[ 2.250129] pci 0000:00:01.0: bridge window [io 0xa000-0xafff]
[ 2.250131] pci 0000:00:01.0: bridge window [mem 0xfe200000-0xfe3fffff]
[ 2.250134] pci 0000:00:01.0: bridge window [mem
0xf0000000-0xf7ffffff 64bit pref]
[ 2.250158] pci 0000:04:00.0: [1000:0073] type 0 class 0x000104
[ 2.250165] pci 0000:04:00.0: reg 10: [io 0xc000-0xc0ff]
[ 2.250172] pci 0000:04:00.0: reg 14: [mem 0xfe6fc000-0xfe6fffff 64bit]
[ 2.250180] pci 0000:04:00.0: reg 1c: [mem 0xfe680000-0xfe6bffff 64bit]
[ 2.250190] pci 0000:04:00.0: reg 30: [mem 0xfe6c0000-0xfe6dffff pref]
[ 2.250204] pci 0000:04:00.0: supports D1 D2
[ 2.270016] pci 0000:00:02.0: PCI bridge to [bus 04-04]
[ 2.270057] pci 0000:00:02.0: bridge window [io 0xc000-0xcfff]
[ 2.270059] pci 0000:00:02.0: bridge window [mem 0xfe600000-0xfe6fffff]
[ 2.270062] pci 0000:00:02.0: bridge window [mem
0xfff00000-0x000fffff pref] (disabled)
[ 2.270088] pci 0000:05:00.0: [1000:0073] type 0 class 0x000104
[ 2.270096] pci 0000:05:00.0: reg 10: [io 0xd000-0xd0ff]
[ 2.270104] pci 0000:05:00.0: reg 14: [mem 0xfe7fc000-0xfe7fffff 64bit]
[ 2.270112] pci 0000:05:00.0: reg 1c: [mem 0xfe780000-0xfe7bffff 64bit]
[ 2.270121] pci 0000:05:00.0: reg 30: [mem 0xfe7c0000-0xfe7dffff pref]
[ 2.270136] pci 0000:05:00.0: supports D1 D2
[ 2.290016] pci 0000:00:03.0: PCI bridge to [bus 05-05]
[ 2.290054] pci 0000:00:03.0: bridge window [io 0xd000-0xdfff]
[ 2.290057] pci 0000:00:03.0: bridge window [mem 0xfe700000-0xfe7fffff]
[ 2.290060] pci 0000:00:03.0: bridge window [mem
0xfff00000-0x000fffff pref] (disabled)
[ 2.290091] pci 0000:07:00.0: [11ab:7042] type 0 class 0x000100
[ 2.290104] pci 0000:07:00.0: reg 10: [mem 0xfe900000-0xfe9fffff 64bit]
[ 2.290110] pci 0000:07:00.0: reg 18: [io 0xe800-0xe8ff]
[ 2.290133] pci 0000:07:00.0: reg 30: [mem 0xfe800000-0xfe8fffff pref]
[ 2.290161] pci 0000:07:00.0: disabling ASPM on pre-1.1 PCIe device.
You can enable it with 'pcie_aspm=force'
[ 2.290198] pci 0000:00:04.0: PCI bridge to [bus 07-07]
[ 2.290230] pci 0000:00:04.0: bridge window [io 0xe000-0xefff]
[ 2.290232] pci 0000:00:04.0: bridge window [mem 0xfe800000-0xfe9fffff]
[ 2.290235] pci 0000:00:04.0: bridge window [mem
0xfff00000-0x000fffff pref] (disabled)
[ 2.290264] pci 0000:02:00.0: [1033:0194] type 0 class 0x000c03
[ 2.290277] pci 0000:02:00.0: reg 10: [mem 0xfe4fe000-0xfe4fffff 64bit]
[ 2.290327] pci 0000:02:00.0: PME# supported from D0 D3hot D3cold
[ 2.290330] pci 0000:02:00.0: PME# disabled
[ 2.310021] pci 0000:00:09.0: PCI bridge to [bus 02-02]
[ 2.310056] pci 0000:00:09.0: bridge window [io 0xf000-0x0000]
(disabled)
[ 2.310058] pci 0000:00:09.0: bridge window [mem 0xfe400000-0xfe4fffff]
[ 2.310061] pci 0000:00:09.0: bridge window [mem
0xfff00000-0x000fffff pref] (disabled)
[ 2.310092] pci 0000:03:00.0: [10ec:8168] type 0 class 0x000200
[ 2.310104] pci 0000:03:00.0: reg 10: [io 0xb800-0xb8ff]
[ 2.310122] pci 0000:03:00.0: reg 18: [mem 0xfdfff000-0xfdffffff
64bit pref]
[ 2.310134] pci 0000:03:00.0: reg 20: [mem 0xfdff8000-0xfdffbfff
64bit pref]
[ 2.310142] pci 0000:03:00.0: reg 30: [mem 0xfe5e0000-0xfe5fffff pref]
[ 2.310167] pci 0000:03:00.0: supports D1 D2
[ 2.310169] pci 0000:03:00.0: PME# supported from D0 D1 D2 D3hot D3cold
[ 2.310172] pci 0000:03:00.0: PME# disabled
[ 2.330022] pci 0000:00:0a.0: PCI bridge to [bus 03-03]
[ 2.330056] pci 0000:00:0a.0: bridge window [io 0xb000-0xbfff]
[ 2.330058] pci 0000:00:0a.0: bridge window [mem 0xfe500000-0xfe5fffff]
[ 2.330061] pci 0000:00:0a.0: bridge window [mem
0xfdf00000-0xfdffffff 64bit pref]
[ 2.330109] pci 0000:00:14.4: PCI bridge to [bus 06-06] (subtractive
decode)
[ 2.330142] pci 0000:00:14.4: bridge window [io 0xf000-0x0000]
(disabled)
[ 2.330145] pci 0000:00:14.4: bridge window [mem
0xfff00000-0x000fffff] (disabled)
[ 2.330148] pci 0000:00:14.4: bridge window [mem
0xfff00000-0x000fffff pref] (disabled)
[ 2.330150] pci 0000:00:14.4: bridge window [io 0x0000-0x0cf7]
(subtractive decode)
[ 2.330152] pci 0000:00:14.4: bridge window [io 0x0d00-0xffff]
(subtractive decode)
[ 2.330153] pci 0000:00:14.4: bridge window [mem
0x000a0000-0x000bffff] (subtractive decode)
[ 2.330155] pci 0000:00:14.4: bridge window [mem
0xf0000000-0xfebfffff] (subtractive decode)
[ 2.330168] pci_bus 0000:00: on NUMA node 0
[ 2.330171] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0._PRT]
[ 2.330251] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.P0P1._PRT]
[ 2.330268] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.PCE2._PRT]
[ 2.330283] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.PCE3._PRT]
[ 2.330296] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.PCE4._PRT]
[ 2.330310] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.PCE9._PRT]
[ 2.330324] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.PCEA._PRT]
[ 2.330343] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.P0PC._PRT]
[ 2.330382] pci0000:00: Requesting ACPI _OSC control (0x1d)
[ 2.330420] pci0000:00: ACPI _OSC request failed (AE_NOT_FOUND),
returned control mask: 0x1d
[ 2.330453] ACPI _OSC control for PCIe not granted, disabling ASPM
[ 2.330576] ACPI: PCI Interrupt Link [LNKA] (IRQs 10 11 14 15) *0
[ 2.330576] ACPI: PCI Interrupt Link [LNKB] (IRQs 10 11 14 15) *0
[ 2.330576] ACPI: PCI Interrupt Link [LNKC] (IRQs 10 11 14 15) *0
[ 2.330763] ACPI: PCI Interrupt Link [LNKD] (IRQs 10 11 14 15) *0
[ 2.331002] ACPI: PCI Interrupt Link [LNKE] (IRQs 10 11 14 15) *0
[ 2.331234] ACPI: PCI Interrupt Link [LNKF] (IRQs 10 11 14 15) *0
[ 2.331466] ACPI: PCI Interrupt Link [LNKG] (IRQs 10 11 14 15) *0
[ 2.331698] ACPI: PCI Interrupt Link [LNKH] (IRQs 10 11 14 15) *0
[ 2.331926] vgaarb: device added:
PCI:0000:01:05.0,decodes=io+mem,owns=io+mem,locks=none
[ 2.331926] vgaarb: loaded
[ 2.331926] vgaarb: bridge control possible 0000:01:05.0
[ 2.331926] SCSI subsystem initialized
[ 2.331926] libata version 3.00 loaded.
[ 2.331926] PCI: Using ACPI for IRQ routing
[ 2.341813] PCI: pci_cache_line_size set to 64 bytes
[ 2.341867] reserve RAM buffer: 000000000009c400 - 000000000009ffff
[ 2.341869] reserve RAM buffer: 00000000dbf80000 - 00000000dbffffff
[ 2.341881] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0
[ 2.341881] hpet0: 3 comparators, 32-bit 14.318180 MHz counter
[ 2.342205] Switching to clocksource hpet
[ 2.342293] pnp: PnP ACPI init
[ 2.342293] ACPI: bus type pnp registered
[ 2.342293] pnp 00:00: [bus 00-ff]
[ 2.342293] pnp 00:00: [io 0x0cf8-0x0cff]
[ 2.342293] pnp 00:00: [io 0x0000-0x0cf7 window]
[ 2.342293] pnp 00:00: [io 0x0d00-0xffff window]
[ 2.342293] pnp 00:00: [mem 0x000a0000-0x000bffff window]
[ 2.342293] pnp 00:00: [mem 0x000d0000-0x000dffff window]
[ 2.342293] pnp 00:00: [mem 0xe0000000-0xdfffffff window disabled]
[ 2.342293] pnp 00:00: [mem 0xf0000000-0xfebfffff window]
[ 2.342293] pnp 00:00: Plug and Play ACPI device, IDs PNP0a03 (active)
[ 2.342293] pnp 00:01: [mem 0x00000000-0xffffffffffffffff disabled]
[ 2.342293] pnp 00:01: [mem 0x00000000-0xffffffffffffffff disabled]
[ 2.342293] system 00:01: Plug and Play ACPI device, IDs PNP0c02 (active)
[ 2.342293] pnp 00:02: [dma 4]
[ 2.342293] pnp 00:02: [io 0x0000-0x000f]
[ 2.342293] pnp 00:02: [io 0x0081-0x0083]
[ 2.342293] pnp 00:02: [io 0x0087]
[ 2.342293] pnp 00:02: [io 0x0089-0x008b]
[ 2.342293] pnp 00:02: [io 0x008f]
[ 2.342293] pnp 00:02: [io 0x00c0-0x00df]
[ 2.342293] pnp 00:02: Plug and Play ACPI device, IDs PNP0200 (active)
[ 2.342293] pnp 00:03: [io 0x0070-0x0071]
[ 2.342293] pnp 00:03: [irq 8]
[ 2.342293] pnp 00:03: Plug and Play ACPI device, IDs PNP0b00 (active)
[ 2.342293] pnp 00:04: [io 0x0061]
[ 2.342293] pnp 00:04: Plug and Play ACPI device, IDs PNP0800 (active)
[ 2.342293] pnp 00:05: [io 0x00f0-0x00ff]
[ 2.342293] pnp 00:05: [irq 13]
[ 2.342293] pnp 00:05: Plug and Play ACPI device, IDs PNP0c04 (active)
[ 2.342293] pnp 00:06: [io 0x02f8-0x02ff]
[ 2.342293] pnp 00:06: [irq 3]
[ 2.342293] pnp 00:06: [dma 0 disabled]
[ 2.342293] pnp 00:06: Plug and Play ACPI device, IDs PNP0510 (active)
[ 2.342293] pnp 00:07: [mem 0xfed00000-0xfed003ff]
[ 2.342293] pnp 00:07: Plug and Play ACPI device, IDs PNP0103 (active)
[ 2.342293] pnp 00:08: [mem 0xfec00000-0xfec00fff]
[ 2.342293] pnp 00:08: [mem 0xfee00000-0xfee00fff]
[ 2.342293] system 00:08: [mem 0xfec00000-0xfec00fff] could not be
reserved
[ 2.342293] system 00:08: [mem 0xfee00000-0xfee00fff] has been reserved
[ 2.342293] system 00:08: Plug and Play ACPI device, IDs PNP0c02 (active)
[ 2.342293] pnp 00:09: [io 0x0010-0x001f]
[ 2.342293] pnp 00:09: [io 0x0022-0x003f]
[ 2.342293] pnp 00:09: [io 0x0062-0x0063]
[ 2.342293] pnp 00:09: [io 0x0065-0x006f]
[ 2.342293] pnp 00:09: [io 0x0072-0x007f]
[ 2.342293] pnp 00:09: [io 0x0080]
[ 2.342293] pnp 00:09: [io 0x0084-0x0086]
[ 2.342293] pnp 00:09: [io 0x0088]
[ 2.342293] pnp 00:09: [io 0x008c-0x008e]
[ 2.342293] pnp 00:09: [io 0x0090-0x009f]
[ 2.342293] pnp 00:09: [io 0x00a2-0x00bf]
[ 2.342293] pnp 00:09: [io 0x00b1]
[ 2.342293] pnp 00:09: [io 0x00e0-0x00ef]
[ 2.342293] pnp 00:09: [io 0x04d0-0x04d1]
[ 2.342293] pnp 00:09: [io 0x040b]
[ 2.342293] pnp 00:09: [io 0x04d6]
[ 2.342293] pnp 00:09: [io 0x0c00-0x0c01]
[ 2.342293] pnp 00:09: [io 0x0c14]
[ 2.342293] pnp 00:09: [io 0x0c50-0x0c51]
[ 2.342293] pnp 00:09: [io 0x0c52]
[ 2.342293] pnp 00:09: [io 0x0c6c]
[ 2.342293] pnp 00:09: [io 0x0c6f]
[ 2.342293] pnp 00:09: [io 0x0cd0-0x0cd1]
[ 2.342293] pnp 00:09: [io 0x0cd2-0x0cd3]
[ 2.342293] pnp 00:09: [io 0x0cd4-0x0cd5]
[ 2.342293] pnp 00:09: [io 0x0cd6-0x0cd7]
[ 2.342293] pnp 00:09: [io 0x0cd8-0x0cdf]
[ 2.342293] pnp 00:09: [io 0x0800-0x089f]
[ 2.342293] pnp 00:09: [io 0x0000-0xffffffffffffffff disabled]
[ 2.342293] pnp 00:09: [io 0x0b00-0x0b1f]
[ 2.342293] pnp 00:09: [io 0x0b20-0x0b3f]
[ 2.342293] pnp 00:09: [io 0x0900-0x090f]
[ 2.342293] pnp 00:09: [io 0x0910-0x091f]
[ 2.342293] pnp 00:09: [io 0xfe00-0xfefe]
[ 2.342293] pnp 00:09: [io 0x0060-0x005f disabled]
[ 2.342293] pnp 00:09: [io 0x0064-0x0063 disabled]
[ 2.342293] pnp 00:09: [mem 0xffb80000-0xffbfffff]
[ 2.342293] pnp 00:09: [mem 0xfec10000-0xfec1001f]
[ 2.342293] pnp 00:09: [mem 0xfed80000-0xfed80fff]
[ 2.342293] pnp 00:09: [mem 0x00000000-0xffffffffffffffff disabled]
[ 2.342293] system 00:09: [io 0x04d0-0x04d1] has been reserved
[ 2.342293] system 00:09: [io 0x040b] has been reserved
[ 2.342293] system 00:09: [io 0x04d6] has been reserved
[ 2.342293] system 00:09: [io 0x0c00-0x0c01] has been reserved
[ 2.342293] system 00:09: [io 0x0c14] has been reserved
[ 2.342293] system 00:09: [io 0x0c50-0x0c51] has been reserved
[ 2.342293] system 00:09: [io 0x0c52] has been reserved
[ 2.342293] system 00:09: [io 0x0c6c] has been reserved
[ 2.342293] system 00:09: [io 0x0c6f] has been reserved
[ 2.342293] system 00:09: [io 0x0cd0-0x0cd1] has been reserved
[ 2.342293] system 00:09: [io 0x0cd2-0x0cd3] has been reserved
[ 2.342293] system 00:09: [io 0x0cd4-0x0cd5] has been reserved
[ 2.342293] system 00:09: [io 0x0cd6-0x0cd7] has been reserved
[ 2.342293] system 00:09: [io 0x0cd8-0x0cdf] has been reserved
[ 2.342293] system 00:09: [io 0x0800-0x089f] has been reserved
[ 2.342293] system 00:09: [io 0x0b00-0x0b1f] has been reserved
[ 2.342293] system 00:09: [io 0x0b20-0x0b3f] has been reserved
[ 2.342293] system 00:09: [io 0x0900-0x090f] has been reserved
[ 2.342293] system 00:09: [io 0x0910-0x091f] has been reserved
[ 2.342293] system 00:09: [io 0xfe00-0xfefe] has been reserved
[ 2.342293] system 00:09: [mem 0xffb80000-0xffbfffff] has been reserved
[ 2.342293] system 00:09: [mem 0xfec10000-0xfec1001f] has been reserved
[ 2.342293] system 00:09: [mem 0xfed80000-0xfed80fff] has been reserved
[ 2.342293] system 00:09: Plug and Play ACPI device, IDs PNP0c02 (active)
[ 2.342293] pnp 00:0a: [io 0x0060]
[ 2.342293] pnp 00:0a: [io 0x0064]
[ 2.342293] pnp 00:0a: [irq 1]
[ 2.342293] pnp 00:0a: Plug and Play ACPI device, IDs PNP0303 PNP030b
(active)
[ 2.342293] pnp 00:0b: [io 0x03f8-0x03ff]
[ 2.342293] pnp 00:0b: [irq 4]
[ 2.342293] pnp 00:0b: [dma 0 disabled]
[ 2.342293] pnp 00:0b: Plug and Play ACPI device, IDs PNP0501 (active)
[ 2.342316] pnp 00:0c: [io 0x0000-0xffffffffffffffff disabled]
[ 2.342317] pnp 00:0c: [io 0x0000-0xffffffffffffffff disabled]
[ 2.342319] pnp 00:0c: [io 0x0290-0x029f]
[ 2.342365] system 00:0c: [io 0x0290-0x029f] has been reserved
[ 2.342396] system 00:0c: Plug and Play ACPI device, IDs PNP0c02 (active)
[ 2.342412] pnp 00:0d: [mem 0xe0000000-0xefffffff]
[ 2.342457] system 00:0d: [mem 0xe0000000-0xefffffff] has been reserved
[ 2.342489] system 00:0d: Plug and Play ACPI device, IDs PNP0c02 (active)
[ 2.342549] pnp 00:0e: [mem 0x00000000-0x0009ffff]
[ 2.342550] pnp 00:0e: [mem 0x000c0000-0x000cffff]
[ 2.342551] pnp 00:0e: [mem 0x000e0000-0x000fffff]
[ 2.342553] pnp 00:0e: [mem 0x00100000-0xdfffffff]
[ 2.342554] pnp 00:0e: [mem 0xfec00000-0xffffffff]
[ 2.342629] system 00:0e: [mem 0x00000000-0x0009ffff] could not be
reserved
[ 2.342660] system 00:0e: [mem 0x000c0000-0x000cffff] could not be
reserved
[ 2.342691] system 00:0e: [mem 0x000e0000-0x000fffff] could not be
reserved
[ 2.342723] system 00:0e: [mem 0x00100000-0xdfffffff] could not be
reserved
[ 2.342754] system 00:0e: [mem 0xfec00000-0xffffffff] could not be
reserved
[ 2.342785] system 00:0e: Plug and Play ACPI device, IDs PNP0c01 (active)
[ 2.342835] pnp: PnP ACPI: found 15 devices
[ 2.342865] ACPI: ACPI bus type pnp unregistered
[ 2.345616] Switched to NOHz mode on CPU #0
[ 2.345673] Switched to NOHz mode on CPU #3
[ 2.345731] Switched to NOHz mode on CPU #1
[ 2.345734] Switched to NOHz mode on CPU #2
[ 2.345746] Switched to NOHz mode on CPU #4
[ 2.345749] Switched to NOHz mode on CPU #5
[ 2.349869] PCI: max bus depth: 1 pci_try_num: 2
[ 2.349888] pci 0000:00:01.0: PCI bridge to [bus 01-01]
[ 2.349919] pci 0000:00:01.0: bridge window [io 0xa000-0xafff]
[ 2.349951] pci 0000:00:01.0: bridge window [mem 0xfe200000-0xfe3fffff]
[ 2.349982] pci 0000:00:01.0: bridge window [mem
0xf0000000-0xf7ffffff 64bit pref]
[ 2.350059] pci 0000:00:02.0: PCI bridge to [bus 04-04]
[ 2.350090] pci 0000:00:02.0: bridge window [io 0xc000-0xcfff]
[ 2.350122] pci 0000:00:02.0: bridge window [mem 0xfe600000-0xfe6fffff]
[ 2.350153] pci 0000:00:02.0: bridge window [mem pref disabled]
[ 2.350185] pci 0000:00:03.0: PCI bridge to [bus 05-05]
[ 2.350216] pci 0000:00:03.0: bridge window [io 0xd000-0xdfff]
[ 2.350249] pci 0000:00:03.0: bridge window [mem 0xfe700000-0xfe7fffff]
[ 2.350280] pci 0000:00:03.0: bridge window [mem pref disabled]
[ 2.350312] pci 0000:00:04.0: PCI bridge to [bus 07-07]
[ 2.350343] pci 0000:00:04.0: bridge window [io 0xe000-0xefff]
[ 2.350375] pci 0000:00:04.0: bridge window [mem 0xfe800000-0xfe9fffff]
[ 2.350411] pci 0000:00:04.0: bridge window [mem pref disabled]
[ 2.350443] pci 0000:00:09.0: PCI bridge to [bus 02-02]
[ 2.350473] pci 0000:00:09.0: bridge window [io disabled]
[ 2.350504] pci 0000:00:09.0: bridge window [mem 0xfe400000-0xfe4fffff]
[ 2.350535] pci 0000:00:09.0: bridge window [mem pref disabled]
[ 2.350567] pci 0000:00:0a.0: PCI bridge to [bus 03-03]
[ 2.350598] pci 0000:00:0a.0: bridge window [io 0xb000-0xbfff]
[ 2.350629] pci 0000:00:0a.0: bridge window [mem 0xfe500000-0xfe5fffff]
[ 2.350661] pci 0000:00:0a.0: bridge window [mem
0xfdf00000-0xfdffffff 64bit pref]
[ 2.350694] pci 0000:00:14.4: PCI bridge to [bus 06-06]
[ 2.350724] pci 0000:00:14.4: bridge window [io disabled]
[ 2.350757] pci 0000:00:14.4: bridge window [mem disabled]
[ 2.350789] pci 0000:00:14.4: bridge window [mem pref disabled]
[ 2.350829] pci 0000:00:02.0: PCI INT A -> GSI 18 (level, low) -> IRQ 18
[ 2.350861] pci 0000:00:02.0: setting latency timer to 64
[ 2.350865] pci 0000:00:03.0: PCI INT A -> GSI 19 (level, low) -> IRQ 19
[ 2.350897] pci 0000:00:03.0: setting latency timer to 64
[ 2.350901] pci 0000:00:04.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
[ 2.350932] pci 0000:00:04.0: setting latency timer to 64
[ 2.350936] pci 0000:00:09.0: PCI INT A -> GSI 17 (level, low) -> IRQ 17
[ 2.350967] pci 0000:00:09.0: setting latency timer to 64
[ 2.350970] pci 0000:00:0a.0: PCI INT A -> GSI 18 (level, low) -> IRQ 18
[ 2.351001] pci 0000:00:0a.0: setting latency timer to 64
[ 2.351006] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7]
[ 2.351007] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff]
[ 2.351009] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff]
[ 2.351010] pci_bus 0000:00: resource 7 [mem 0xf0000000-0xfebfffff]
[ 2.351012] pci_bus 0000:01: resource 0 [io 0xa000-0xafff]
[ 2.351013] pci_bus 0000:01: resource 1 [mem 0xfe200000-0xfe3fffff]
[ 2.351014] pci_bus 0000:01: resource 2 [mem 0xf0000000-0xf7ffffff
64bit pref]
[ 2.351016] pci_bus 0000:04: resource 0 [io 0xc000-0xcfff]
[ 2.351017] pci_bus 0000:04: resource 1 [mem 0xfe600000-0xfe6fffff]
[ 2.351019] pci_bus 0000:05: resource 0 [io 0xd000-0xdfff]
[ 2.351020] pci_bus 0000:05: resource 1 [mem 0xfe700000-0xfe7fffff]
[ 2.351021] pci_bus 0000:07: resource 0 [io 0xe000-0xefff]
[ 2.351023] pci_bus 0000:07: resource 1 [mem 0xfe800000-0xfe9fffff]
[ 2.351024] pci_bus 0000:02: resource 1 [mem 0xfe400000-0xfe4fffff]
[ 2.351025] pci_bus 0000:03: resource 0 [io 0xb000-0xbfff]
[ 2.351027] pci_bus 0000:03: resource 1 [mem 0xfe500000-0xfe5fffff]
[ 2.351028] pci_bus 0000:03: resource 2 [mem 0xfdf00000-0xfdffffff
64bit pref]
[ 2.351030] pci_bus 0000:06: resource 4 [io 0x0000-0x0cf7]
[ 2.351031] pci_bus 0000:06: resource 5 [io 0x0d00-0xffff]
[ 2.351032] pci_bus 0000:06: resource 6 [mem 0x000a0000-0x000bffff]
[ 2.351034] pci_bus 0000:06: resource 7 [mem 0xf0000000-0xfebfffff]
[ 2.351057] NET: Registered protocol family 2
[ 2.351135] IP route cache hash table entries: 524288 (order: 10,
4194304 bytes)
[ 2.352087] TCP established hash table entries: 262144 (order: 10,
4194304 bytes)
[ 2.353248] TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)
[ 2.353567] TCP: Hash tables configured (established 262144 bind 65536)
[ 2.353601] TCP reno registered
[ 2.353632] UDP hash table entries: 8192 (order: 6, 262144 bytes)
[ 2.353743] UDP-Lite hash table entries: 8192 (order: 6, 262144 bytes)
[ 2.353926] NET: Registered protocol family 1
[ 2.354056] RPC: Registered named UNIX socket transport module.
[ 2.354087] RPC: Registered udp transport module.
[ 2.354117] RPC: Registered tcp transport module.
[ 2.354147] RPC: Registered tcp NFSv4.1 backchannel transport module.
[ 2.354182] pci 0000:00:01.0: MSI quirk detected; subordinate MSI
disabled
[ 2.500089] pci 0000:01:05.0: Boot video device
[ 2.500118] PCI: CLS 64 bytes, default 64
[ 2.500152] Unpacking initramfs...
[ 2.539130] Freeing initrd memory: 3168k freed
[ 2.539863] PCI-DMA: Disabling AGP.
[ 2.540036] PCI-DMA: aperture base @ d0000000 size 65536 KB
[ 2.540067] PCI-DMA: using GART IOMMU.
[ 2.540098] PCI-DMA: Reserving 64MB of IOMMU area in the AGP aperture
[ 2.574189] HugeTLB registered 2 MB page size, pre-allocated 0 pages
[ 2.636100] Installing knfsd (copyright (C) 1996 okir@monad.swb.de).
[ 2.636310] msgmni has been set to 31992
[ 2.636457] Block layer SCSI generic (bsg) driver version 0.4 loaded
(major 254)
[ 2.636490] io scheduler noop registered
[ 2.636519] io scheduler deadline registered
[ 2.636606] io scheduler cfq registered (default)
[ 2.636711] pcieport 0000:00:02.0: setting latency timer to 64
[ 2.636730] pcieport 0000:00:02.0: irq 40 for MSI/MSI-X
[ 2.636781] pcieport 0000:00:03.0: setting latency timer to 64
[ 2.636795] pcieport 0000:00:03.0: irq 41 for MSI/MSI-X
[ 2.636839] pcieport 0000:00:04.0: setting latency timer to 64
[ 2.636854] pcieport 0000:00:04.0: irq 42 for MSI/MSI-X
[ 2.636898] pcieport 0000:00:09.0: setting latency timer to 64
[ 2.636912] pcieport 0000:00:09.0: irq 43 for MSI/MSI-X
[ 2.636956] pcieport 0000:00:0a.0: setting latency timer to 64
[ 2.636971] pcieport 0000:00:0a.0: irq 44 for MSI/MSI-X
[ 2.637246] input: Power Button as
/devices/LNXSYSTM:00/device:00/PNP0C0C:00/input/input0
[ 2.637280] ACPI: Power Button [PWRB]
[ 2.637353] input: Power Button as
/devices/LNXSYSTM:00/LNXPWRBN:00/input/input1
[ 2.637386] ACPI: Power Button [PWRF]
[ 2.637446] ACPI: acpi_idle registered with cpuidle
[ 2.658560] Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled
[ 2.679110] serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
[ 2.699710] serial8250: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
[ 2.720581] 00:0b: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
[ 2.720829] Real Time Clock Driver v1.12b
[ 2.721594] loop: module loaded
[ 2.721752] tun: Universal TUN/TAP device driver, 1.6
[ 2.721782] tun: (C) 1999-2004 Max Krasnyansky <maxk@qualcomm.com>
[ 2.721866] r8169 Gigabit Ethernet driver 2.3LK-NAPI loaded
[ 2.721909] r8169 0000:03:00.0: PCI INT A -> GSI 18 (level, low) ->
IRQ 18
[ 2.721993] r8169 0000:03:00.0: setting latency timer to 64
[ 2.722028] r8169 0000:03:00.0: irq 45 for MSI/MSI-X
[ 2.722217] r8169 0000:03:00.0: eth0: RTL8168e/8111e at
0xffffc9000000e000, 00:25:22:55:3e:2d, XID 0c100000 IRQ 45
[ 2.722289] netconsole: local port 6666
[ 2.722332] netconsole: local IP 192.168.2.1
[ 2.722361] netconsole: interface 'eth0'
[ 2.722391] netconsole: remote port 5050
[ 2.722420] netconsole: remote IP 192.168.2.87
[ 2.722451] netconsole: remote ethernet address 00:16:cb:a7:dd:d1
[ 2.722482] netconsole: device eth0 not up yet, forcing it
[ 3.540028] Refined TSC clocksource calibration: 3213.418 MHz.
[ 3.540067] Switching to clocksource tsc
[ 63.200106] r8169 0000:03:00.0: eth0: unable to load firmware patch
rtl_nic/rtl8168e-1.fw (-2)
[ 63.223480] r8169 0000:03:00.0: eth0: link down
[ 63.223523] r8169 0000:03:00.0: eth0: link down
[ 65.179915] r8169 0000:03:00.0: eth0: link up
[ 65.182530] console [netcon0] enabled
[ 65.182562] netconsole: network logging started
[ 65.182738] i8042: PNP: PS/2 Controller [PNP0303:PS2K] at 0x60,0x64 irq 1
[ 65.182771] i8042: PNP: PS/2 appears to have AUX port disabled, if
this is incorrect please boot with i8042.nopnp
[ 65.183304] serio: i8042 KBD port at 0x60,0x64 irq 1
[ 65.183474] mousedev: PS/2 mouse device common for all mice
[ 65.183532] i2c /dev entries driver
[ 65.183588] md: linear personality registered for level -1
[ 65.183619] md: raid0 personality registered for level 0
[ 65.183651] md: raid1 personality registered for level 1
[ 65.183683] md: raid10 personality registered for level 10
[ 65.183714] md: raid6 personality registered for level 6
[ 65.183746] md: raid5 personality registered for level 5
[ 65.183779] md: raid4 personality registered for level 4
[ 65.183968] device-mapper: ioctl: 4.20.0-ioctl (2011-02-02)
initialised: dm-devel@redhat.com
[ 65.184375] cpuidle: using governor ladder
[ 65.185144] cpuidle: using governor menu
[ 65.185339] TCP cubic registered
[ 65.185377] NET: Registered protocol family 17
[ 65.185431] Bridge firewalling registered
[ 65.185468] 802.1Q VLAN Support v1.8
[ 65.185502] Registering the dns_resolver key type
[ 65.185730] registered taskstats version 1
[ 65.216255] Freeing unused kernel memory: 468k freed
[ 65.226393] udev[1318]: starting version 164
[ 65.237596] input: AT Translated Set 2 keyboard as
/devices/platform/i8042/serio0/input/input2
[ 65.243627] megasas: 00.00.05.38-rc1 Wed. May. 11 17:00:00 PDT 2011
[ 65.243694] megasas: 0x1000:0x0073:0x1014:0x03b1: bus 4:slot 0:func 0
[ 65.243765] megaraid_sas 0000:04:00.0: PCI INT A -> GSI 18 (level,
low) -> IRQ 18
[ 65.243803] megaraid_sas 0000:04:00.0: setting latency timer to 64
[ 65.244850] megasas: FW now in Ready state
[ 65.245232] megaraid_sas 0000:04:00.0: irq 46 for MSI/MSI-X
[ 65.246538] sata_mv 0000:07:00.0: version 1.28
[ 65.246610] sata_mv 0000:07:00.0: PCI INT A -> GSI 16 (level, low) ->
IRQ 16
[ 65.246785] sata_mv 0000:07:00.0: Gen-IIE 32 slots 4 ports SCSI mode
IRQ via INTx
[ 65.246825] sata_mv 0000:07:00.0: setting latency timer to 64
[ 65.247046] ahci 0000:00:11.0: version 3.0
[ 65.247063] ahci 0000:00:11.0: PCI INT A -> GSI 19 (level, low) -> IRQ 19
[ 65.247146] ahci 0000:00:11.0: irq 47 for MSI/MSI-X
[ 65.247256] ahci 0000:00:11.0: AHCI 0001.0200 32 slots 6 ports 6 Gbps
0x3f impl SATA mode
[ 65.247293] ahci 0000:00:11.0: flags: 64bit ncq sntf ilck pm led clo
pmp pio slum part
[ 65.247385] usbcore: registered new interface driver usbfs
[ 65.247443] usbcore: registered new interface driver hub
[ 65.248070] scsi1 : sata_mv
[ 65.248189] usbcore: registered new device driver usb
[ 65.248450] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[ 65.248497] scsi3 : sata_mv
[ 65.248728] scsi2 : ahci
[ 65.249183] scsi4 : sata_mv
[ 65.249270] scsi6 : sata_mv
[ 65.249318] ata1: SATA max UDMA/133 mmio m1048576@0xfe900000 port
0xfe922000 irq 16
[ 65.249354] ata2: SATA max UDMA/133 mmio m1048576@0xfe900000 port
0xfe924000 irq 16
[ 65.249390] ata3: SATA max UDMA/133 mmio m1048576@0xfe900000 port
0xfe926000 irq 16
[ 65.249425] ata4: SATA max UDMA/133 mmio m1048576@0xfe900000 port
0xfe928000 irq 16
[ 65.249981] scsi5 : ahci
[ 65.250068] scsi7 : ahci
[ 65.250436] scsi8 : ahci
[ 65.250534] scsi9 : ahci
[ 65.250685] scsi10 : ahci
[ 65.250746] ata5: SATA max UDMA/133 abar m1024@0xfe1ffc00 port
0xfe1ffd00 irq 47
[ 65.250783] ata6: SATA max UDMA/133 abar m1024@0xfe1ffc00 port
0xfe1ffd80 irq 47
[ 65.250820] ata7: SATA max UDMA/133 abar m1024@0xfe1ffc00 port
0xfe1ffe00 irq 47
[ 65.250857] ata8: SATA max UDMA/133 abar m1024@0xfe1ffc00 port
0xfe1ffe80 irq 47
[ 65.250895] ata9: SATA max UDMA/133 abar m1024@0xfe1ffc00 port
0xfe1fff00 irq 47
[ 65.250933] ata10: SATA max UDMA/133 abar m1024@0xfe1ffc00 port
0xfe1fff80 irq 47
[ 65.250996] ehci_hcd 0000:00:12.2: PCI INT B -> GSI 17 (level, low)
-> IRQ 17
[ 65.251056] ehci_hcd 0000:00:12.2: EHCI Host Controller
[ 65.251101] ehci_hcd 0000:00:12.2: new USB bus registered, assigned
bus number 1
[ 65.251148] ehci_hcd 0000:00:12.2: applying AMD
SB700/SB800/Hudson-2/3 EHCI dummy qh workaround
[ 65.251199] QUIRK: Enable AMD PLL fix
[ 65.251208] ehci_hcd 0000:00:12.2: debug port 1
[ 65.251257] ehci_hcd 0000:00:12.2: irq 17, io mem 0xfe1ff800
[ 65.270037] ehci_hcd 0000:00:12.2: USB 2.0 started, EHCI 1.00
[ 65.270214] hub 1-0:1.0: USB hub found
[ 65.270254] hub 1-0:1.0: 5 ports detected
[ 65.270351] ehci_hcd 0000:00:13.2: PCI INT B -> GSI 17 (level, low)
-> IRQ 17
[ 65.270404] ehci_hcd 0000:00:13.2: EHCI Host Controller
[ 65.270454] ehci_hcd 0000:00:13.2: new USB bus registered, assigned
bus number 2
[ 65.270496] ehci_hcd 0000:00:13.2: applying AMD
SB700/SB800/Hudson-2/3 EHCI dummy qh workaround
[ 65.270551] ehci_hcd 0000:00:13.2: debug port 1
[ 65.270595] ehci_hcd 0000:00:13.2: irq 17, io mem 0xfe1ff400
[ 65.270605] megasas_init_mfi: fw_support_ieee=67108864
[ 65.270606] megasas: INIT adapter done
[ 65.290144] ehci_hcd 0000:00:13.2: USB 2.0 started, EHCI 1.00
[ 65.290258] hub 2-0:1.0: USB hub found
[ 65.290292] hub 2-0:1.0: 5 ports detected
[ 65.290362] ehci_hcd 0000:00:16.2: PCI INT B -> GSI 17 (level, low)
-> IRQ 17
[ 65.290407] ehci_hcd 0000:00:16.2: EHCI Host Controller
[ 65.290440] ehci_hcd 0000:00:16.2: new USB bus registered, assigned
bus number 3
[ 65.290478] ehci_hcd 0000:00:16.2: applying AMD
SB700/SB800/Hudson-2/3 EHCI dummy qh workaround
[ 65.290533] ehci_hcd 0000:00:16.2: debug port 1
[ 65.290573] ehci_hcd 0000:00:16.2: irq 17, io mem 0xfe1ff000
[ 65.320044] ehci_hcd 0000:00:16.2: USB 2.0 started, EHCI 1.00
[ 65.320151] hub 3-0:1.0: USB hub found
[ 65.320184] hub 3-0:1.0: 4 ports detected
[ 65.348414] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
[ 65.348479] ohci_hcd 0000:00:12.0: PCI INT A -> GSI 18 (level, low)
-> IRQ 18
[ 65.348539] ohci_hcd 0000:00:12.0: OHCI Host Controller
[ 65.348574] ohci_hcd 0000:00:12.0: new USB bus registered, assigned
bus number 4
[ 65.348629] ohci_hcd 0000:00:12.0: irq 18, io mem 0xfe1fe000
[ 65.390058] scsi0 : LSI SAS based MegaRAID driver
[ 65.402010] scsi 0:0:10:0: Direct-Access ATA WDC
WD20EARS-60M AB51 PQ: 0 ANSI: 5
[ 65.402165] sd 0:0:10:0: Attached scsi generic sg0 type 0
[ 65.404137] hub 4-0:1.0: USB hub found
[ 65.404173] hub 4-0:1.0: 5 ports detected
[ 65.404224] ohci_hcd 0000:00:13.0: PCI INT A -> GSI 18 (level, low)
-> IRQ 18
[ 65.404265] ohci_hcd 0000:00:13.0: OHCI Host Controller
[ 65.404298] ohci_hcd 0000:00:13.0: new USB bus registered, assigned
bus number 5
[ 65.404342] ohci_hcd 0000:00:13.0: irq 18, io mem 0xfe1fd000
[ 65.408855] sd 0:0:10:0: [sda] physical block alignment offset: 4096
[ 65.408893] sd 0:0:10:0: [sda] 3907029168 512-byte logical blocks:
(2.00 TB/1.81 TiB)
[ 65.408928] sd 0:0:10:0: [sda] 4096-byte physical blocks
[ 65.410683] scsi 0:0:11:0: Direct-Access ATA WDC
WD20EARS-00M AB51 PQ: 0 ANSI: 5
[ 65.410807] sd 0:0:11:0: Attached scsi generic sg1 type 0
[ 65.415759] sd 0:0:11:0: [sdb] 3907029168 512-byte logical blocks:
(2.00 TB/1.81 TiB)
[ 65.423407] scsi 0:0:12:0: Direct-Access ATA WDC
WD20EARS-60M AB51 PQ: 0 ANSI: 5
[ 65.423536] sd 0:0:12:0: Attached scsi generic sg2 type 0
[ 65.430537] sd 0:0:12:0: [sdc] physical block alignment offset: 4096
[ 65.430574] sd 0:0:12:0: [sdc] 3907029168 512-byte logical blocks:
(2.00 TB/1.81 TiB)
[ 65.430608] sd 0:0:12:0: [sdc] 4096-byte physical blocks
[ 65.437120] sd 0:0:11:0: [sdb] Write Protect is off
[ 65.437156] sd 0:0:11:0: [sdb] Mode Sense: 73 00 00 08
[ 65.437730] sd 0:0:10:0: [sda] Write Protect is off
[ 65.437766] sd 0:0:10:0: [sda] Mode Sense: 73 00 00 08
[ 65.441716] scsi 0:0:13:0: Direct-Access ATA WDC
WD20EARS-60M AB51 PQ: 0 ANSI: 5
[ 65.441835] sd 0:0:13:0: Attached scsi generic sg3 type 0
[ 65.444683] sd 0:0:11:0: [sdb] Write cache: enabled, read cache:
enabled, doesn't support DPO or FUA
[ 65.445330] scsi 0:0:18:0: Direct-Access SEAGATE ST3300655SS
S515 PQ: 0 ANSI: 5
[ 65.445467] sd 0:0:18:0: Attached scsi generic sg4 type 0
[ 65.451625] sd 0:0:10:0: [sda] Write cache: enabled, read cache:
enabled, doesn't support DPO or FUA
[ 65.451711] sd 0:0:13:0: [sdd] physical block alignment offset: 4096
[ 65.451747] sd 0:0:13:0: [sdd] 3907029168 512-byte logical blocks:
(2.00 TB/1.81 TiB)
[ 65.451790] sd 0:0:13:0: [sdd] 4096-byte physical blocks
[ 65.452285] sd 0:0:18:0: [sde] 585937500 512-byte logical blocks:
(300 GB/279 GiB)
[ 65.453321] scsi 0:0:19:0: Direct-Access SEAGATE ST3300655SS
A005 PQ: 0 ANSI: 5
[ 65.453456] sd 0:0:19:0: Attached scsi generic sg5 type 0
[ 65.458128] sd 0:0:18:0: [sde] Write Protect is off
[ 65.458163] sd 0:0:18:0: [sde] Mode Sense: b3 00 10 08
[ 65.459325] sd 0:0:19:0: [sdf] 585937500 512-byte logical blocks:
(300 GB/279 GiB)
[ 65.459448] scsi 0:0:20:0: Direct-Access SEAGATE ST3300655SS
A005 PQ: 0 ANSI: 5
[ 65.459583] sd 0:0:20:0: Attached scsi generic sg6 type 0
[ 65.464128] hub 5-0:1.0: USB hub found
[ 65.464165] hub 5-0:1.0: 5 ports detected
[ 65.464214] ohci_hcd 0000:00:14.5: PCI INT C -> GSI 18 (level, low)
-> IRQ 18
[ 65.464255] ohci_hcd 0000:00:14.5: OHCI Host Controller
[ 65.464288] ohci_hcd 0000:00:14.5: new USB bus registered, assigned
bus number 6
[ 65.464331] ohci_hcd 0000:00:14.5: irq 18, io mem 0xfe1fc000
[ 65.464611] sd 0:0:18:0: [sde] Write cache: enabled, read cache:
enabled, supports DPO and FUA
[ 65.465337] sd 0:0:20:0: [sdg] 585937500 512-byte logical blocks:
(300 GB/279 GiB)
[ 65.465748] scsi 0:0:21:0: Direct-Access SEAGATE ST3300655SS
S515 PQ: 0 ANSI: 5
[ 65.465906] sd 0:0:21:0: Attached scsi generic sg7 type 0
[ 65.470327] sd 0:0:19:0: [sdf] Write Protect is off
[ 65.470362] sd 0:0:19:0: [sdf] Mode Sense: b3 00 10 08
[ 65.476731] sd 0:0:21:0: [sdh] 585937500 512-byte logical blocks:
(300 GB/279 GiB)
[ 65.476846] sd 0:0:20:0: [sdg] Write Protect is off
[ 65.476892] sd 0:0:20:0: [sdg] Mode Sense: b3 00 10 08
[ 65.476916] sd 0:0:19:0: [sdf] Write cache: enabled, read cache:
enabled, supports DPO and FUA
[ 65.482548] sd 0:0:21:0: [sdh] Write Protect is off
[ 65.482586] sd 0:0:21:0: [sdh] Mode Sense: b3 00 10 08
[ 65.483436] sd 0:0:20:0: [sdg] Write cache: enabled, read cache:
enabled, supports DPO and FUA
[ 65.488248] sd 0:0:21:0: [sdh] Write cache: enabled, read cache:
enabled, supports DPO and FUA
[ 65.494395] sde: unknown partition table
[ 65.501262] sdb: unknown partition table
[ 65.503732] sd 0:0:12:0: [sdc] Write Protect is off
[ 65.503776] sd 0:0:12:0: [sdc] Mode Sense: 73 00 00 08
[ 65.505389] sdf: unknown partition table
[ 65.506463] sd 0:0:18:0: [sde] Attached SCSI disk
[ 65.511538] sdh: unknown partition table
[ 65.512582] sda: unknown partition table
[ 65.519104] sd 0:0:12:0: [sdc] Write cache: enabled, read cache:
enabled, doesn't support DPO or FUA
[ 65.521490] sdg: unknown partition table
[ 65.524125] hub 6-0:1.0: USB hub found
[ 65.524162] hub 6-0:1.0: 2 ports detected
[ 65.524211] ohci_hcd 0000:00:16.0: PCI INT A -> GSI 18 (level, low)
-> IRQ 18
[ 65.524252] ohci_hcd 0000:00:16.0: OHCI Host Controller
[ 65.524285] ohci_hcd 0000:00:16.0: new USB bus registered, assigned
bus number 7
[ 65.524332] ohci_hcd 0000:00:16.0: irq 18, io mem 0xfe1f7000
[ 65.526953] sd 0:0:19:0: [sdf] Attached SCSI disk
[ 65.529657] sd 0:0:21:0: [sdh] Attached SCSI disk
[ 65.531531] sd 0:0:13:0: [sdd] Write Protect is off
[ 65.531569] sd 0:0:13:0: [sdd] Mode Sense: 73 00 00 08
[ 65.532606] sd 0:0:20:0: [sdg] Attached SCSI disk
[ 65.537818] megasas: 0x1000:0x0073:0x1014:0x03b1: bus 5:slot 0:func 0
[ 65.537887] megaraid_sas 0000:05:00.0: PCI INT A -> GSI 19 (level,
low) -> IRQ 19
[ 65.537923] megaraid_sas 0000:05:00.0: setting latency timer to 64
[ 65.537974] megasas: FW now in Ready state
[ 65.538034] megaraid_sas 0000:05:00.0: irq 48 for MSI/MSI-X
[ 65.543918] sd 0:0:13:0: [sdd] Write cache: enabled, read cache:
enabled, doesn't support DPO or FUA
[ 65.545106] sd 0:0:11:0: [sdb] Attached SCSI disk
[ 65.560038] megasas_init_mfi: fw_support_ieee=67108864
[ 65.560079] megasas: INIT adapter done
[ 65.563406] sd 0:0:10:0: [sda] Attached SCSI disk
[ 65.570045] sdc: unknown partition table
[ 65.584124] hub 7-0:1.0: USB hub found
[ 65.584161] hub 7-0:1.0: 4 ports detected
[ 65.593514] sdd: unknown partition table
[ 65.600047] ata9: SATA link down (SStatus 0 SControl 300)
[ 65.600107] ata7: SATA link down (SStatus 0 SControl 300)
[ 65.600170] ata8: SATA link down (SStatus 0 SControl 300)
[ 65.600261] ata10: SATA link down (SStatus 0 SControl 300)
[ 65.623341] sd 0:0:12:0: [sdc] Attached SCSI disk
[ 65.642020] sd 0:0:13:0: [sdd] Attached SCSI disk
[ 65.650037] scsi11 : LSI SAS based MegaRAID driver
[ 65.661988] scsi 11:0:9:0: Direct-Access ATA WDC
WD20EARS-60M AB51 PQ: 0 ANSI: 5
[ 65.662113] sd 11:0:9:0: Attached scsi generic sg8 type 0
[ 65.668949] sd 11:0:9:0: [sdi] physical block alignment offset: 4096
[ 65.668986] sd 11:0:9:0: [sdi] 3907029168 512-byte logical blocks:
(2.00 TB/1.81 TiB)
[ 65.669020] sd 11:0:9:0: [sdi] 4096-byte physical blocks
[ 65.675059] scsi 11:0:10:0: Direct-Access ATA WDC
WD20EARS-60M AB51 PQ: 0 ANSI: 5
[ 65.675182] sd 11:0:10:0: Attached scsi generic sg9 type 0
[ 65.681812] sd 11:0:10:0: [sdj] physical block alignment offset: 4096
[ 65.681849] sd 11:0:10:0: [sdj] 3907029168 512-byte logical blocks:
(2.00 TB/1.81 TiB)
[ 65.681884] sd 11:0:10:0: [sdj] 4096-byte physical blocks
[ 65.698957] sd 11:0:9:0: [sdi] Write Protect is off
[ 65.698993] sd 11:0:9:0: [sdi] Mode Sense: 73 00 00 08
[ 65.709639] scsi 11:0:13:0: Direct-Access ATA WDC
WD20EARS-60M AB51 PQ: 0 ANSI: 5
[ 65.709759] sd 11:0:13:0: Attached scsi generic sg10 type 0
[ 65.711168] sd 11:0:9:0: [sdi] Write cache: enabled, read cache:
enabled, doesn't support DPO or FUA
[ 65.716961] sd 11:0:13:0: [sdk] physical block alignment offset: 4096
[ 65.716998] sd 11:0:13:0: [sdk] 3907029168 512-byte logical blocks:
(2.00 TB/1.81 TiB)
[ 65.717035] sd 11:0:13:0: [sdk] 4096-byte physical blocks
[ 65.742927] scsi 11:0:14:0: Direct-Access ATA WDC
WD20EARS-60M AB51 PQ: 0 ANSI: 5
[ 65.743040] sd 11:0:14:0: Attached scsi generic sg11 type 0
[ 65.747118] sd 11:0:13:0: [sdk] Write Protect is off
[ 65.747153] sd 11:0:13:0: [sdk] Mode Sense: 73 00 00 08
[ 65.749113] sd 11:0:14:0: [sdl] physical block alignment offset: 4096
[ 65.749150] sd 11:0:14:0: [sdl] 3907029168 512-byte logical blocks:
(2.00 TB/1.81 TiB)
[ 65.749184] sd 11:0:14:0: [sdl] 4096-byte physical blocks
[ 65.749947] sd 11:0:10:0: [sdj] Write Protect is off
[ 65.749982] sd 11:0:10:0: [sdj] Mode Sense: 73 00 00 08
[ 65.750068] ata1: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
[ 65.759269] sd 11:0:13:0: [sdk] Write cache: enabled, read cache:
enabled, doesn't support DPO or FUA
[ 65.759275] sdi: unknown partition table
[ 65.761631] sd 11:0:10:0: [sdj] Write cache: enabled, read cache:
enabled, doesn't support DPO or FUA
[ 65.777351] sd 11:0:14:0: [sdl] Write Protect is off
[ 65.777386] sd 11:0:14:0: [sdl] Mode Sense: 73 00 00 08
[ 65.787708] sd 11:0:14:0: [sdl] Write cache: enabled, read cache:
enabled, doesn't support DPO or FUA
[ 65.790285] ata1.00: ATA-8: FM-25S2S-60GBP2, 2.1, max UDMA/133
[ 65.790321] ata1.00: 117231408 sectors, multi 1: LBA48 NCQ (depth 31/32)
[ 65.800049] ata6: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[ 65.800102] ata5: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[ 65.803492] scsi 11:0:15:0: Direct-Access ATA WDC
WD20EARS-60M AB51 PQ: 0 ANSI: 5
[ 65.803611] sd 11:0:15:0: Attached scsi generic sg12 type 0
[ 65.807891] sd 11:0:9:0: [sdi] Attached SCSI disk
[ 65.808134] sdj: unknown partition table
[ 65.808954] ata5.00: ATA-8: WDC WD1002FAEX-00Z3A0, 05.01D05, max UDMA/133
[ 65.808989] ata5.00: 1953525168 sectors, multi 0: LBA48 NCQ (depth
31/32), AA
[ 65.809689] sdk: unknown partition table
[ 65.810820] sd 11:0:15:0: [sdm] physical block alignment offset: 4096
[ 65.810865] sd 11:0:15:0: [sdm] 3907029168 512-byte logical blocks:
(2.00 TB/1.81 TiB)
[ 65.810902] sd 11:0:15:0: [sdm] 4096-byte physical blocks
[ 65.811042] ata5.00: configured for UDMA/133
[ 65.812159] ata6.00: ATA-8: WDC WD1002FAEX-00Z3A0, 05.01D05, max UDMA/133
[ 65.812194] ata6.00: 1953525168 sectors, multi 0: LBA48 NCQ (depth
31/32), AA
[ 65.814325] ata6.00: configured for UDMA/133
[ 65.833497] sdl: unknown partition table
[ 65.841554] sd 11:0:15:0: [sdm] Write Protect is off
[ 65.841590] sd 11:0:15:0: [sdm] Mode Sense: 73 00 00 08
[ 65.850283] ata1.00: configured for UDMA/133
[ 65.850387] scsi 1:0:0:0: Direct-Access ATA FM-25S2S-60GBP2
2.1 PQ: 0 ANSI: 5
[ 65.850502] sd 1:0:0:0: Attached scsi generic sg13 type 0
[ 65.850529] sd 1:0:0:0: [sdn] 117231408 512-byte logical blocks:
(60.0 GB/55.8 GiB)
[ 65.850653] sd 1:0:0:0: [sdn] Write Protect is off
[ 65.850687] sd 1:0:0:0: [sdn] Mode Sense: 00 3a 00 00
[ 65.850712] sd 1:0:0:0: [sdn] Write cache: enabled, read cache:
enabled, doesn't support DPO or FUA
[ 65.851123] sdn: unknown partition table
[ 65.851320] sd 1:0:0:0: [sdn] Attached SCSI disk
[ 65.853501] sd 11:0:15:0: [sdm] Write cache: enabled, read cache:
enabled, doesn't support DPO or FUA
[ 65.853583] sd 11:0:10:0: [sdj] Attached SCSI disk
[ 65.860607] sd 11:0:13:0: [sdk] Attached SCSI disk
[ 65.876691] sd 11:0:14:0: [sdl] Attached SCSI disk
[ 65.890059] usb 4-2: new full speed USB device number 2 using ohci_hcd
[ 65.902817] sdm: unknown partition table
[ 65.921026] scsi 11:0:16:0: Direct-Access ATA WDC
WD20EARS-60M AB51 PQ: 0 ANSI: 5
[ 65.921149] sd 11:0:16:0: Attached scsi generic sg14 type 0
[ 65.928106] sd 11:0:16:0: [sdo] physical block alignment offset: 4096
[ 65.928143] sd 11:0:16:0: [sdo] 3907029168 512-byte logical blocks:
(2.00 TB/1.81 TiB)
[ 65.928177] sd 11:0:16:0: [sdo] 4096-byte physical blocks
[ 65.953117] sd 11:0:15:0: [sdm] Attached SCSI disk
[ 66.030173] sd 11:0:16:0: [sdo] Write Protect is off
[ 66.030211] sd 11:0:16:0: [sdo] Mode Sense: 73 00 00 08
[ 66.043679] sd 11:0:16:0: [sdo] Write cache: enabled, read cache:
enabled, doesn't support DPO or FUA
[ 66.092356] sdo: unknown partition table
[ 66.182327] sd 11:0:16:0: [sdo] Attached SCSI disk
[ 66.200059] ata2: SATA link down (SStatus 0 SControl 300)
[ 66.550061] ata3: SATA link down (SStatus 0 SControl 300)
[ 66.900053] ata4: SATA link down (SStatus 0 SControl 300)
[ 66.900198] scsi 2:0:0:0: Direct-Access ATA WDC WD1002FAEX-0
05.0 PQ: 0 ANSI: 5
[ 66.900325] sd 2:0:0:0: [sdp] 1953525168 512-byte logical blocks:
(1.00 TB/931 GiB)
[ 66.900327] sd 2:0:0:0: Attached scsi generic sg15 type 0
[ 66.900427] scsi 5:0:0:0: Direct-Access ATA WDC WD1002FAEX-0
05.0 PQ: 0 ANSI: 5
[ 66.900464] sd 2:0:0:0: [sdp] Write Protect is off
[ 66.900467] sd 2:0:0:0: [sdp] Mode Sense: 00 3a 00 00
[ 66.900491] sd 2:0:0:0: [sdp] Write cache: enabled, read cache:
enabled, doesn't support DPO or FUA
[ 66.900643] sd 5:0:0:0: [sdq] 1953525168 512-byte logical blocks:
(1.00 TB/931 GiB)
[ 66.900645] sd 5:0:0:0: Attached scsi generic sg16 type 0
[ 66.900773] sd 5:0:0:0: [sdq] Write Protect is off
[ 66.900808] sd 5:0:0:0: [sdq] Mode Sense: 00 3a 00 00
[ 66.900829] sd 5:0:0:0: [sdq] Write cache: enabled, read cache:
enabled, doesn't support DPO or FUA
[ 66.904434] sdq: sdq1 sdq2
[ 66.904681] sd 5:0:0:0: [sdq] Attached SCSI disk
[ 66.909577] sdp: sdp1 sdp2
[ 66.909810] sd 2:0:0:0: [sdp] Attached SCSI disk
[ 67.191179] md: md1 stopped.
[ 67.191764] md: bind<sdq1>
[ 67.191895] md: bind<sdp1>
[ 67.192591] bio: create slab <bio-1> at 1
[ 67.192670] md/raid1:md1: active with 2 out of 2 mirrors
[ 67.192714] md1: detected capacity change from 0 to 524222464
[ 67.200588] md1: unknown partition table
[ 67.207511] md: md2 stopped.
[ 67.208237] md: bind<sdq2>
[ 67.209779] md: bind<sdp2>
[ 67.210543] md/raid10:md2: active with 2 out of 2 devices
[ 67.210673] created bitmap (8 pages) for device md2
[ 67.210934] md2: bitmap initialized from disk: read 1/1 pages, set 2
of 14897 bits
[ 67.219474] md2: detected capacity change from 0 to 999677755392
[ 67.224116] md2: p1 p2 p3
[ 67.230750] md: md10 stopped.
[ 67.244399] md: bind<sde>
[ 67.244619] md: bind<sdf>
[ 67.244823] md: bind<sdg>
[ 67.248205] md: bind<sdh>
[ 67.248955] md/raid10:md10: active with 4 out of 4 devices
[ 67.252191] created bitmap (5 pages) for device md10
[ 67.252505] md10: bitmap initialized from disk: read 1/1 pages, set 0
of 8941 bits
[ 67.264422] md10: detected capacity change from 0 to 599997284352
[ 67.271735] md10: unknown partition table
[ 67.296795] md: md0 stopped.
[ 67.398685] md: bind<sdd>
[ 67.404514] md: bind<sdm>
[ 67.404703] md: bind<sdb>
[ 67.408534] md: bind<sdj>
[ 67.411132] md: bind<sdi>
[ 67.411724] md: bind<sdl>
[ 67.430316] md: bind<sda>
[ 67.430607] md: bind<sdc>
[ 67.430770] md: bind<sdk>
[ 67.430922] md: bind<sdo>
[ 67.431657] md/raid:md0: device sdo operational as raid disk 0
[ 67.431694] md/raid:md0: device sdk operational as raid disk 9
[ 67.431726] md/raid:md0: device sdc operational as raid disk 8
[ 67.431758] md/raid:md0: device sda operational as raid disk 7
[ 67.431793] md/raid:md0: device sdl operational as raid disk 6
[ 67.431824] md/raid:md0: device sdi operational as raid disk 5
[ 67.431856] md/raid:md0: device sdj operational as raid disk 4
[ 67.431891] md/raid:md0: device sdb operational as raid disk 3
[ 67.431925] md/raid:md0: device sdm operational as raid disk 2
[ 67.431956] md/raid:md0: device sdd operational as raid disk 1
[ 67.432356] md/raid:md0: allocated 10572kB
[ 67.432418] md/raid:md0: raid level 6 active with 10 out of 10
devices, algorithm 2
[ 67.432457] RAID conf printout:
[ 67.432458] --- level:6 rd:10 wd:10
[ 67.432459] disk 0, o:1, dev:sdo
[ 67.432461] disk 1, o:1, dev:sdd
[ 67.432462] disk 2, o:1, dev:sdm
[ 67.432463] disk 3, o:1, dev:sdb
[ 67.432463] disk 4, o:1, dev:sdj
[ 67.432464] disk 5, o:1, dev:sdi
[ 67.432465] disk 6, o:1, dev:sdl
[ 67.432466] disk 7, o:1, dev:sda
[ 67.432467] disk 8, o:1, dev:sdc
[ 67.432468] disk 9, o:1, dev:sdk
[ 67.466604] created bitmap (15 pages) for device md0
[ 67.466953] md0: bitmap initialized from disk: read 1/1 pages, set 0
of 29809 bits
[ 67.472708] md0: detected capacity change from 0 to 16003181314048
[ 67.511792] md0: unknown partition table
[ 67.680098] EXT4-fs (md2p1): INFO: recovery required on readonly
filesystem
[ 67.680135] EXT4-fs (md2p1): write access will be enabled during recovery
[ 69.810256] EXT4-fs (md2p1): recovery complete
[ 69.810604] EXT4-fs (md2p1): mounted filesystem with ordered data
mode. Opts: (null)
[ 70.548329] udev[1803]: starting version 164
[ 70.676012] usbcore: registered new interface driver usbserial
[ 70.676059] USB Serial support registered for generic
[ 70.676103] usbcore: registered new interface driver usbserial_generic
[ 70.676147] usbserial: USB Serial Driver core
[ 70.683832] USB Serial support registered for pl2303
[ 70.683881] pl2303 4-2:1.0: pl2303 converter detected
[ 70.698118] piix4_smbus 0000:00:14.0: SMBus Host Controller at 0xb00,
revision 0
[ 70.701760] xhci_hcd 0000:02:00.0: PCI INT A -> GSI 17 (level, low)
-> IRQ 17
[ 70.701834] xhci_hcd 0000:02:00.0: setting latency timer to 64
[ 70.701837] xhci_hcd 0000:02:00.0: xHCI Host Controller
[ 70.701877] xhci_hcd 0000:02:00.0: new USB bus registered, assigned
bus number 8
[ 70.702046] xhci_hcd 0000:02:00.0: irq 17, io mem 0xfe4fe000
[ 70.702134] xhci_hcd 0000:02:00.0: irq 49 for MSI/MSI-X
[ 70.702138] xhci_hcd 0000:02:00.0: irq 50 for MSI/MSI-X
[ 70.702140] xhci_hcd 0000:02:00.0: irq 51 for MSI/MSI-X
[ 70.702142] xhci_hcd 0000:02:00.0: irq 52 for MSI/MSI-X
[ 70.702144] xhci_hcd 0000:02:00.0: irq 53 for MSI/MSI-X
[ 70.702146] xhci_hcd 0000:02:00.0: irq 54 for MSI/MSI-X
[ 70.702148] xhci_hcd 0000:02:00.0: irq 55 for MSI/MSI-X
[ 70.702325] xHCI xhci_add_endpoint called for root hub
[ 70.702327] xHCI xhci_check_bandwidth called for root hub
[ 70.702347] hub 8-0:1.0: USB hub found
[ 70.702384] hub 8-0:1.0: 2 ports detected
[ 70.702526] xhci_hcd 0000:02:00.0: xHCI Host Controller
[ 70.702567] xhci_hcd 0000:02:00.0: new USB bus registered, assigned
bus number 9
[ 70.704246] usb 4-2: pl2303 converter now attached to ttyUSB0
[ 70.704320] usbcore: registered new interface driver pl2303
[ 70.704367] pl2303: Prolific PL2303 USB to serial adaptor driver
[ 70.705493] xHCI xhci_add_endpoint called for root hub
[ 70.705495] xHCI xhci_check_bandwidth called for root hub
[ 70.705526] hub 9-0:1.0: USB hub found
[ 70.705569] hub 9-0:1.0: 2 ports detected
[ 71.350688] EXT4-fs (md2p1): re-mounted. Opts: (null)
[ 71.586339] EXT4-fs (md2p1): re-mounted. Opts:
errors=remount-ro,commit=30
[ 71.688584] kvm: Nested Virtualization enabled
[ 71.688624] kvm: Nested Paging enabled
[ 71.715947] powernow-k8: Found 1 AMD Phenom(tm) II X6 1090T Processor
(6 cpu cores) (version 2.20.00)
[ 71.715994] powernow-k8: Core Performance Boosting: on.
[ 71.716050] powernow-k8: 0 : pstate 0 (3200 MHz)
[ 71.716081] powernow-k8: 1 : pstate 1 (2400 MHz)
[ 71.716112] powernow-k8: 2 : pstate 2 (1600 MHz)
[ 71.716144] powernow-k8: 3 : pstate 3 (800 MHz)
[ 71.741872] w83627ehf: Found NCT6775F chip at 0x290
[ 113.081227] fuse init (API version 7.16)
[ 113.463322] EXT4-fs (md2p3): mounted filesystem with ordered data
mode. Opts: commit=30
[ 113.523249] EXT4-fs (md2p2): mounted filesystem with ordered data
mode. Opts: commit=30
[ 113.603188] EXT4-fs (md10): mounted filesystem with ordered data
mode. Opts: commit=30
[ 114.034563] EXT4-fs (md0): mounted filesystem with ordered data mode.
Opts: commit=30,acl
[ 116.104972] Adding 20971516k swap on /raid10/SWAP/srv.swap.
Priority:-1 extents:13 across:22421500k
[ 116.578585] NET: Registered protocol family 15
[ 138.220721] device br1_dummy entered promiscuous mode
[ 180.833074] u32 classifier
[ 180.833129] Actions configured
[ 191.222339] PPP generic driver version 2.4.2
[ 192.068503] NET: Registered protocol family 24
[ 192.759974] ip_tables: (C) 2000-2006 Netfilter Core Team
[ 192.772071] nf_conntrack version 0.5.0 (16384 buckets, 65536 max)
[ 193.055922] HTB: quantum of class 10040 is small. Consider r2q change.
[ 204.360085] sshd (3856): /proc/3856/oom_adj is deprecated, please use
/proc/3856/oom_score_adj instead.
[ 212.869525] NFSD: Using /var/lib/nfs/v4recovery as the NFSv4 state
recovery directory
[ 212.875925] NFSD: starting 90-second grace period
brad@srv:~$ lsmod
Module Size Used by
nfs 294438 1
xt_length 1200 2
xt_CLASSIFY 1075 5
sch_sfq 6676 2
xt_CHECKSUM 1213 2
iptable_mangle 1471 1
xt_TCPMSS 2551 2
ipt_REJECT 2229 1
ipt_MASQUERADE 1743 10
ipt_REDIRECT 1117 1
xt_recent 8207 2
xt_state 1210 6
xt_tcpudp 2451 28
iptable_filter 1426 1
iptable_nat 3977 1
nf_nat 16789 3 ipt_MASQUERADE,ipt_REDIRECT,iptable_nat
nf_conntrack_ipv4 11852 9 iptable_nat,nf_nat
nf_conntrack 60821 5
ipt_MASQUERADE,xt_state,iptable_nat,nf_nat,nf_conntrack_ipv4
nf_defrag_ipv4 1401 1 nf_conntrack_ipv4
ip_tables 16538 3 iptable_mangle,iptable_filter,iptable_nat
x_tables 20342 14
xt_length,xt_CLASSIFY,xt_CHECKSUM,iptable_mangle,xt_TCPMSS,ipt_REJECT,ipt_MASQUERADE,ipt_REDIRECT,xt_recent,xt_state,xt_tcpudp,iptable_filter,iptable_nat,ip_tables
pppoe 9558 2
pppox 2156 1 pppoe
ppp_generic 24227 6 pppoe,pppox
slhc 5277 1 ppp_generic
cls_u32 6356 6
sch_htb 14352 2
deflate 1921 0
zlib_deflate 21196 1 deflate
des_generic 16119 0
cbc 2705 0
ecb 1959 0
crypto_blkcipher 13629 2 cbc,ecb
sha1_generic 2079 0
md5 3985 0
hmac 2961 0
crypto_hash 14471 3 sha1_generic,md5,hmac
cryptomgr 2652 0
aead 6121 1 cryptomgr
crypto_algapi 15273 9
deflate,des_generic,cbc,ecb,crypto_blkcipher,hmac,crypto_hash,cryptomgr,aead
af_key 27356 0
fuse 66684 1
w83627ehf 32036 0
hwmon_vid 2851 1 w83627ehf
vhost_net 17239 5
powernow_k8 12884 0
mperf 1247 1 powernow_k8
kvm_amd 54109 20
kvm 238605 1 kvm_amd
xhci_hcd 66960 0
k10temp 3199 0
i2c_piix4 8375 0
pl2303 12716 1
usbserial 34420 3 pl2303
ohci_hcd 17897 0
ehci_hcd 33721 0
usbcore 131192 5 xhci_hcd,pl2303,usbserial,ohci_hcd,ehci_hcd
ahci 20732 4
libahci 21186 1 ahci
sata_mv 26923 0
megaraid_sas 71785 14
^ permalink raw reply [flat|nested] 111+ messages in thread
end of thread, other threads:[~2011-08-22 11:46 UTC | newest]
Thread overview: 111+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2011-05-31 1:24 KVM induced panic on 2.6.38[2367] & 2.6.39 Brad Campbell
2011-05-31 5:47 ` Borislav Petkov
2011-05-31 5:47 ` Borislav Petkov
2011-05-31 9:26 ` Brad Campbell
2011-05-31 9:26 ` Brad Campbell
2011-05-31 10:38 ` Borislav Petkov
2011-05-31 10:38 ` Borislav Petkov
2011-05-31 14:24 ` Brad Campbell
2011-05-31 14:24 ` Brad Campbell
2011-05-31 14:24 ` Brad Campbell
2011-05-31 22:31 ` Hugh Dickins
2011-05-31 22:31 ` Hugh Dickins
2011-06-01 0:18 ` Brad Campbell
2011-06-01 0:18 ` Brad Campbell
2011-06-01 0:37 ` Brad Campbell
2011-06-01 0:37 ` Brad Campbell
2011-06-01 1:15 ` Andrea Arcangeli
2011-06-01 1:15 ` Andrea Arcangeli
2011-06-01 2:03 ` Brad Campbell
2011-06-01 2:03 ` Brad Campbell
2011-06-01 4:52 ` Hugh Dickins
2011-06-01 4:52 ` Hugh Dickins
2011-06-01 6:31 ` Brad Campbell
2011-06-01 6:31 ` Brad Campbell
2011-06-01 6:56 ` Avi Kivity
2011-06-01 6:56 ` Avi Kivity
2011-06-01 9:29 ` Brad Campbell
2011-06-01 9:29 ` Brad Campbell
2011-06-01 9:29 ` Brad Campbell
2011-06-01 9:40 ` Avi Kivity
2011-06-01 9:40 ` Avi Kivity
2011-06-01 9:41 ` Avi Kivity
2011-06-01 9:41 ` Avi Kivity
2011-06-01 10:53 ` Brad Campbell
2011-06-01 10:53 ` Brad Campbell
2011-06-01 11:09 ` Avi Kivity
2011-06-01 11:09 ` Avi Kivity
2011-06-01 11:18 ` CaT
2011-06-01 11:18 ` CaT
2011-06-01 11:52 ` Brad Campbell
2011-06-01 11:52 ` Brad Campbell
2011-06-01 23:03 ` CaT
2011-06-01 23:03 ` CaT
2011-06-03 13:38 ` Brad Campbell
2011-06-03 13:38 ` Brad Campbell
2011-06-03 15:50 ` Bernhard Held
2011-06-03 15:50 ` Bernhard Held
2011-06-03 15:50 ` Bernhard Held
2011-06-03 16:07 ` Brad Campbell
2011-06-03 16:07 ` Brad Campbell
2011-06-06 20:10 ` Bart De Schuymer
2011-06-06 20:10 ` Bart De Schuymer
2011-06-06 20:23 ` Eric Dumazet
2011-06-06 20:23 ` Eric Dumazet
2011-06-06 20:23 ` Eric Dumazet
2011-06-07 3:33 ` Brad Campbell
2011-06-07 3:33 ` Brad Campbell
2011-06-07 13:30 ` Patrick McHardy
2011-06-07 13:30 ` Patrick McHardy
2011-06-07 14:40 ` Brad Campbell
2011-06-07 14:40 ` Brad Campbell
2011-06-07 15:35 ` Patrick McHardy
2011-06-07 15:35 ` Patrick McHardy
2011-06-07 18:31 ` Eric Dumazet
2011-06-07 18:31 ` Eric Dumazet
2011-06-07 18:31 ` Eric Dumazet
2011-06-07 22:57 ` Patrick McHardy
2011-06-07 22:57 ` Patrick McHardy
2011-06-07 22:57 ` Patrick McHardy
2011-06-08 0:18 ` Brad Campbell
2011-06-08 0:18 ` Brad Campbell
2011-06-08 0:18 ` Brad Campbell
2011-06-08 3:59 ` Eric Dumazet
2011-06-08 3:59 ` Eric Dumazet
2011-06-08 3:59 ` Eric Dumazet
2011-06-08 17:02 ` Brad Campbell
2011-06-08 17:02 ` Brad Campbell
2011-06-08 17:02 ` Brad Campbell
2011-06-08 21:22 ` Eric Dumazet
2011-06-08 21:22 ` Eric Dumazet
2011-06-08 21:22 ` Eric Dumazet
2011-06-10 2:52 ` Simon Horman
2011-06-10 2:52 ` Simon Horman
2011-06-10 12:37 ` Mark Lord
2011-06-10 12:37 ` Mark Lord
2011-06-10 16:43 ` Henrique de Moraes Holschuh
2011-06-10 16:43 ` Henrique de Moraes Holschuh
2011-06-12 15:38 ` Avi Kivity
2011-06-12 15:38 ` Avi Kivity
2011-06-07 23:43 ` Brad Campbell
2011-06-07 23:43 ` Brad Campbell
2011-06-07 18:04 ` Bart De Schuymer
2011-06-07 18:04 ` Bart De Schuymer
2011-06-08 0:15 ` Brad Campbell
2011-06-08 0:15 ` Brad Campbell
2011-06-05 8:14 ` Avi Kivity
2011-06-05 8:14 ` Avi Kivity
2011-06-05 13:45 ` Brad Campbell
2011-06-05 13:45 ` Brad Campbell
2011-06-05 13:58 ` Avi Kivity
2011-06-05 13:58 ` Avi Kivity
2011-06-06 20:22 ` Eric Dumazet
2011-06-06 20:22 ` Eric Dumazet
2011-06-06 20:22 ` Eric Dumazet
2011-06-07 13:27 ` Brad Campbell
2011-06-07 13:37 ` Eric Dumazet
2011-06-07 15:15 ` Brad Campbell
2011-08-20 13:16 ` Brad Campbell
2011-08-22 6:36 ` Avi Kivity
2011-08-22 6:45 ` Eric Dumazet
2011-08-22 11:45 ` Brad Campbell
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.