From: Dave Jones <davej@redhat.com>
To: Linux Kernel <linux-kernel@vger.kernel.org>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Subject: 3.6rc6 slab corruption.
Date: Tue, 18 Sep 2012 10:35:04 -0400 [thread overview]
Message-ID: <20120918143504.GA30585@redhat.com> (raw)
I was chasing a networking bug, and had trinity reduced to just making read & setsockopt calls,
and let that run overnight. I woke up to 800mb of traces from a different bug..
The traces look mostly like this..
=============================================================================
BUG kmalloc-64 (Not tainted): Redzone overwritten
-----------------------------------------------------------------------------
INFO: 0xffff88001f4b4970-0xffff88001f4b4977. First byte 0xbb instead of 0xcc
INFO: Allocated in u32_array_read+0xd1/0x110 age=0 cpu=6 pid=32767
__slab_alloc+0x516/0x5a5
__kmalloc+0x213/0x2c0
u32_array_read+0xd1/0x110
vfs_read+0xac/0x180
sys_read+0x4d/0x90
system_call_fastpath+0x1a/0x1f
INFO: Freed in u32_array_read+0x99/0x110 age=0 cpu=0 pid=32749
__slab_free+0x3f/0x3bf
kfree+0x2d5/0x310
u32_array_read+0x99/0x110
vfs_read+0xac/0x180
sys_read+0x4d/0x90
system_call_fastpath+0x1a/0x1f
INFO: Slab 0xffffea00007d2d00 objects=41 used=14 fp=0xffff88001f4b7410 flags=0x10000000004081
INFO: Object 0xffff88001f4b4930 @offset=2352 fp=0xffff88001f4b7410
Bytes b4 ffff88001f4b4920: 1b 20 1c 00 01 00 00 00 5a 5a 5a 5a 5a 5a 5a 5a . ......ZZZZZZZZ
Object ffff88001f4b4930: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Object ffff88001f4b4940: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Object ffff88001f4b4950: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Object ffff88001f4b4960: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b a5 kkkkkkkkkkkkkkk.
Redzone ffff88001f4b4970: bb bb bb bb bb bb bb bb ........
Padding ffff88001f4b4ab0: 5a 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZ
Pid: 32756, comm: trinity-child52 Not tainted 3.6.0-rc6+ #44
Call Trace:
[<ffffffff811c10ad>] ? print_section+0x3d/0x40
[<ffffffff811c23fe>] print_trailer+0xfe/0x160
[<ffffffff811c2592>] check_bytes_and_report+0xe2/0x120
[<ffffffff81023b79>] ? native_sched_clock+0x19/0x80
[<ffffffff811c2b5b>] check_object+0x18b/0x250
[<ffffffff8169b7d7>] free_debug_processing+0xc0/0x1fd
[<ffffffff812d2e29>] ? u32_array_read+0x99/0x110
[<ffffffff8169ba5c>] __slab_free+0x3f/0x3bf
[<ffffffff81365a1c>] ? debug_check_no_obj_freed+0x16c/0x210
[<ffffffff810db04f>] ? lock_release_holdtime.part.26+0xf/0x180
[<ffffffff812d2e29>] ? u32_array_read+0x99/0x110
[<ffffffff811c5725>] kfree+0x2d5/0x310
[<ffffffff812d2e29>] u32_array_read+0x99/0x110
[<ffffffff811df88c>] vfs_read+0xac/0x180
[<ffffffff811df9ad>] sys_read+0x4d/0x90
[<ffffffff816aea2d>] system_call_fastpath+0x1a/0x1f
FIX kmalloc-64: Restoring 0xffff88001f4b4970-0xffff88001f4b4977=0xcc
=============================================================================
Which looks like we read some file (probably something in sysfs/procfs) that corrupted some internal state.
Any ideas on what I could do to narrow this down ?
The full traces are at http://www.codemonkey.org.uk/junk/slab-corrupt.txt
They vary a little later, but it looks like it's probably all the same problem to me.
Sometimes it flip-flops between "First byte 0xbb instead of 0xcc" and "First byte 0xcc instead of 0xbb"
The one outlier being this weird message..
Sep 18 02:00:13 bitcrush kernel: [36617.487681] hrtimer: interrupt took 242337 ns
Which is weird, but probably unrelated.
Dave
next reply other threads:[~2012-09-18 14:35 UTC|newest]
Thread overview: 33+ messages / expand[flat|nested] mbox.gz Atom feed top
2012-09-18 14:35 Dave Jones [this message]
2012-09-18 18:38 ` 3.6rc6 slab corruption Linus Torvalds
2012-09-18 18:53 ` Dave Jones
2012-09-19 14:00 ` Raghavendra K T
2012-09-19 17:09 ` Linus Torvalds
2012-09-19 21:27 ` David Rientjes
2012-09-19 21:41 ` Dave Jones
2012-09-18 19:23 ` Konrad Rzeszutek Wilk
2012-09-18 20:24 ` David Rientjes
2012-09-18 20:31 ` Konrad Rzeszutek Wilk
2012-09-18 20:36 ` Linus Torvalds
2012-09-19 0:57 ` Raghavendra K T
2012-09-18 20:35 ` Linus Torvalds
2012-09-18 20:37 ` Konrad Rzeszutek Wilk
2012-09-18 20:49 ` Linus Torvalds
2012-09-19 1:16 ` Raghavendra K T
2012-09-19 19:16 ` Konrad Rzeszutek Wilk
2012-09-19 21:29 ` David Rientjes
2012-09-19 21:49 ` David Rientjes
2012-09-19 23:01 ` Linus Torvalds
2012-09-19 23:20 ` David Rientjes
2012-09-20 21:14 ` Konrad Rzeszutek Wilk
2012-09-20 2:29 ` Raghavendra K T
2012-09-20 2:46 ` David Rientjes
2012-09-20 2:55 ` Raghavendra K T
2012-09-20 21:18 ` Konrad Rzeszutek Wilk
2012-09-21 9:16 ` [patch for-3.6] fs, debugfs: fix race in u32_array_read and allocate array at open David Rientjes
2012-09-21 10:22 ` Raghavendra K T
2012-09-24 22:26 ` David Rientjes
2012-09-25 2:54 ` Raghavendra K T
2012-09-20 12:57 ` 3.6rc6 slab corruption Raghavendra K T
2012-09-20 21:18 ` Konrad Rzeszutek Wilk
2012-09-20 12:55 ` Raghavendra K T
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20120918143504.GA30585@redhat.com \
--to=davej@redhat.com \
--cc=linux-kernel@vger.kernel.org \
--cc=torvalds@linux-foundation.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).