Re: [Oops] i386 mm/slab.c (cache_flusharray)

Re: [Oops] i386 mm/slab.c (cache_flusharray)

[pinotj]

OK, I tried again the patch on "small kernel" test11 with CONFIG_DEBUG_PAGEALLOC only. Here are the first results. I will do more tests later because it seems weird. This time I have very different behavior for XFS and Ext3.

I got an oops at boot time, when system try to mount root filesystem (XFS).

But when I tried on a root Ext3, I got no problem at all. I even compiled 2 kernel straight without any problems. It seems to be the first time a test11 works flawless on my system.

Of course, XFS and Ext3 were respectively enabled in the kernels.

I hope I didn't make any mistake, I will confirm tomorrow.


Jerome Pinot

The XFS oops (I was too lazy to write down all the backtraces, all about xfs_* things, I can send if needed):

---
kernel BUG at include/linux/mm.h:267!
invalid operand: 0000 [#1]
CPU:    0
EIP:    0060:[<c01aa5ac>]    Not tainted
Using defaults from ksymoops -t elf32-i386 -a i386
EFLAGS: 00010246
eax: 00000000   ebx: cef3d000   ecx: cef3c000   edx: c1254e28
esi: 00310020   edi: 00000001   ebp: c12f3b18   esp: c12f3b0c
ds: 007b   es: 007b   ss: 0068
Stack: 00000100 00000000 ceee0000 c12f3b28 c01aaf90 c0253d40 cef3d000 c12f3b7c
       c019669b cef3d000 00000100 00000000 00000100 00000000 00000a02 00000000
       00000100 00000000 cef3d000 00000100 00000000 00000a02 00000000 00000802
Call Trace:
 [<c01aaf90>] pagebuf_free+0x24/0x30
 [<c019669b>] xlog_find_verify_cycle+0x18b/0x1e0
Code: 0f 0b 0b 01 d6 cb 1f c0 eb 8e ff 73 54 eb b3 90 53 e8 0e 11


>>EIP; c01aa5ac <_pagebuf_free_object+108/134>   <=====

>>ebx; cef3d000 <_end+ece0e4c/3fda1e4c>
>>ecx; cef3c000 <_end+ecdfe4c/3fda1e4c>
>>edx; c1254e28 <_end+ff8c74/3fda1e4c>
>>ebp; c12f3b18 <_end+1097964/3fda1e4c>
>>esp; c12f3b0c <_end+1097958/3fda1e4c>

Trace; c01aaf90 <pagebuf_free+24/30>
Trace; c019669b <xlog_find_verify_cycle+18b/1e0>

Code;  c01aa5ac <_pagebuf_free_object+108/134>
00000000 <_EIP>:
Code;  c01aa5ac <_pagebuf_free_object+108/134>   <=====
   0:   0f 0b                     ud2a      <=====
Code;  c01aa5ae <_pagebuf_free_object+10a/134>
   2:   0b 01                     or     (%ecx),%eax
Code;  c01aa5b0 <_pagebuf_free_object+10c/134>
   4:   d6                        (bad)
Code;  c01aa5b1 <_pagebuf_free_object+10d/134>
   5:   cb                        lret
Code;  c01aa5b2 <_pagebuf_free_object+10e/134>
   6:   1f                        pop    %ds
Code;  c01aa5b3 <_pagebuf_free_object+10f/134>
   7:   c0 eb 8e                  shr    $0x8e,%bl
Code;  c01aa5b6 <_pagebuf_free_object+112/134>
   a:   ff 73 54                  pushl  0x54(%ebx)
Code;  c01aa5b9 <_pagebuf_free_object+115/134>
   d:   eb b3                     jmp    ffffffc2 <_EIP+0xffffffc2>
Code;  c01aa5bb <_pagebuf_free_object+117/134>
   f:   90                        nop
Code;  c01aa5bc <_pagebuf_free_object+118/134>
  10:   53                        push   %ebx
Code;  c01aa5bd <_pagebuf_free_object+119/134>
  11:   e8 0e 11 00 00            call   1124 <_EIP+0x1124>

 <0>Kernel panic: Attempted to kill the idle task!
---

Re: [Oops] i386 mm/slab.c (cache_flusharray)

[Linus Torvalds]

On Thu, 4 Dec 2003 pinotj@club-internet.fr wrote:
>
> OK, I tried again the patch on "small kernel" test11 with
> CONFIG_DEBUG_PAGEALLOC only. Here are the first results. I will do more
> tests later because it seems weird. This time I have very different
> behavior for XFS and Ext3.

Ok, interesting indeed.

> I got an oops at boot time, when system try to mount root filesystem (XFS).
>
> But when I tried on a root Ext3, I got no problem at all. I even
> compiled 2 kernel straight without any problems. It seems to be the
> first time a test11 works flawless on my system.

All right. That may or may not mean that the bug is actually in mm/slab.c,
rather than anywhere else. It doesn't explain why _you_ hit it and few
others do, but it's still an interesting fact.

Manfred, any ideas? What's different between 2.6.x and 2.4.x in slab?

But it may also be that the bug is in some slab user - since my slab-
translates-to-page-alloc hack always calls the slab constructor function
on every allocation, and the destructor gets called immediately after the
free, my debug version might hide some usage bugs.

> The XFS oops (I was too lazy to write down all the backtraces, all about
> xfs_* things, I can send if needed):
>
> ---
> kernel BUG at include/linux/mm.h:267!

YEAH! That's "put_page_testzero()", and either the BUG_ON() or the
atomic_dec_and_test() noticing bad things.

So that's the "test for count going negative" bug, and it seems to have
found somebody doing a double free on a page. Which is _exactly_ what we
expected from the XFS problems, and would explain the "struct page"
corruption that people report.

> Call Trace:
>  [<c01aaf90>] pagebuf_free+0x24/0x30
>  [<c019669b>] xlog_find_verify_cycle+0x18b/0x1e0
> Code: 0f 0b 0b 01 d6 cb 1f c0 eb 8e ff 73 54 eb b3 90 53 e8 0e 11

It would be good to get the full backtrace, though.

Nathan - did you see the two debug patches I sent out that caught this?

One adds checks to "atomic_dec_and_test()" to verify that the count never
goes negative. The other basically disables all the slab code, and
replaces them with straight page allocations, and that together with
CONFIG_DEBUG_PAGEALLOC helps find bad behaviour (touching allocations
after they are free'd etc).

		Linus

Re: [Oops] i386 mm/slab.c (cache_flusharray)

[Linus Torvalds]

On Thu, 4 Dec 2003, Linus Torvalds wrote:
> > ---
> > kernel BUG at include/linux/mm.h:267!
>
> YEAH! That's "put_page_testzero()", and either the BUG_ON() or the
> atomic_dec_and_test() noticing bad things.

Oh, damn. Looking closer, it appears that it's actually XFS being a bit
too intimate with slab knowledge: the code does

                        if (pb->pb_pages) {
                                /* release the pages in the address list */
                                if (pb->pb_pages[0] &&
                                    PageSlab(pb->pb_pages[0])) {
                                        /*
                                         * This came from the slab
                                         * allocator free it as such
                                         */
                                        kfree(pb->pb_addr);
                                } else {
                                        _pagebuf_freepages(pb);
                                }

and that code gets really confused by the fact that I'm bypassing the slab
logic (and thus the PageSlab flag never gets set).

So the oops it found was apparently triggered by the debugging changes,
not necessarily by a real bug.

Ugh, that XFS code is _broken_. Instead of keeping track of how it got the
memory, it totally forgets where the memory came from, and then it later
asks "oh, btw, how the hell did I allocate this?".

			Linus

Re: [Oops] i386 mm/slab.c (cache_flusharray)

[Manfred Spraul]

[-- Attachment #1: Type: text/plain, Size: 1224 bytes --]

Linus Torvalds wrote:

>Manfred, any ideas? What's different between 2.6.x and 2.4.x in slab?
>
>But it may also be that the bug is in some slab user - since my slab-
>translates-to-page-alloc hack always calls the slab constructor function
>on every allocation, and the destructor gets called immediately after the
>free, my debug version might hide some usage bugs.
>  
>
The changes between 2.4 and 2.6 are huge, for both debug and non-debug. 
Slab with debugging enabled now calls the destructors/constructor on 
every alloc. If page debugging is enabled, then all objects larger than 
128 bytes get their own page and are unmapped after kmem_cache_free(). 
The bio structure is smaller than 128 bytes - that probably explains why 
slab didn't catch the oopses that were mentioned in the other thread.
Perhaps something like the attached patch could help to trigger the 
oops: It increase the size of the bio structures, then they are handled 
by slab debugging.
If it oopses, then call ptrinfo() from the trap handler - it prints the 
name of the cache and the caller of the last slab operation. And hexdump 
the object (after ptrinfo mapped it), it contains a backtrace from the 
kmem_cache_free call.

--
    Manfred

[-- Attachment #2: patch-bio --]
[-- Type: text/plain, Size: 690 bytes --]

--- 2.6/fs/bio.c	2003-10-25 20:43:54.000000000 +0200
+++ build-2.6/fs/bio.c	2003-12-04 20:13:52.000000000 +0100
@@ -798,7 +798,7 @@
 
 		size = bp->nr_vecs * sizeof(struct bio_vec);
 
-		bp->slab = kmem_cache_create(bp->name, size, 0,
+		bp->slab = kmem_cache_create(bp->name, max(128,size), 0,
 						SLAB_HWCACHE_ALIGN, NULL, NULL);
 		if (!bp->slab)
 			panic("biovec: can't init slab cache\n");
@@ -815,7 +815,7 @@
 
 static int __init init_bio(void)
 {
-	bio_slab = kmem_cache_create("bio", sizeof(struct bio), 0,
+	bio_slab = kmem_cache_create("bio", max(128U,sizeof(struct bio)), 0,
 					SLAB_HWCACHE_ALIGN, NULL, NULL);
 	if (!bio_slab)
 		panic("bio: can't create slab cache\n");

Re: [Oops] i386 mm/slab.c (cache_flusharray)

[Nathan Scott]

On Thu, Dec 04, 2003 at 11:09:29AM -0800, Linus Torvalds wrote:
> 
> 
> On Thu, 4 Dec 2003, Linus Torvalds wrote:
> > > ---
> > > kernel BUG at include/linux/mm.h:267!
> >
> > YEAH! That's "put_page_testzero()", and either the BUG_ON() or the
> > atomic_dec_and_test() noticing bad things.
> 
> Oh, damn. Looking closer, it appears that it's actually XFS being a bit
> too intimate with slab knowledge: the code does
> 
>                         if (pb->pb_pages) {
>                                 /* release the pages in the address list */
>                                 if (pb->pb_pages[0] &&
>                                     PageSlab(pb->pb_pages[0])) {
>                                         /*
>                                          * This came from the slab
>                                          * allocator free it as such
>                                          */
>                                         kfree(pb->pb_addr);
>                                 } else {
>                                         _pagebuf_freepages(pb);
>                                 }
> 
> and that code gets really confused by the fact that I'm bypassing the slab
> logic (and thus the PageSlab flag never gets set).
> 
> So the oops it found was apparently triggered by the debugging changes,
> not necessarily by a real bug.
> 
> Ugh, that XFS code is _broken_. Instead of keeping track of how it got the
> memory, it totally forgets where the memory came from, and then it later
> asks "oh, btw, how the hell did I allocate this?".

Yeah, thats pretty silly stuff - and should be fairly easy to
fix by using a pagebuf flag to differentiate the two.  Will do.

thanks.

-- 
Nathan

Re: [Oops] i386 mm/slab.c (cache_flusharray)

[Nathan Scott]

On Thu, Dec 04, 2003 at 10:49:07AM -0800, Linus Torvalds wrote:
> 
> Nathan - did you see the two debug patches I sent out that caught this?
> 
> One adds checks to "atomic_dec_and_test()" to verify that the count never
> goes negative. The other basically disables all the slab code, and
> replaces them with straight page allocations, and that together with
> CONFIG_DEBUG_PAGEALLOC helps find bad behaviour (touching allocations
> after they are free'd etc).
> 

Yep - I'll remove that PageSlab use in there, and start testing
with these too.

thanks.

-- 
Nathan

Re: [Oops] i386 mm/slab.c (cache_flusharray)

[Nathan Scott]

On Thu, Dec 04, 2003 at 11:09:29AM -0800, Linus Torvalds wrote:
> ...
> So the oops it found was apparently triggered by the debugging changes,
> not necessarily by a real bug.
> 
> Ugh, that XFS code is _broken_. Instead of keeping track of how it got the
> memory, it totally forgets where the memory came from, and then it later
> asks "oh, btw, how the hell did I allocate this?".
> 

This patch removes that code, fixes a small memory leak that was
lurking in there too, and adds the missing-bio_put-on-error case
that Neil found in pagebuf.

Neil, with this & Linus' 2 patches (and CONFIG_SLAB_DEBUG off ;)
I now have what looks like a 100% reproducible test case for the
handle_stripe already-freed-bio panic.  This doesn't tickle the
raid5.c BUG_ON you sent me but its exactly the same spot as last
time (i.e. handle_stripe+0xda6), every time.

# raidstart /dev/md0
# mkfs.xfs -f /dev/md0
# mount /dev/md0
# umount /dev/md0
# mount /dev/md0

On my (quad p3) test machine, this second mount panics every time.

cheers.

-- 
Nathan


--- fs/xfs/pagebuf/page_buf.h.orig	2003-12-05 13:47:12.275589232 +1100
+++ fs/xfs/pagebuf/page_buf.h	2003-12-05 13:43:30.898243704 +1100
@@ -123,12 +123,13 @@
 	_PBF_PRIVATE_BH = (1 << 17), /* do not use public buffer heads	   */
 	_PBF_ALL_PAGES_MAPPED = (1 << 18), /* all pages in range mapped	   */
 	_PBF_ADDR_ALLOCATED = (1 << 19), /* pb_addr space was allocated	   */
-	_PBF_MEM_ALLOCATED = (1 << 20), /* pb_mem+underlying pages alloc'd */
+	_PBF_MEM_ALLOCATED = (1 << 20), /* underlying pages are allocated  */
+	_PBF_MEM_SLAB = (1 << 21), /* underlying pages are slab allocated  */
 
-	PBF_FORCEIO = (1 << 21),
-	PBF_FLUSH = (1 << 22),	/* flush disk write cache		   */
-	PBF_READ_AHEAD = (1 << 23),
-	PBF_RUN_QUEUES = (1 << 24), /* run block device task queue	   */
+	PBF_FORCEIO = (1 << 22),
+	PBF_FLUSH = (1 << 23),	/* flush disk write cache		   */
+	PBF_READ_AHEAD = (1 << 24),
+	PBF_RUN_QUEUES = (1 << 25), /* run block device task queue	   */
 
 } page_buf_flags_t;
 
--- fs/xfs/pagebuf/page_buf.c.orig	2003-12-05 13:47:06.888408208 +1100
+++ fs/xfs/pagebuf/page_buf.c	2003-12-05 13:43:30.888245224 +1100
@@ -343,9 +343,6 @@
 			page_cache_release(page);
 		}
 	}
-
-	if (pb->pb_pages != pb->pb_page_array)
-		kfree(pb->pb_pages);
 }
 
 /*
@@ -384,20 +381,17 @@
 		if (pb->pb_flags & _PBF_MEM_ALLOCATED) {
 			if (pb->pb_pages) {
 				/* release the pages in the address list */
-				if (pb->pb_pages[0] &&
-				    PageSlab(pb->pb_pages[0])) {
-					/*
-					 * This came from the slab
-					 * allocator free it as such
-					 */
+				if ((pb->pb_pages[0]) &&
+				    (pb->pb_flags & _PBF_MEM_SLAB)) {
 					kfree(pb->pb_addr);
 				} else {
 					_pagebuf_freepages(pb);
 				}
-
+				if (pb->pb_pages != pb->pb_page_array)
+					kfree(pb->pb_pages);
 				pb->pb_pages = NULL;
 			}
-			pb->pb_flags &= ~_PBF_MEM_ALLOCATED;
+			pb->pb_flags &= ~(_PBF_MEM_ALLOCATED | _PBF_MEM_SLAB);
 		}
 	}
 
@@ -944,7 +938,7 @@
 		return NULL;
 	}
 	/* otherwise pagebuf_free just ignores it */
-	pb->pb_flags |= _PBF_MEM_ALLOCATED;
+	pb->pb_flags |= (_PBF_MEM_ALLOCATED | _PBF_MEM_SLAB);
 	PB_CLEAR_OWNER(pb);
 	up(&pb->pb_sema);	/* Return unlocked pagebuf */
 
@@ -1412,6 +1406,7 @@
 		if (size)
 			goto next_chunk;
 	} else {
+		bio_put(bio);
 		pagebuf_ioerror(pb, EIO);
 	}
 

Re: [Oops] i386 mm/slab.c (cache_flusharray)

[Linus Torvalds]

On Fri, 5 Dec 2003, Nathan Scott wrote:
>
> This patch removes that code, fixes a small memory leak that was
> lurking in there too, and adds the missing-bio_put-on-error case
> that Neil found in pagebuf.

Ok, so we've got the RAID5 issue explained, does anybody have any idea
about what's wrong with the non-RAID5 cases? We've got at least one report
of page corruption on RAID0, and one on a plain disk. Both of which looked
XFS-related - or at least shared that in their configs.

Jerome - can you test Nathan's patch together with my "avoid the
complicated slab logic"? The slab avoidance thing got ext3 stable for you,
now with Nathan's patch hopefully XFS will be stable too.

Which still doesn't _explain_ anything, but it would be interesting to see
if that removes your problems. It would definitely point to a slab issue,
but the big question is why it's not more common if so. Compiler bug?
Other strange trigger?

			Linus

Re: [Oops] i386 mm/slab.c (cache_flusharray)

[Christoph Hellwig]

On Fri, Dec 05, 2003 at 08:21:10AM +1100, Nathan Scott wrote:
> Yeah, thats pretty silly stuff - and should be fairly easy to
> fix by using a pagebuf flag to differentiate the two.  Will do.

IMHO a flags is wrong here.  Just maek pb_addr always a pointer and
for the case it's the preallocated array make it point pb_page_array
or something like that.  Then check whether pb_addr is pointing to the
preallocated array.

Re: [Oops] i386 mm/slab.c (cache_flusharray)

[Nathan Scott]

On Fri, Dec 05, 2003 at 07:14:55AM +0000, Christoph Hellwig wrote:
> On Fri, Dec 05, 2003 at 08:21:10AM +1100, Nathan Scott wrote:
> > Yeah, thats pretty silly stuff - and should be fairly easy to
> > fix by using a pagebuf flag to differentiate the two.  Will do.
> 
> IMHO a flags is wrong here.  Just maek pb_addr always a pointer and
> for the case it's the preallocated array make it point pb_page_array
> or something like that.  Then check whether pb_addr is pointing to the
> preallocated array.

You might be mixing up pb_pages and pb_addr there?  pb_addr is
always a pointer.  We need to distinguish whether it was slab
alloc'd or whether it points into page cache pages, so we know
whether to page_cache_release the pages or kfree the pointer
when we're done with the pagebuf.

The pb_page_array works just as you describe, with a prealloc'd
array of page pointers, and pb_pages either points to the array
of to a larger kmalloc'd array as necessary.

cheers.

-- 
Nathan

Re: [Oops] i386 mm/slab.c (cache_flusharray)

[Christoph Hellwig]

On Fri, Dec 05, 2003 at 08:34:25PM +1100, Nathan Scott wrote:
> You might be mixing up pb_pages and pb_addr there?  pb_addr is
> always a pointer.  We need to distinguish whether it was slab
> alloc'd or whether it points into page cache pages, so we know
> whether to page_cache_release the pages or kfree the pointer
> when we're done with the pagebuf.
> 
> The pb_page_array works just as you describe, with a prealloc'd
> array of page pointers, and pb_pages either points to the array
> of to a larger kmalloc'd array as necessary.

Indeed.  I'm not remembering the code as good as I hoped to :)

Sorry for the noise.

Re: [Oops] i386 mm/slab.c (cache_flusharray)

[Manfred Spraul]

Christoph Hellwig wrote:

>Looks like whe're better of fixing mm/slab.c
>
Maybe I'm blind, but I don't see how kmem_cache_alloc can return NULL 
with __GFP_NOFAIL:
- kmem_cache_alloc calls __cache_alloc.
- cache_alloc only return 0 if cache_alloc_refill returns 0.
- cache_alloc_refill only return 0 if cache_grow returns 0.
- cache_grow only return 0 if
    * SLAB_NO_GROW is set in flags.
    * get_free_pages(flags) fails.
    * kmem_cache_alloc(flags&SLAB_LEVEL_MASK) fails.
       SLAB_LEVEL_MASK contains __GFP_NOFAIL.

--
    Manfred

Re: [Oops] i386 mm/slab.c (cache_flusharray)

[Christoph Hellwig]

On Wed, Dec 10, 2003 at 10:58:32AM +1100, Nathan Scott wrote:
> > > [ Christoph, is this failure expected?  I think you/Steve made
> > > some changes there to use __GFP_NOFAIL and assume it wont fail?
> > > (in 2.4 we do memory allocations differently to better handle
> > > failures, but that code was removed...) ]
> > 
> > It looks like the slab allocator doesn't like __GFP_NOFAIL, we'll
> > probably have to revert the XFS memory allocation wrappers to the
> > 2.4 versions.
> > 
> 
> OK, thanks - I'll look into it.

This starts to look really fishy:

hch@bird:/repo/repo/linux-2.5/fs/jbd$ egrep -r NOFAIL *
journal.c:      new_bh = alloc_buffer_head(GFP_NOFS|__GFP_NOFAIL);
journal.c:      return kmalloc(size, flags | (retry ? __GFP_NOFAIL :
0));

both of these end up in the slab layer, just like XFS - except
that __cache_alloc still may fail.  Looks like whe're better of
fixing mm/slab.c

Re: [Oops] i386 mm/slab.c (cache_flusharray)

[Nathan Scott]

On Tue, Dec 09, 2003 at 08:21:32AM +0100, Christoph Hellwig wrote:
> On Tue, Dec 09, 2003 at 01:03:22PM +1100, Nathan Scott wrote:
> > [ Christoph, is this failure expected?  I think you/Steve made
> > some changes there to use __GFP_NOFAIL and assume it wont fail?
> > (in 2.4 we do memory allocations differently to better handle
> > failures, but that code was removed...) ]
> 
> It looks like the slab allocator doesn't like __GFP_NOFAIL, we'll
> probably have to revert the XFS memory allocation wrappers to the
> 2.4 versions.
> 

OK, thanks - I'll look into it.

cheers.

-- 
Nathan

Re: [Oops] i386 mm/slab.c (cache_flusharray)

[Christoph Hellwig]

On Tue, Dec 09, 2003 at 01:03:22PM +1100, Nathan Scott wrote:
> [ Christoph, is this failure expected?  I think you/Steve made
> some changes there to use __GFP_NOFAIL and assume it wont fail?
> (in 2.4 we do memory allocations differently to better handle
> failures, but that code was removed...) ]

It looks like the slab allocator doesn't like __GFP_NOFAIL, we'll
probably have to revert the XFS memory allocation wrappers to the
2.4 versions.

Re: [Oops] i386 mm/slab.c (cache_flusharray)

[Nathan Scott]

On Tue, Dec 09, 2003 at 01:57:35AM +0100, pinotj@club-internet.fr wrote:
> Results about testing on test11 this week-end.

thanks.

>   ---
>   ld: page allocation failure. order:0, mode:0x8d0
>   Unable to handle kernel NULL pointer dereference at virtual address 00000074
>   c01d4cbd
>   *pde = 00000000
>   Oops: 0002 [#1]
>   CPU:    0
>   EIP:    0060:[_xfs_trans_alloc+149/160]    Not tainted
>   EIP:    0060:[<c01d4cbd>]    Not tainted
>   ---

Ah, yes, I know what this is (and can reproduce it) and I don't
have a fix as yet.  This is an unrelated problem - basically,
we are doing kmem_cache_alloc with __GFP_NOFAIL set within XFS,
but the allocations are failing and returning NULL (but we are
assuming they will not).

You (and I) are hitting this more frequently now because Linus'
patch bypasses the slab allocator and is more expensive in terms
of memory used.  However, I have heard of one person who hit this
"in the wild" too, so its something we will need to address.

[ Christoph, is this failure expected?  I think you/Steve made
some changes there to use __GFP_NOFAIL and assume it wont fail?
(in 2.4 we do memory allocations differently to better handle
failures, but that code was removed...) ]

> 
>  B. With Ext3 (and without XFS)
> 
>   1. no patch
>   same as I.A.1
>   2. patch-xfs & patch-slab
>   Compilations looked good but I got a lot of errors in my logs:
> 
>   ---
>   kernel: ld: page allocation failure. order:0, mode:0x50
>   last message repeated 31 times
>   klogd: page allocation failure. order:0, mode:0x50
>   last message repeated 63 times
>   kswapd0: page allocation failure. order:0, mode:0x50
>   ENOMEM in journal_alloc_journal_head, retrying.
>   ion failure. order:0, mode:0x50
>   kswapd0: page allocation failure. order:0, mode:0x50
>   last message repeated 291 times

I expect that this is a similar issue to the above, but from
ext2/3's point of view.

> 
> Tests in I. confirm that it's not an XFS-only problem but seems to
> affect page allocation for fs in general.
> I hope these oops will be clearer to you. I still have no problem with test9.

FWIW and IIRC, we didn't push any XFS changes (nothing major
anyway, that I could foresee causing this) on to Linus in
between -test9 and -test11.

cheers.

-- 
Nathan

Re: [Oops] i386 mm/slab.c (cache_flusharray)

[Linus Torvalds]

On Thu, 4 Dec 2003 pinotj@club-internet.fr wrote:
>
> [slab.c patch from Linus]
>
> I tried the patch on the same small config (XFS and
> CONFIG_DEBUG_PAGEALLOC enabled) and I got oops at the beginning of boot
> sequence. I spent some times to write this down but I'm not so sure it's
> a good news. Just say me it's not a hw problem...

Sorry - it's not a hw problem, and in fact it's a problem with my slab
debugging patch: please don't use CONFIG_DEBUG_SLAB together with my
crappy patch. My patch wants _only_ CONFIG_DEBUG_PAGEALLOC instead of
using the SLAB debugger.

So please turn the slab debugger off, and depend on CONFIG_DEBUG_PAGEALLOC
entirely and try again,

		Linus

Re: [Oops] i386 mm/slab.c (cache_flusharray)

[pinotj]

I tried the "small kernel" test11 with Ext3 support instead of XFS by booting on my slack partition.

I confirm to have the same problem without XFS, oops as usual when I try to compile a kernel:

---
slab: double free detected in cache 'buffer_head' objp c605733c, objnr 9, slabp c6057000, s_mem c6057120, bufctl 12
---

[slab.c patch from Linus]

I tried the patch on the same small config (XFS and CONFIG_DEBUG_PAGEALLOC enabled) and I got oops at the beginning of boot sequence. I spent some times to write this down but I'm not so sure it's a good news. Just say me it's not a hw problem...

---
kernel BUG at mm/slab.c:1655!
invalid operand: 0000 [#1]
CPU:    0
EIP:    0060:[<c013bf35>]    Not tainted
Using defaults from ksymoops -t elf32-i386 -a i386
EFLAGS: 00010046
eax: c0243c00   ebx: 0027f448   ecx: 00000000   edx: cffb5000
esi: cffb6000   edi: cffb5000   ebp: c027bf70   esp: c027bf60
ds: 007b   es: 007b   ss: 0068
Stack: 00000286 00000000 cffb6000 cffb5000 c027bf94 c013c1a5 cffb5000 cffb6044
       cffb6000 cffb5000 cffb6000 cffb6000 c029be80 c027bfb0 c013c413 cffb6000
       00000008 00000004 00000000 000000d0 c027bfec c0283d8a cffb6000 00000090
Call Trace:
 [<c013c1a5>] do_tune_cpucache+0x1cd/0x3fc
 [<c013c413>] enable_cpucache+0x3f/0x64
 [<c0283d8a>] kmem_cache_init+0x1ae/0x280
 [<c0105000>] _stext+0x0/0x20
 [<c027c5af>] start_kernel+0xb3/0x138
Code: 0f 0b 77 06 75 2f 24 c0 8b 15 c4 bf 29 c0 eb af 8d 76 00 57


>>EIP; c013bf35 <kfree+81/b0>   <=====

>>eax; c0243c00 <bpp4tab+5f40/13436>
>>edx; cffb5000 <_end+fd0c990/3fd55990>
>>esi; cffb6000 <_end+fd0d990/3fd55990>
>>edi; cffb5000 <_end+fd0c990/3fd55990>
>>ebp; c027bf70 <init_thread_union+1f70/2000>
>>esp; c027bf60 <init_thread_union+1f60/2000>

Trace; c013c1a5 <do_tune_cpucache+1cd/3fc>
Trace; c013c413 <enable_cpucache+3f/64>
Trace; c0283d8a <kmem_cache_init+1ae/280>
Trace; c0105000 <_stext+0/0>
Trace; c027c5af <start_kernel+b3/138>

Code;  c013bf35 <kfree+81/b0>
00000000 <_EIP>:
Code;  c013bf35 <kfree+81/b0>   <=====
   0:   0f 0b                     ud2a      <=====
Code;  c013bf37 <kfree+83/b0>
   2:   77 06                     ja     a <_EIP+0xa>
Code;  c013bf39 <kfree+85/b0>
   4:   75 2f                     jne    35 <_EIP+0x35>
Code;  c013bf3b <kfree+87/b0>
   6:   24 c0                     and    $0xc0,%al
Code;  c013bf3d <kfree+89/b0>
   8:   8b 15 c4 bf 29 c0         mov    0xc029bfc4,%edx
Code;  c013bf43 <kfree+8f/b0>
   e:   eb af                     jmp    ffffffbf <_EIP+0xffffffbf>
Code;  c013bf45 <kfree+91/b0>
  10:   8d 76 00                  lea    0x0(%esi),%esi
Code;  c013bf48 <kfree+94/b0>
  13:   57                        push   %edi

 <0>Kernel panic: Attempted to kill the idle task!
---

Jerome

Re: [Oops] i386 mm/slab.c (cache_flusharray)

[Nathan Scott]

On Tue, Dec 02, 2003 at 10:05:40AM -0800, Mike Fedyk wrote:
> On Tue, Dec 02, 2003 at 05:44:18PM +1100, Nathan Scott wrote:
> > I'm not seeing anything to suggest random slab corruption, and I'm
> > so far unable to trip things up as easily as you're able to Jerome.
> > Do you have just a very small amount of memory perhaps?  I can try
> > running while very low on memory, but thats the only other obvious
> > thing I can think of atm.
> 
> How about XFS on DM on RAID?

I didn't see references to those in Jeromes original description,
so haven't been testing those in this context.  But I need to do
more testing on those subsystems anyway, so, will do.

cheers.

-- 
Nathan

Re: [Oops] i386 mm/slab.c (cache_flusharray)

[Mike Fedyk]

On Tue, Dec 02, 2003 at 05:44:18PM +1100, Nathan Scott wrote:
> I'm not seeing anything to suggest random slab corruption, and I'm
> so far unable to trip things up as easily as you're able to Jerome.
> Do you have just a very small amount of memory perhaps?  I can try
> running while very low on memory, but thats the only other obvious
> thing I can think of atm.

How about XFS on DM on RAID?

Re: [Oops] i386 mm/slab.c (cache_flusharray)

[Nathan Scott]

Hi there,

On Tue, Dec 02, 2003 at 12:37:16PM +1100, Nathan Scott wrote:
> On Mon, Dec 01, 2003 at 04:36:33PM -0800, Linus Torvalds wrote:
> > 
> > I assume it's not an option to try another filesystem on this setup, but
> > it's entirely possible that the 2.6.x buffer-head removal has impacted XFS
> > negatively - although I'm a bit surprised at how easily you seem to show
> > problems, since XFS actually has active maintenance.
> > 
> > Nathan - I don't know if you follow linux-kernel, but Jerome Pinot has
> 
> Yep, although I try to filter out "noise" and have inadvertently
> missed this discussion so far.
> 
> > been having bad slab problems for some time now. Do normal XFS users
> > compile with slab debugging turned on?
> 
> Hmm - I know I do - my nightly QA testing runs with this set.
> Let me dig through the archives and catch up a bit on this issue;
> I'll get back to you.

OK, I've run XFS through hours and hours of very heavy stress now,
using a variety of different tests, and have tried different mount
and mkfs options as well.  And with a few kernel compiles thrown in
in the background for good measure.  Either we have quite different
hardware configs, compilers, etc; or this is something else.  This
was done with preempt enabled too (which I usually test without).

I'm not seeing anything to suggest random slab corruption, and I'm
so far unable to trip things up as easily as you're able to Jerome.
Do you have just a very small amount of memory perhaps?  I can try
running while very low on memory, but thats the only other obvious
thing I can think of atm.

cheers.

-- 
Nathan

Re: [Oops] i386 mm/slab.c (cache_flusharray)

[Nathan Scott]

On Mon, Dec 01, 2003 at 04:36:33PM -0800, Linus Torvalds wrote:
> 
> I assume it's not an option to try another filesystem on this setup, but
> it's entirely possible that the 2.6.x buffer-head removal has impacted XFS
> negatively - although I'm a bit surprised at how easily you seem to show
> problems, since XFS actually has active maintenance.
> 
> Nathan - I don't know if you follow linux-kernel, but Jerome Pinot has

Yep, although I try to filter out "noise" and have inadvertently
missed this discussion so far.

> been having bad slab problems for some time now. Do normal XFS users
> compile with slab debugging turned on?

Hmm - I know I do - my nightly QA testing runs with this set.
Let me dig through the archives and catch up a bit on this issue;
I'll get back to you.

thanks.

-- 
Nathan

Re: [Oops] i386 mm/slab.c (cache_flusharray)

[Mike Fedyk]

On Thu, Nov 27, 2003 at 07:42:39PM +0100, pinotj@club-internet.fr wrote:
> first, some news
> 
> 2.6.0-test11 makes same oops during second compilation of kernel. The vanilla kernel with PREEMPT always oops the same way. No matter, it's always reproductible.
> 
> 2.6.0-test11 + Manfred's patch doesn't hang but I found a slab error in the logs that occured during a compilation. (I didn't find this for -test10, I was lucky ?)
> 
> So, there is no more way for my system to run a kernel > -test9 without problem.

Can you try some of the test9-bkN kernels and see where the problem starts?

Re: [Oops] i386 mm/slab.c (cache_flusharray)

[Manfred Spraul]

pinotj@club-internet.fr wrote:

>Thanks for your explanation.
>Should I try with L1 and/or L2 cache disable on my computer (I don't know if it's safe) ?
>I trust my hardware but it's better to get some facts.
>
No, it wouldn't help. Something in the kernel randomly corrupts memory. 
I'm certain that it's not slab. I'm also fairly certain that it's not 
the hardware - IBM guys reproduced corruptions on both ppc64 and i386 
systems (bugzilla 1097 and 1497). The corrupted object is the slab 
structure or the bufctl entries - data near the beginning of a page. But 
I have no idea how to pinpoint it.

--
    Manfred

Re: [Oops] i386 mm/slab.c (cache_flusharray)

[Manfred Spraul]

Linus Torvalds wrote:

>On Tue, 25 Nov 2003 pinotj@club-internet.fr wrote:
>  
>
>>3. 2.6.0-test10 vanilla + PREEMPT_CONFIG=y + patch printk + patch magic numbers
>>The patch solves the problem, I can compile 4 times a kernel and do heavy work in parallele (load average around 1.2 during 2 hours) without any problems.
>>    
>>
>
>Those magic numbers don't make any sense. In particular, SLAB_LIMIT is
>clearly bogus both in the original version and in the "magic number
>patch". The only place that uses SLAB_LIMIT is the code that decides how
>many entries fit in one slab, and quite frankly, it makes no _sense_ to
>have a SLAB_LIMIT that is big enough to be unsigned.
>
Object numbers  (kmem_bufctl_t) are unsigned, but some values have a 
special meaning:
"-1" is the magic value for end-of-list.
"-2" is the magic value for object in use.
All other values are valid object numbers. Right now object numbers are 
unsigned int, but initially I considered unsigned char or unsigned 
short. And then an explicit SLAB_LIMIT is necessary - with unsigned 
char, the limit would be 253 objects per slab, which could be reached if 
someone creates objects smaller than 16 bytes.

In Jerome's case, the debug checks noticed that the object-in-use 
sentinel was not in the bufctl entry during free, instead there was a 
"-1". There are several sources for the "-1": My initial guess was 
either a bug in slab, or a bad memory cell (only one bit difference). 
Thus I sent him a patch that changes multiple bits. Result: It remained 
a single bit change, i.e it's proven that slab doesn't write BUFCTL_END 
into the wrong slot.

--
    Manfred

Re: [Oops] i386 mm/slab.c (cache_flusharray)

[Manfred Spraul]

pinotj@club-internet.fr wrote:

>c6fd7870: redzone 1: 0x170fc2a5, redzone 2: 0x160fc2a5.
>
Single bit error: redzone 2 must be 0x170fc2a5.

>---
>System looks OK, I tried a second compilation just after and this time I got an oops:
>---
>slab: double free detected in cache 'buffer_head', objp cc3f9798, objnr 26, slabp cc3f9000, s_mem cc3f9180 bufctl f7ffffff.
>  
>
Good.

+#define BUFCTL_END	0xfeffFFFF
+#define BUFCTL_FREE	0xf7ffFFFE
+#define	SLAB_LIMIT	0xf0ffFFFD

f7ffffff is not a valid value, slab never writes that into a bufctl. 
Someone did a ++ or "|= 1", or a hw bug.
I think the Athlon cpus have ECC for the L2 cache - could you check in 
the bios that ECC checking is enabled?

--
    Manfred

Re: [Oops] i386 mm/slab.c (cache_flusharray)

[Manfred Spraul]

[-- Attachment #1: Type: text/plain, Size: 1016 bytes --]

pinotj@club-internet.fr wrote:

>0. Increase verbosity of the printk (thanks to Manfred):
>(compilation of kernel)
>slab: double free detected in cache 'buffer_head', objp c4c8e3d8, objnr 10,
>slabp c4c8e000, s_mem c4c8e180, bufctl ffffffff.
>(compilation of firebird)
>slab: double free detected in cache 'pte_chain', objp c18a6600, objnr 10,
>slabp c18a6000, s_mem c18a6100, bufctl ffffffff.
>  
>
The correct value for the bufctl would be 0xfffffffe - a single bit is 
wrong, but OTHO 0xffffffff is also a valid value.
Two different caches are affected.
The addresses of the corrupted variable differ, the offset into the page 
is identical. I think that rules out bad memory. That leaves a bug in 
slab.c, a bad bit in the L1/L2 cache, or random pointer scribbling.
Jerome, could you try the attached patch? It changes the magic constants 
that are used by slab.c. And please pay attention to the objnr: Is it 
always objnr 10, slabp xxxxx000, or do you see other values as well?

--
    Manfred

--
    Manfred

[-- Attachment #2: x --]
[-- Type: text/plain, Size: 496 bytes --]

--- 2.6/mm/slab.c	2003-11-18 18:18:20.000000000 +0100
+++ build-2.6/mm/slab.c	2003-11-21 19:50:02.000000000 +0100
@@ -153,9 +153,9 @@
  * is less than 512 (PAGE_SIZE<<3), but greater than 256.
  */
 
-#define BUFCTL_END	0xffffFFFF
-#define BUFCTL_FREE	0xffffFFFE
-#define	SLAB_LIMIT	0xffffFFFD
+#define BUFCTL_END	0xfeffFFFF
+#define BUFCTL_FREE	0xf7ffFFFE
+#define	SLAB_LIMIT	0xf0ffFFFD
 typedef unsigned int kmem_bufctl_t;
 
 /* Max number of objs-per-slab for caches which use off-slab slabs.

Re: [Oops] i386 mm/slab.c (cache_flusharray)

[Andrew Morton]

pinotj@club-internet.fr wrote:
>
> kernel BUG at mm/slab.c:1957!
>  ---
> 
>  >Don't know, sorry.
> 
>  Is there any thing I can do to help figure out where does the problem comes from ? 

Well it's interesting that it is repeatable.

First thing to do is to eliminate hardware failures:

1: Is the oops always the same, or does the machine crash in other ways,
   with different backtraces?

2: Try running memtest86 on that machine for 12 hours or more.

3: Can the problem be reproduced on other machines?

4: try a different compiler version.

Thanks.

Re: [Oops] i386 mm/slab.c (cache_flusharray)

[Andrew Morton]

pinotj@club-internet.fr wrote:
>
> kernel BUG at mm/slab.c:1957!
> invalid operand: 0000 [#1]
> CPU:    0
> EIP:    0060:[free_block+336/752]    Not tainted
> EIP:    0060:[<c015ad40>]    Not tainted
> Using defaults from ksymoops -t elf32-i386 -a i386
> EFLAGS: 00010096
> eax: 00000045   ebx: 00000006   ecx: c0693854   edx: c056e4f8
> esi: cd09a000   edi: cd09a018   ebp: cf821c68   esp: cf821c3c
> ds: 007b   es: 007b   ss: 0068
> Stack: c0502240 c0502e1d cd09af18 c0652a00 00000001 0000003a cd09af18 0000000f
>        cffdef08 c4bcd180 00000010 cf821ca0 c015afba cffed800 cffdef08 00000010
>        00000282 c1161ca0 00000000 00000001 cffee730 00000010 00010c00 c4bcd180
> Call Trace:
>  [<c015afba>] cache_flusharray+0xda/0x2b0
>  [<c015b7ad>] kmem_cache_free+0x1ad/0x3a0
>  [<c018158c>] free_buffer_head+0x2c/0x60
>  [<c018158c>] free_buffer_head+0x2c/0x60

urgh, there are several reports of this and it's always the buffer_head
slab.  The code in there is trivial so perhaps it's just that the large
number of buffer_heads makes them a fat target.

You should have also seen the message "slab: double free detected in cache
'buffer_head', objp 0xNNNNNNNN".

Don't know, sorry.

[Oops] i386 mm/slab.c (cache_flusharray)

[pinotj]

Here is an Oops from the kernel 2.6.0-test9 + cset-20031115_0206.txt.gz (means all current patches till 03/11/14 [PATCH] PPC32: cancel syscall restart on signal delivery ).

Seems to come from the cache_flusharray function of mm/slab.c
It occurs in general during heavy load (compilation of kernel or xfree86...), is quite reproductible but not automatic.
Same Oops on different distro/gcc (slack 9.1 et lfs 5.0).
Arch is i386 (AMD athlon-tbird 1.2GHz)

I never had this problem with 2.6.0-test8 and before.

This oops is very annoying, I got it around 5 times a day. I'm sorry but I don't have the skill to patch this. If someone can help :-) Please CC me in answer, I'm not on the lkml.

Regards,

Jerome Pinot

---
ksymoops 2.4.9 on i686 2.6.0-test9.  Options used
     -V (default)
     -K (specified)
     -l /proc/modules (default)
     -o /mnt/lfs/lib/modules/2.6.0-test9 (specified)
     -m /mnt/lfs/boot/System.map (specified)

No modules in ksyms, skipping objects
No ksyms, skipping lsmod
kernel BUG at mm/slab.c:1957!
invalid operand: 0000 [#1]
CPU:    0
EIP:    0060:[free_block+336/752]    Not tainted
EIP:    0060:[<c015ad40>]    Not tainted
Using defaults from ksymoops -t elf32-i386 -a i386
EFLAGS: 00010096
eax: 00000045   ebx: 00000006   ecx: c0693854   edx: c056e4f8
esi: cd09a000   edi: cd09a018   ebp: cf821c68   esp: cf821c3c
ds: 007b   es: 007b   ss: 0068
Stack: c0502240 c0502e1d cd09af18 c0652a00 00000001 0000003a cd09af18 0000000f
       cffdef08 c4bcd180 00000010 cf821ca0 c015afba cffed800 cffdef08 00000010
       00000282 c1161ca0 00000000 00000001 cffee730 00000010 00010c00 c4bcd180
Call Trace:
 [<c015afba>] cache_flusharray+0xda/0x2b0
 [<c015b7ad>] kmem_cache_free+0x1ad/0x3a0
 [<c018158c>] free_buffer_head+0x2c/0x60
 [<c018158c>] free_buffer_head+0x2c/0x60
 [<c0181353>] try_to_free_buffers+0x143/0x220
 [<c0294bab>] linvfs_release_page+0x7b/0x80
 [<c017f243>] try_to_release_page+0x53/0x60
 [<c015fec7>] shrink_list+0x8b7/0xac0
 [<c010cadc>] common_interrupt+0x18/0x20
 [<c015e489>] __pagevec_release+0x29/0x40
 [<c01602d5>] shrink_cache+0x205/0x610
 [<c0161228>] shrink_zone+0x78/0xa0
 [<c01615fa>] balance_pgdat+0x17a/0x200
 [<c0161759>] kswapd+0xd9/0xf0
 [<c01274f0>] autoremove_wake_function+0x0/0x50
 [<c010c046>] ret_from_fork+0x6/0x14
 [<c01274f0>] autoremove_wake_function+0x0/0x50
 [<c0161680>] kswapd+0x0/0xf0
 [<c01092a9>] kernel_thread_helper+0x5/0xc
Code: 0f 0b a5 07 82 15 50 c0 8b 46 14 8b 4d e8 31 db 89 04 8f 89


>>EIP; c015ad40 <free_block+150/2f0>   <=====

>>ecx; c0693854 <per_cpu__runqueues+4b4/940>
>>edx; c056e4f8 <log_wait+18/20>
>>esi; cd09a000 <__crc_unregister_sound_dsp+164f0/974e1>
>>edi; cd09a018 <__crc_unregister_sound_dsp+16508/974e1>
>>ebp; cf821c68 <__crc_scsi_register_driver+6c7b9/1f2ec0>
>>esp; cf821c3c <__crc_scsi_register_driver+6c78d/1f2ec0>

Trace; c015afba <cache_flusharray+da/2b0>
Trace; c015b7ad <kmem_cache_free+1ad/3a0>
Trace; c018158c <free_buffer_head+2c/60>
Trace; c018158c <free_buffer_head+2c/60>
Trace; c0181353 <try_to_free_buffers+143/220>
Trace; c0294bab <linvfs_release_page+7b/80>
Trace; c017f243 <try_to_release_page+53/60>
Trace; c015fec7 <shrink_list+8b7/ac0>
Trace; c010cadc <common_interrupt+18/20>
Trace; c015e489 <__pagevec_release+29/40>
Trace; c01602d5 <shrink_cache+205/610>
Trace; c0161228 <shrink_zone+78/a0>
Trace; c01615fa <balance_pgdat+17a/200>
Trace; c0161759 <kswapd+d9/f0>
Trace; c01274f0 <autoremove_wake_function+0/50>
Trace; c010c046 <ret_from_fork+6/14>
Trace; c01274f0 <autoremove_wake_function+0/50>
Trace; c0161680 <kswapd+0/f0>
Trace; c01092a9 <kernel_thread_helper+5/c>

Code;  c015ad40 <free_block+150/2f0>
00000000 <_EIP>:
Code;  c015ad40 <free_block+150/2f0>   <=====
   0:   0f 0b                     ud2a      <=====
Code;  c015ad42 <free_block+152/2f0>
   2:   a5                        movsl  %ds:(%esi),%es:(%edi)
Code;  c015ad43 <free_block+153/2f0>
   3:   07                        pop    %es
Code;  c015ad44 <free_block+154/2f0>
   4:   82                        (bad)
Code;  c015ad45 <free_block+155/2f0>
   5:   15 50 c0 8b 46            adc    $0x468bc050,%eax
Code;  c015ad4a <free_block+15a/2f0>
   a:   14 8b                     adc    $0x8b,%al
Code;  c015ad4c <free_block+15c/2f0>
   c:   4d                        dec    %ebp
Code;  c015ad4d <free_block+15d/2f0>
   d:   e8 31 db 89 04            call   489db43 <_EIP+0x489db43>
Code;  c015ad52 <free_block+162/2f0>
  12:   8f 89 00 00 00 00         popl   0x0(%ecx)

Unable to handle kernel paging request at virtual address 00100104
c015ac3c
*pde = 00000000
Oops: 0002 [#2]
CPU:    0
EIP:    0060:[free_block+76/752]    Not tainted
EIP:    0060:[<c015ac3c>]    Not tainted
EFLAGS: 00010017
eax: 00100100   ebx: 00000000   ecx: cd09a810   edx: 00200200
esi: cd09a000   edi: c7430018   ebp: c558bafc   esp: c558bad0
ds: 007b   es: 007b   ss: 0068
Stack: c558badc 00000086 80010c00 ccf77000 cffdc080 0000001a cd09a810 0000000b 
       cffdef08 cd09a180 00000010 c558bb34 c015afba cffed800 cffdef08 00000010
       00000286 c11d0c68 00000000 00000001 cffee730 00000010 00010c00 cd09a180 
Call Trace:
 [<c015afba>] cache_flusharray+0xda/0x2b0
 [<c015b7ad>] kmem_cache_free+0x1ad/0x3a0
 [<c018158c>] free_buffer_head+0x2c/0x60
 [<c018158c>] free_buffer_head+0x2c/0x60
 [<c0181353>] try_to_free_buffers+0x143/0x220
 [<c0294bab>] linvfs_release_page+0x7b/0x80
 [<c017f243>] try_to_release_page+0x53/0x60
 [<c015fec7>] shrink_list+0x8b7/0xac0
 [<c0123eef>] schedule+0x36f/0x8a0
 [<c015e489>] __pagevec_release+0x29/0x40
 [<c01602d5>] shrink_cache+0x205/0x610
 [<c0127515>] autoremove_wake_function+0x25/0x50
 [<c0161228>] shrink_zone+0x78/0xa0
 [<c01612ee>] shrink_caches+0x9e/0xc0
 [<c01613b7>] try_to_free_pages+0xa7/0x170
 [<c015541a>] __alloc_pages+0x1fa/0x380
 [<c0165c17>] do_anonymous_page+0xc7/0x520
 [<c017a899>] do_sync_write+0x89/0xc0
 [<c01668f2>] handle_mm_fault+0x132/0x310
 [<c0121940>] do_page_fault+0x350/0x56a
 [<c01698de>] do_brk+0x14e/0x230
 [<c016790e>] sys_brk+0xee/0x120
 [<c01215f0>] do_page_fault+0x0/0x56a
 [<c010cb79>] error_code+0x2d/0x38
Code: 89 50 04 89 02 c7 46 04 00 02 20 00 c7 06 00 01 10 00 89 c8


>>EIP; c015ac3c <free_block+4c/2f0>   <=====

>>eax; 00100100 <__crc_prepare_to_wait_exclusive+ce3e5/1424fd>
>>ecx; cd09a810 <__crc_unregister_sound_dsp+16d00/974e1>
>>edx; 00200200 <__crc___user_walk+3d8ad/1cb584>
>>esi; cd09a000 <__crc_unregister_sound_dsp+164f0/974e1>
>>edi; c7430018 <__crc_inet_getsockopt+60fc6/20c3c2>
>>ebp; c558bafc <__crc_fat_write_inode+336fc/668510>
>>esp; c558bad0 <__crc_fat_write_inode+336d0/668510>

Trace; c015afba <cache_flusharray+da/2b0>
Trace; c015b7ad <kmem_cache_free+1ad/3a0>
Trace; c018158c <free_buffer_head+2c/60>
Trace; c018158c <free_buffer_head+2c/60>
Trace; c0181353 <try_to_free_buffers+143/220>
Trace; c0294bab <linvfs_release_page+7b/80>
Trace; c017f243 <try_to_release_page+53/60>
Trace; c015fec7 <shrink_list+8b7/ac0>
Trace; c0123eef <schedule+36f/8a0>
Trace; c015e489 <__pagevec_release+29/40>
Trace; c01602d5 <shrink_cache+205/610>
Trace; c0127515 <autoremove_wake_function+25/50>
Trace; c0161228 <shrink_zone+78/a0>
Trace; c01612ee <shrink_caches+9e/c0>
Trace; c01613b7 <try_to_free_pages+a7/170>
Trace; c015541a <__alloc_pages+1fa/380>
Trace; c0165c17 <do_anonymous_page+c7/520>
Trace; c017a899 <do_sync_write+89/c0>
Trace; c01668f2 <handle_mm_fault+132/310>
Trace; c0121940 <do_page_fault+350/56a>
Trace; c01698de <do_brk+14e/230>
Trace; c016790e <sys_brk+ee/120>
Trace; c01215f0 <do_page_fault+0/56a>
Trace; c010cb79 <error_code+2d/38>

Code;  c015ac3c <free_block+4c/2f0>
00000000 <_EIP>:
Code;  c015ac3c <free_block+4c/2f0>   <=====
   0:   89 50 04                  mov    %edx,0x4(%eax)   <=====
Code;  c015ac3f <free_block+4f/2f0>
   3:   89 02                     mov    %eax,(%edx)
Code;  c015ac41 <free_block+51/2f0>
   5:   c7 46 04 00 02 20 00      movl   $0x200200,0x4(%esi)
Code;  c015ac48 <free_block+58/2f0>
   c:   c7 06 00 01 10 00         movl   $0x100100,(%esi)
Code;  c015ac4e <free_block+5e/2f0>
  12:   89 c8                     mov    %ecx,%eax

Call Trace:
 [<c0124411>] schedule+0x891/0x8a0
 [<c0163ae1>] unmap_page_range+0x41/0x70
 [<c0163d24>] unmap_vmas+0x214/0x330
 [<c0169adb>] exit_mmap+0xcb/0x2b0
 [<c0127935>] mmput+0xb5/0x150
 [<c012e062>] do_exit+0x1b2/0x960
 [<c01215f0>] do_page_fault+0x0/0x56a
 [<c01215f0>] do_page_fault+0x0/0x56a
 [<c010d367>] die+0x227/0x230
 [<c01217d6>] do_page_fault+0x1e6/0x56a
 [<c0122d1e>] recalc_task_prio+0x8e/0x1b0
 [<c0122f9a>] try_to_wake_up+0x15a/0x290
 [<c01215f0>] do_page_fault+0x0/0x56a
 [<c010cb79>] error_code+0x2d/0x38
 [<c015ac3c>] free_block+0x4c/0x2f0
 [<c015afba>] cache_flusharray+0xda/0x2b0
 [<c015b7ad>] kmem_cache_free+0x1ad/0x3a0
 [<c018158c>] free_buffer_head+0x2c/0x60
 [<c018158c>] free_buffer_head+0x2c/0x60
 [<c0181353>] try_to_free_buffers+0x143/0x220
 [<c0294bab>] linvfs_release_page+0x7b/0x80
 [<c017f243>] try_to_release_page+0x53/0x60
 [<c015fec7>] shrink_list+0x8b7/0xac0
 [<c0123eef>] schedule+0x36f/0x8a0
 [<c015e489>] __pagevec_release+0x29/0x40
 [<c01602d5>] shrink_cache+0x205/0x610
 [<c0127515>] autoremove_wake_function+0x25/0x50
 [<c0161228>] shrink_zone+0x78/0xa0
 [<c01612ee>] shrink_caches+0x9e/0xc0
 [<c01613b7>] try_to_free_pages+0xa7/0x170
 [<c015541a>] __alloc_pages+0x1fa/0x380
 [<c0165c17>] do_anonymous_page+0xc7/0x520
 [<c017a899>] do_sync_write+0x89/0xc0
 [<c01668f2>] handle_mm_fault+0x132/0x310
 [<c0121940>] do_page_fault+0x350/0x56a
 [<c01698de>] do_brk+0x14e/0x230
 [<c016790e>] sys_brk+0xee/0x120
 [<c01215f0>] do_page_fault+0x0/0x56a
 [<c010cb79>] error_code+0x2d/0x38
Call Trace:
 [<c0124411>] schedule+0x891/0x8a0
 [<c0163ae1>] unmap_page_range+0x41/0x70
 [<c0163d24>] unmap_vmas+0x214/0x330
 [<c0169adb>] exit_mmap+0xcb/0x2b0
 [<c0127935>] mmput+0xb5/0x150
 [<c012e062>] do_exit+0x1b2/0x960
 [<c01215f0>] do_page_fault+0x0/0x56a
 [<c01215f0>] do_page_fault+0x0/0x56a
 [<c010d367>] die+0x227/0x230
 [<c01217d6>] do_page_fault+0x1e6/0x56a
 [<c0122d1e>] recalc_task_prio+0x8e/0x1b0
 [<c0122f9a>] try_to_wake_up+0x15a/0x290
 [<c01215f0>] do_page_fault+0x0/0x56a
 [<c010cb79>] error_code+0x2d/0x38
 [<c015ac3c>] free_block+0x4c/0x2f0
 [<c015afba>] cache_flusharray+0xda/0x2b0
 [<c015b7ad>] kmem_cache_free+0x1ad/0x3a0
 [<c018158c>] free_buffer_head+0x2c/0x60
 [<c018158c>] free_buffer_head+0x2c/0x60
 [<c0181353>] try_to_free_buffers+0x143/0x220
 [<c0294bab>] linvfs_release_page+0x7b/0x80
 [<c017f243>] try_to_release_page+0x53/0x60
 [<c015fec7>] shrink_list+0x8b7/0xac0
 [<c0123eef>] schedule+0x36f/0x8a0
 [<c015e489>] __pagevec_release+0x29/0x40
 [<c01602d5>] shrink_cache+0x205/0x610
 [<c0127515>] autoremove_wake_function+0x25/0x50
 [<c0161228>] shrink_zone+0x78/0xa0
 [<c01612ee>] shrink_caches+0x9e/0xc0
 [<c01613b7>] try_to_free_pages+0xa7/0x170
 [<c015541a>] __alloc_pages+0x1fa/0x380
 [<c0165c17>] do_anonymous_page+0xc7/0x520
 [<c017a899>] do_sync_write+0x89/0xc0
 [<c01668f2>] handle_mm_fault+0x132/0x310
 [<c0121940>] do_page_fault+0x350/0x56a
 [<c01698de>] do_brk+0x14e/0x230
 [<c016790e>] sys_brk+0xee/0x120
 [<c01215f0>] do_page_fault+0x0/0x56a
 [<c010cb79>] error_code+0x2d/0x38
Call Trace:
 [<c0124411>] schedule+0x891/0x8a0
 [<c0163ae1>] unmap_page_range+0x41/0x70
 [<c0163d24>] unmap_vmas+0x214/0x330
 [<c0169adb>] exit_mmap+0xcb/0x2b0
 [<c0127935>] mmput+0xb5/0x150
 [<c012e062>] do_exit+0x1b2/0x960
 [<c01215f0>] do_page_fault+0x0/0x56a
 [<c01215f0>] do_page_fault+0x0/0x56a
 [<c010d367>] die+0x227/0x230
 [<c01217d6>] do_page_fault+0x1e6/0x56a
 [<c0122d1e>] recalc_task_prio+0x8e/0x1b0
 [<c0122f9a>] try_to_wake_up+0x15a/0x290
 [<c01215f0>] do_page_fault+0x0/0x56a
 [<c010cb79>] error_code+0x2d/0x38
 [<c015ac3c>] free_block+0x4c/0x2f0
 [<c015afba>] cache_flusharray+0xda/0x2b0
 [<c015b7ad>] kmem_cache_free+0x1ad/0x3a0
 [<c018158c>] free_buffer_head+0x2c/0x60
 [<c018158c>] free_buffer_head+0x2c/0x60
 [<c0181353>] try_to_free_buffers+0x143/0x220
 [<c0294bab>] linvfs_release_page+0x7b/0x80
 [<c017f243>] try_to_release_page+0x53/0x60
 [<c015fec7>] shrink_list+0x8b7/0xac0
 [<c0123eef>] schedule+0x36f/0x8a0
 [<c015e489>] __pagevec_release+0x29/0x40
 [<c01602d5>] shrink_cache+0x205/0x610
 [<c0127515>] autoremove_wake_function+0x25/0x50
 [<c0161228>] shrink_zone+0x78/0xa0
 [<c01612ee>] shrink_caches+0x9e/0xc0
 [<c01613b7>] try_to_free_pages+0xa7/0x170
 [<c015541a>] __alloc_pages+0x1fa/0x380
 [<c0165c17>] do_anonymous_page+0xc7/0x520
 [<c017a899>] do_sync_write+0x89/0xc0
 [<c01668f2>] handle_mm_fault+0x132/0x310
 [<c0121940>] do_page_fault+0x350/0x56a
 [<c01698de>] do_brk+0x14e/0x230
 [<c016790e>] sys_brk+0xee/0x120
 [<c01215f0>] do_page_fault+0x0/0x56a
 [<c010cb79>] error_code+0x2d/0x38
Call Trace:
 [<c0126b1b>] __might_sleep+0xab/0xd0
 [<c01677b9>] remove_shared_vm_struct+0x39/0xa0
 [<c0169be1>] exit_mmap+0x1d1/0x2b0
 [<c0127935>] mmput+0xb5/0x150
 [<c012e062>] do_exit+0x1b2/0x960
 [<c01215f0>] do_page_fault+0x0/0x56a
 [<c01215f0>] do_page_fault+0x0/0x56a
 [<c010d367>] die+0x227/0x230
 [<c01217d6>] do_page_fault+0x1e6/0x56a
 [<c0122d1e>] recalc_task_prio+0x8e/0x1b0
 [<c0122f9a>] try_to_wake_up+0x15a/0x290
 [<c01215f0>] do_page_fault+0x0/0x56a
 [<c010cb79>] error_code+0x2d/0x38
 [<c015ac3c>] free_block+0x4c/0x2f0
 [<c015b7ad>] kmem_cache_free+0x1ad/0x3a0
 [<c018158c>] free_buffer_head+0x2c/0x60
 [<c018158c>] free_buffer_head+0x2c/0x60
 [<c0181353>] try_to_free_buffers+0x143/0x220
 [<c0294bab>] linvfs_release_page+0x7b/0x80
 [<c017f243>] try_to_release_page+0x53/0x60
 [<c015fec7>] shrink_list+0x8b7/0xac0
 [<c0123eef>] schedule+0x36f/0x8a0
 [<c015e489>] __pagevec_release+0x29/0x40
 [<c01602d5>] shrink_cache+0x205/0x610
 [<c0127515>] autoremove_wake_function+0x25/0x50
 [<c0161228>] shrink_zone+0x78/0xa0
 [<c01612ee>] shrink_caches+0x9e/0xc0
 [<c01613b7>] try_to_free_pages+0xa7/0x170
 [<c015541a>] __alloc_pages+0x1fa/0x380
 [<c0165c17>] do_anonymous_page+0xc7/0x520
 [<c017a899>] do_sync_write+0x89/0xc0
 [<c01668f2>] handle_mm_fault+0x132/0x310
 [<c0121940>] do_page_fault+0x350/0x56a
 [<c01698de>] do_brk+0x14e/0x230
 [<c016790e>] sys_brk+0xee/0x120
 [<c01215f0>] do_page_fault+0x0/0x56a
 [<c010cb79>] error_code+0x2d/0x38
Warning (Oops_read): Code line not seen, dumping what data is available


Trace; c0124411 <schedule+891/8a0>
Trace; c0163ae1 <unmap_page_range+41/70>
Trace; c0163d24 <unmap_vmas+214/330>
Trace; c0169adb <exit_mmap+cb/2b0>
Trace; c0127935 <mmput+b5/150>
Trace; c012e062 <do_exit+1b2/960>
Trace; c01215f0 <do_page_fault+0/56a>
Trace; c01215f0 <do_page_fault+0/56a>
Trace; c010d367 <die+227/230>
Trace; c01217d6 <do_page_fault+1e6/56a>
Trace; c0122d1e <recalc_task_prio+8e/1b0>
Trace; c0122f9a <try_to_wake_up+15a/290>
Trace; c01215f0 <do_page_fault+0/56a>
Trace; c010cb79 <error_code+2d/38>
Trace; c015ac3c <free_block+4c/2f0>
Trace; c015afba <cache_flusharray+da/2b0>
Trace; c015b7ad <kmem_cache_free+1ad/3a0>
Trace; c018158c <free_buffer_head+2c/60>
Trace; c018158c <free_buffer_head+2c/60>
Trace; c0181353 <try_to_free_buffers+143/220>
Trace; c0294bab <linvfs_release_page+7b/80>
Trace; c017f243 <try_to_release_page+53/60>
Trace; c015fec7 <shrink_list+8b7/ac0>
Trace; c0123eef <schedule+36f/8a0>
Trace; c015e489 <__pagevec_release+29/40>
Trace; c01602d5 <shrink_cache+205/610>
Trace; c0127515 <autoremove_wake_function+25/50>
Trace; c0161228 <shrink_zone+78/a0>
Trace; c01612ee <shrink_caches+9e/c0>
Trace; c01613b7 <try_to_free_pages+a7/170>
Trace; c015541a <__alloc_pages+1fa/380>
Trace; c0165c17 <do_anonymous_page+c7/520>
Trace; c017a899 <do_sync_write+89/c0>
Trace; c01668f2 <handle_mm_fault+132/310>
Trace; c0121940 <do_page_fault+350/56a>
Trace; c01698de <do_brk+14e/230>
Trace; c016790e <sys_brk+ee/120>
Trace; c01215f0 <do_page_fault+0/56a>
Trace; c010cb79 <error_code+2d/38>
Trace; c0124411 <schedule+891/8a0>
Trace; c0163ae1 <unmap_page_range+41/70>
Trace; c0163d24 <unmap_vmas+214/330>
Trace; c0169adb <exit_mmap+cb/2b0>
Trace; c0127935 <mmput+b5/150>
Trace; c012e062 <do_exit+1b2/960>
Trace; c01215f0 <do_page_fault+0/56a>
Trace; c01215f0 <do_page_fault+0/56a>
Trace; c010d367 <die+227/230>
Trace; c01217d6 <do_page_fault+1e6/56a>
Trace; c0122d1e <recalc_task_prio+8e/1b0>
Trace; c0122f9a <try_to_wake_up+15a/290>
Trace; c01215f0 <do_page_fault+0/56a>
Trace; c010cb79 <error_code+2d/38>
Trace; c015ac3c <free_block+4c/2f0>
Trace; c015afba <cache_flusharray+da/2b0>
Trace; c015b7ad <kmem_cache_free+1ad/3a0>
Trace; c018158c <free_buffer_head+2c/60>
Trace; c018158c <free_buffer_head+2c/60>
Trace; c0181353 <try_to_free_buffers+143/220>
Trace; c0294bab <linvfs_release_page+7b/80>
Trace; c017f243 <try_to_release_page+53/60>
Trace; c015fec7 <shrink_list+8b7/ac0>
Trace; c0123eef <schedule+36f/8a0>
Trace; c015e489 <__pagevec_release+29/40>
Trace; c01602d5 <shrink_cache+205/610>
Trace; c0127515 <autoremove_wake_function+25/50>
Trace; c0161228 <shrink_zone+78/a0>
Trace; c01612ee <shrink_caches+9e/c0>
Trace; c01613b7 <try_to_free_pages+a7/170>
Trace; c015541a <__alloc_pages+1fa/380>
Trace; c0165c17 <do_anonymous_page+c7/520>
Trace; c017a899 <do_sync_write+89/c0>
Trace; c01668f2 <handle_mm_fault+132/310>
Trace; c0121940 <do_page_fault+350/56a>
Trace; c01698de <do_brk+14e/230>
Trace; c016790e <sys_brk+ee/120>
Trace; c01215f0 <do_page_fault+0/56a>
Trace; c010cb79 <error_code+2d/38>
Trace; c0124411 <schedule+891/8a0>
Trace; c0163ae1 <unmap_page_range+41/70>
Trace; c0163d24 <unmap_vmas+214/330>
Trace; c0169adb <exit_mmap+cb/2b0>
Trace; c0127935 <mmput+b5/150>
Trace; c012e062 <do_exit+1b2/960>
Trace; c01215f0 <do_page_fault+0/56a>
Trace; c01215f0 <do_page_fault+0/56a>
Trace; c010d367 <die+227/230>
Trace; c01217d6 <do_page_fault+1e6/56a>
Trace; c0122d1e <recalc_task_prio+8e/1b0>
Trace; c0122f9a <try_to_wake_up+15a/290>
Trace; c01215f0 <do_page_fault+0/56a>
Trace; c010cb79 <error_code+2d/38>
Trace; c015ac3c <free_block+4c/2f0>
Trace; c015afba <cache_flusharray+da/2b0>
Trace; c015b7ad <kmem_cache_free+1ad/3a0>
Trace; c018158c <free_buffer_head+2c/60>
Trace; c018158c <free_buffer_head+2c/60>
Trace; c0181353 <try_to_free_buffers+143/220>
Trace; c0294bab <linvfs_release_page+7b/80>
Trace; c017f243 <try_to_release_page+53/60>
Trace; c015fec7 <shrink_list+8b7/ac0>
Trace; c0123eef <schedule+36f/8a0>
Trace; c015e489 <__pagevec_release+29/40>
Trace; c01602d5 <shrink_cache+205/610>
Trace; c0127515 <autoremove_wake_function+25/50>
Trace; c0161228 <shrink_zone+78/a0>
Trace; c01612ee <shrink_caches+9e/c0>
Trace; c01613b7 <try_to_free_pages+a7/170>
Trace; c015541a <__alloc_pages+1fa/380>
Trace; c0165c17 <do_anonymous_page+c7/520>
Trace; c017a899 <do_sync_write+89/c0>
Trace; c01668f2 <handle_mm_fault+132/310>
Trace; c0121940 <do_page_fault+350/56a>
Trace; c01698de <do_brk+14e/230>
Trace; c016790e <sys_brk+ee/120>
Trace; c01215f0 <do_page_fault+0/56a>
Trace; c010cb79 <error_code+2d/38>
Trace; c0126b1b <__might_sleep+ab/d0>
Trace; c01677b9 <remove_shared_vm_struct+39/a0>
Trace; c0169be1 <exit_mmap+1d1/2b0>
Trace; c0127935 <mmput+b5/150>
Trace; c012e062 <do_exit+1b2/960>
Trace; c01215f0 <do_page_fault+0/56a>
Trace; c01215f0 <do_page_fault+0/56a>
Trace; c010d367 <die+227/230>
Trace; c01217d6 <do_page_fault+1e6/56a>
Trace; c0122d1e <recalc_task_prio+8e/1b0>
Trace; c0122f9a <try_to_wake_up+15a/290>
Trace; c01215f0 <do_page_fault+0/56a>
Trace; c010cb79 <error_code+2d/38>
Trace; c015ac3c <free_block+4c/2f0>
Trace; c015b7ad <kmem_cache_free+1ad/3a0>
Trace; c018158c <free_buffer_head+2c/60>
Trace; c018158c <free_buffer_head+2c/60>
Trace; c0181353 <try_to_free_buffers+143/220>
Trace; c0294bab <linvfs_release_page+7b/80>
Trace; c017f243 <try_to_release_page+53/60>
Trace; c015fec7 <shrink_list+8b7/ac0>
Trace; c0123eef <schedule+36f/8a0>
Trace; c015e489 <__pagevec_release+29/40>
Trace; c01602d5 <shrink_cache+205/610>
Trace; c0127515 <autoremove_wake_function+25/50>
Trace; c0161228 <shrink_zone+78/a0>
Trace; c01612ee <shrink_caches+9e/c0>
Trace; c01613b7 <try_to_free_pages+a7/170>
Trace; c015541a <__alloc_pages+1fa/380>
Trace; c0165c17 <do_anonymous_page+c7/520>
Trace; c017a899 <do_sync_write+89/c0>
Trace; c01668f2 <handle_mm_fault+132/310>
Trace; c0121940 <do_page_fault+350/56a>
Trace; c01698de <do_brk+14e/230>
Trace; c016790e <sys_brk+ee/120>
Trace; c01215f0 <do_page_fault+0/56a>
Trace; c010cb79 <error_code+2d/38>