Re: [Oops] i386 mm/slab.c (cache_flusharray)

Re: [Oops] i386 mm/slab.c (cache_flusharray)

[pinotj]

OK, I tried again the patch on "small kernel" test11 with CONFIG_DEBUG_PAGEALLOC only. Here are the first results. I will do more tests later because it seems weird. This time I have very different behavior for XFS and Ext3.

I got an oops at boot time, when system try to mount root filesystem (XFS).

But when I tried on a root Ext3, I got no problem at all. I even compiled 2 kernel straight without any problems. It seems to be the first time a test11 works flawless on my system.

Of course, XFS and Ext3 were respectively enabled in the kernels.

I hope I didn't make any mistake, I will confirm tomorrow.


Jerome Pinot

The XFS oops (I was too lazy to write down all the backtraces, all about xfs_* things, I can send if needed):

---
kernel BUG at include/linux/mm.h:267!
invalid operand: 0000 [#1]
CPU:    0
EIP:    0060:[<c01aa5ac>]    Not tainted
Using defaults from ksymoops -t elf32-i386 -a i386
EFLAGS: 00010246
eax: 00000000   ebx: cef3d000   ecx: cef3c000   edx: c1254e28
esi: 00310020   edi: 00000001   ebp: c12f3b18   esp: c12f3b0c
ds: 007b   es: 007b   ss: 0068
Stack: 00000100 00000000 ceee0000 c12f3b28 c01aaf90 c0253d40 cef3d000 c12f3b7c
       c019669b cef3d000 00000100 00000000 00000100 00000000 00000a02 00000000
       00000100 00000000 cef3d000 00000100 00000000 00000a02 00000000 00000802
Call Trace:
 [<c01aaf90>] pagebuf_free+0x24/0x30
 [<c019669b>] xlog_find_verify_cycle+0x18b/0x1e0
Code: 0f 0b 0b 01 d6 cb 1f c0 eb 8e ff 73 54 eb b3 90 53 e8 0e 11


>>EIP; c01aa5ac <_pagebuf_free_object+108/134>   <=====

>>ebx; cef3d000 <_end+ece0e4c/3fda1e4c>
>>ecx; cef3c000 <_end+ecdfe4c/3fda1e4c>
>>edx; c1254e28 <_end+ff8c74/3fda1e4c>
>>ebp; c12f3b18 <_end+1097964/3fda1e4c>
>>esp; c12f3b0c <_end+1097958/3fda1e4c>

Trace; c01aaf90 <pagebuf_free+24/30>
Trace; c019669b <xlog_find_verify_cycle+18b/1e0>

Code;  c01aa5ac <_pagebuf_free_object+108/134>
00000000 <_EIP>:
Code;  c01aa5ac <_pagebuf_free_object+108/134>   <=====
   0:   0f 0b                     ud2a      <=====
Code;  c01aa5ae <_pagebuf_free_object+10a/134>
   2:   0b 01                     or     (%ecx),%eax
Code;  c01aa5b0 <_pagebuf_free_object+10c/134>
   4:   d6                        (bad)
Code;  c01aa5b1 <_pagebuf_free_object+10d/134>
   5:   cb                        lret
Code;  c01aa5b2 <_pagebuf_free_object+10e/134>
   6:   1f                        pop    %ds
Code;  c01aa5b3 <_pagebuf_free_object+10f/134>
   7:   c0 eb 8e                  shr    $0x8e,%bl
Code;  c01aa5b6 <_pagebuf_free_object+112/134>
   a:   ff 73 54                  pushl  0x54(%ebx)
Code;  c01aa5b9 <_pagebuf_free_object+115/134>
   d:   eb b3                     jmp    ffffffc2 <_EIP+0xffffffc2>
Code;  c01aa5bb <_pagebuf_free_object+117/134>
   f:   90                        nop
Code;  c01aa5bc <_pagebuf_free_object+118/134>
  10:   53                        push   %ebx
Code;  c01aa5bd <_pagebuf_free_object+119/134>
  11:   e8 0e 11 00 00            call   1124 <_EIP+0x1124>

 <0>Kernel panic: Attempted to kill the idle task!
---

Re: [Oops] i386 mm/slab.c (cache_flusharray)

[Linus Torvalds]

On Thu, 4 Dec 2003 pinotj@club-internet.fr wrote:
>
> OK, I tried again the patch on "small kernel" test11 with
> CONFIG_DEBUG_PAGEALLOC only. Here are the first results. I will do more
> tests later because it seems weird. This time I have very different
> behavior for XFS and Ext3.

Ok, interesting indeed.

> I got an oops at boot time, when system try to mount root filesystem (XFS).
>
> But when I tried on a root Ext3, I got no problem at all. I even
> compiled 2 kernel straight without any problems. It seems to be the
> first time a test11 works flawless on my system.

All right. That may or may not mean that the bug is actually in mm/slab.c,
rather than anywhere else. It doesn't explain why _you_ hit it and few
others do, but it's still an interesting fact.

Manfred, any ideas? What's different between 2.6.x and 2.4.x in slab?

But it may also be that the bug is in some slab user - since my slab-
translates-to-page-alloc hack always calls the slab constructor function
on every allocation, and the destructor gets called immediately after the
free, my debug version might hide some usage bugs.

> The XFS oops (I was too lazy to write down all the backtraces, all about
> xfs_* things, I can send if needed):
>
> ---
> kernel BUG at include/linux/mm.h:267!

YEAH! That's "put_page_testzero()", and either the BUG_ON() or the
atomic_dec_and_test() noticing bad things.

So that's the "test for count going negative" bug, and it seems to have
found somebody doing a double free on a page. Which is _exactly_ what we
expected from the XFS problems, and would explain the "struct page"
corruption that people report.

> Call Trace:
>  [<c01aaf90>] pagebuf_free+0x24/0x30
>  [<c019669b>] xlog_find_verify_cycle+0x18b/0x1e0
> Code: 0f 0b 0b 01 d6 cb 1f c0 eb 8e ff 73 54 eb b3 90 53 e8 0e 11

It would be good to get the full backtrace, though.

Nathan - did you see the two debug patches I sent out that caught this?

One adds checks to "atomic_dec_and_test()" to verify that the count never
goes negative. The other basically disables all the slab code, and
replaces them with straight page allocations, and that together with
CONFIG_DEBUG_PAGEALLOC helps find bad behaviour (touching allocations
after they are free'd etc).

		Linus

Re: [Oops] i386 mm/slab.c (cache_flusharray)

[Linus Torvalds]

On Thu, 4 Dec 2003, Linus Torvalds wrote:
> > ---
> > kernel BUG at include/linux/mm.h:267!
>
> YEAH! That's "put_page_testzero()", and either the BUG_ON() or the
> atomic_dec_and_test() noticing bad things.

Oh, damn. Looking closer, it appears that it's actually XFS being a bit
too intimate with slab knowledge: the code does

                        if (pb->pb_pages) {
                                /* release the pages in the address list */
                                if (pb->pb_pages[0] &&
                                    PageSlab(pb->pb_pages[0])) {
                                        /*
                                         * This came from the slab
                                         * allocator free it as such
                                         */
                                        kfree(pb->pb_addr);
                                } else {
                                        _pagebuf_freepages(pb);
                                }

and that code gets really confused by the fact that I'm bypassing the slab
logic (and thus the PageSlab flag never gets set).

So the oops it found was apparently triggered by the debugging changes,
not necessarily by a real bug.

Ugh, that XFS code is _broken_. Instead of keeping track of how it got the
memory, it totally forgets where the memory came from, and then it later
asks "oh, btw, how the hell did I allocate this?".

			Linus

Re: [Oops] i386 mm/slab.c (cache_flusharray)

[Nathan Scott]

On Thu, Dec 04, 2003 at 11:09:29AM -0800, Linus Torvalds wrote:
> 
> 
> On Thu, 4 Dec 2003, Linus Torvalds wrote:
> > > ---
> > > kernel BUG at include/linux/mm.h:267!
> >
> > YEAH! That's "put_page_testzero()", and either the BUG_ON() or the
> > atomic_dec_and_test() noticing bad things.
> 
> Oh, damn. Looking closer, it appears that it's actually XFS being a bit
> too intimate with slab knowledge: the code does
> 
>                         if (pb->pb_pages) {
>                                 /* release the pages in the address list */
>                                 if (pb->pb_pages[0] &&
>                                     PageSlab(pb->pb_pages[0])) {
>                                         /*
>                                          * This came from the slab
>                                          * allocator free it as such
>                                          */
>                                         kfree(pb->pb_addr);
>                                 } else {
>                                         _pagebuf_freepages(pb);
>                                 }
> 
> and that code gets really confused by the fact that I'm bypassing the slab
> logic (and thus the PageSlab flag never gets set).
> 
> So the oops it found was apparently triggered by the debugging changes,
> not necessarily by a real bug.
> 
> Ugh, that XFS code is _broken_. Instead of keeping track of how it got the
> memory, it totally forgets where the memory came from, and then it later
> asks "oh, btw, how the hell did I allocate this?".

Yeah, thats pretty silly stuff - and should be fairly easy to
fix by using a pagebuf flag to differentiate the two.  Will do.

thanks.

-- 
Nathan

Re: [Oops] i386 mm/slab.c (cache_flusharray)

[Christoph Hellwig]

On Fri, Dec 05, 2003 at 08:21:10AM +1100, Nathan Scott wrote:
> Yeah, thats pretty silly stuff - and should be fairly easy to
> fix by using a pagebuf flag to differentiate the two.  Will do.

IMHO a flags is wrong here.  Just maek pb_addr always a pointer and
for the case it's the preallocated array make it point pb_page_array
or something like that.  Then check whether pb_addr is pointing to the
preallocated array.

Re: [Oops] i386 mm/slab.c (cache_flusharray)

[Nathan Scott]

On Fri, Dec 05, 2003 at 07:14:55AM +0000, Christoph Hellwig wrote:
> On Fri, Dec 05, 2003 at 08:21:10AM +1100, Nathan Scott wrote:
> > Yeah, thats pretty silly stuff - and should be fairly easy to
> > fix by using a pagebuf flag to differentiate the two.  Will do.
> 
> IMHO a flags is wrong here.  Just maek pb_addr always a pointer and
> for the case it's the preallocated array make it point pb_page_array
> or something like that.  Then check whether pb_addr is pointing to the
> preallocated array.

You might be mixing up pb_pages and pb_addr there?  pb_addr is
always a pointer.  We need to distinguish whether it was slab
alloc'd or whether it points into page cache pages, so we know
whether to page_cache_release the pages or kfree the pointer
when we're done with the pagebuf.

The pb_page_array works just as you describe, with a prealloc'd
array of page pointers, and pb_pages either points to the array
of to a larger kmalloc'd array as necessary.

cheers.

-- 
Nathan

Re: [Oops] i386 mm/slab.c (cache_flusharray)

[Christoph Hellwig]

On Fri, Dec 05, 2003 at 08:34:25PM +1100, Nathan Scott wrote:
> You might be mixing up pb_pages and pb_addr there?  pb_addr is
> always a pointer.  We need to distinguish whether it was slab
> alloc'd or whether it points into page cache pages, so we know
> whether to page_cache_release the pages or kfree the pointer
> when we're done with the pagebuf.
> 
> The pb_page_array works just as you describe, with a prealloc'd
> array of page pointers, and pb_pages either points to the array
> of to a larger kmalloc'd array as necessary.

Indeed.  I'm not remembering the code as good as I hoped to :)

Sorry for the noise.

Re: [Oops] i386 mm/slab.c (cache_flusharray)

[Nathan Scott]

On Thu, Dec 04, 2003 at 11:09:29AM -0800, Linus Torvalds wrote:
> ...
> So the oops it found was apparently triggered by the debugging changes,
> not necessarily by a real bug.
> 
> Ugh, that XFS code is _broken_. Instead of keeping track of how it got the
> memory, it totally forgets where the memory came from, and then it later
> asks "oh, btw, how the hell did I allocate this?".
> 

This patch removes that code, fixes a small memory leak that was
lurking in there too, and adds the missing-bio_put-on-error case
that Neil found in pagebuf.

Neil, with this & Linus' 2 patches (and CONFIG_SLAB_DEBUG off ;)
I now have what looks like a 100% reproducible test case for the
handle_stripe already-freed-bio panic.  This doesn't tickle the
raid5.c BUG_ON you sent me but its exactly the same spot as last
time (i.e. handle_stripe+0xda6), every time.

# raidstart /dev/md0
# mkfs.xfs -f /dev/md0
# mount /dev/md0
# umount /dev/md0
# mount /dev/md0

On my (quad p3) test machine, this second mount panics every time.

cheers.

-- 
Nathan


--- fs/xfs/pagebuf/page_buf.h.orig	2003-12-05 13:47:12.275589232 +1100
+++ fs/xfs/pagebuf/page_buf.h	2003-12-05 13:43:30.898243704 +1100
@@ -123,12 +123,13 @@
 	_PBF_PRIVATE_BH = (1 << 17), /* do not use public buffer heads	   */
 	_PBF_ALL_PAGES_MAPPED = (1 << 18), /* all pages in range mapped	   */
 	_PBF_ADDR_ALLOCATED = (1 << 19), /* pb_addr space was allocated	   */
-	_PBF_MEM_ALLOCATED = (1 << 20), /* pb_mem+underlying pages alloc'd */
+	_PBF_MEM_ALLOCATED = (1 << 20), /* underlying pages are allocated  */
+	_PBF_MEM_SLAB = (1 << 21), /* underlying pages are slab allocated  */
 
-	PBF_FORCEIO = (1 << 21),
-	PBF_FLUSH = (1 << 22),	/* flush disk write cache		   */
-	PBF_READ_AHEAD = (1 << 23),
-	PBF_RUN_QUEUES = (1 << 24), /* run block device task queue	   */
+	PBF_FORCEIO = (1 << 22),
+	PBF_FLUSH = (1 << 23),	/* flush disk write cache		   */
+	PBF_READ_AHEAD = (1 << 24),
+	PBF_RUN_QUEUES = (1 << 25), /* run block device task queue	   */
 
 } page_buf_flags_t;
 
--- fs/xfs/pagebuf/page_buf.c.orig	2003-12-05 13:47:06.888408208 +1100
+++ fs/xfs/pagebuf/page_buf.c	2003-12-05 13:43:30.888245224 +1100
@@ -343,9 +343,6 @@
 			page_cache_release(page);
 		}
 	}
-
-	if (pb->pb_pages != pb->pb_page_array)
-		kfree(pb->pb_pages);
 }
 
 /*
@@ -384,20 +381,17 @@
 		if (pb->pb_flags & _PBF_MEM_ALLOCATED) {
 			if (pb->pb_pages) {
 				/* release the pages in the address list */
-				if (pb->pb_pages[0] &&
-				    PageSlab(pb->pb_pages[0])) {
-					/*
-					 * This came from the slab
-					 * allocator free it as such
-					 */
+				if ((pb->pb_pages[0]) &&
+				    (pb->pb_flags & _PBF_MEM_SLAB)) {
 					kfree(pb->pb_addr);
 				} else {
 					_pagebuf_freepages(pb);
 				}
-
+				if (pb->pb_pages != pb->pb_page_array)
+					kfree(pb->pb_pages);
 				pb->pb_pages = NULL;
 			}
-			pb->pb_flags &= ~_PBF_MEM_ALLOCATED;
+			pb->pb_flags &= ~(_PBF_MEM_ALLOCATED | _PBF_MEM_SLAB);
 		}
 	}
 
@@ -944,7 +938,7 @@
 		return NULL;
 	}
 	/* otherwise pagebuf_free just ignores it */
-	pb->pb_flags |= _PBF_MEM_ALLOCATED;
+	pb->pb_flags |= (_PBF_MEM_ALLOCATED | _PBF_MEM_SLAB);
 	PB_CLEAR_OWNER(pb);
 	up(&pb->pb_sema);	/* Return unlocked pagebuf */
 
@@ -1412,6 +1406,7 @@
 		if (size)
 			goto next_chunk;
 	} else {
+		bio_put(bio);
 		pagebuf_ioerror(pb, EIO);
 	}
 

Re: [Oops] i386 mm/slab.c (cache_flusharray)

[Linus Torvalds]

On Fri, 5 Dec 2003, Nathan Scott wrote:
>
> This patch removes that code, fixes a small memory leak that was
> lurking in there too, and adds the missing-bio_put-on-error case
> that Neil found in pagebuf.

Ok, so we've got the RAID5 issue explained, does anybody have any idea
about what's wrong with the non-RAID5 cases? We've got at least one report
of page corruption on RAID0, and one on a plain disk. Both of which looked
XFS-related - or at least shared that in their configs.

Jerome - can you test Nathan's patch together with my "avoid the
complicated slab logic"? The slab avoidance thing got ext3 stable for you,
now with Nathan's patch hopefully XFS will be stable too.

Which still doesn't _explain_ anything, but it would be interesting to see
if that removes your problems. It would definitely point to a slab issue,
but the big question is why it's not more common if so. Compiler bug?
Other strange trigger?

			Linus

Re: [Oops] i386 mm/slab.c (cache_flusharray)

[Manfred Spraul]

[-- Attachment #1: Type: text/plain, Size: 1224 bytes --]

Linus Torvalds wrote:

>Manfred, any ideas? What's different between 2.6.x and 2.4.x in slab?
>
>But it may also be that the bug is in some slab user - since my slab-
>translates-to-page-alloc hack always calls the slab constructor function
>on every allocation, and the destructor gets called immediately after the
>free, my debug version might hide some usage bugs.
>  
>
The changes between 2.4 and 2.6 are huge, for both debug and non-debug. 
Slab with debugging enabled now calls the destructors/constructor on 
every alloc. If page debugging is enabled, then all objects larger than 
128 bytes get their own page and are unmapped after kmem_cache_free(). 
The bio structure is smaller than 128 bytes - that probably explains why 
slab didn't catch the oopses that were mentioned in the other thread.
Perhaps something like the attached patch could help to trigger the 
oops: It increase the size of the bio structures, then they are handled 
by slab debugging.
If it oopses, then call ptrinfo() from the trap handler - it prints the 
name of the cache and the caller of the last slab operation. And hexdump 
the object (after ptrinfo mapped it), it contains a backtrace from the 
kmem_cache_free call.

--
    Manfred

[-- Attachment #2: patch-bio --]
[-- Type: text/plain, Size: 690 bytes --]

--- 2.6/fs/bio.c	2003-10-25 20:43:54.000000000 +0200
+++ build-2.6/fs/bio.c	2003-12-04 20:13:52.000000000 +0100
@@ -798,7 +798,7 @@
 
 		size = bp->nr_vecs * sizeof(struct bio_vec);
 
-		bp->slab = kmem_cache_create(bp->name, size, 0,
+		bp->slab = kmem_cache_create(bp->name, max(128,size), 0,
 						SLAB_HWCACHE_ALIGN, NULL, NULL);
 		if (!bp->slab)
 			panic("biovec: can't init slab cache\n");
@@ -815,7 +815,7 @@
 
 static int __init init_bio(void)
 {
-	bio_slab = kmem_cache_create("bio", sizeof(struct bio), 0,
+	bio_slab = kmem_cache_create("bio", max(128U,sizeof(struct bio)), 0,
 					SLAB_HWCACHE_ALIGN, NULL, NULL);
 	if (!bio_slab)
 		panic("bio: can't create slab cache\n");

Re: [Oops] i386 mm/slab.c (cache_flusharray)

[Nathan Scott]

On Thu, Dec 04, 2003 at 10:49:07AM -0800, Linus Torvalds wrote:
> 
> Nathan - did you see the two debug patches I sent out that caught this?
> 
> One adds checks to "atomic_dec_and_test()" to verify that the count never
> goes negative. The other basically disables all the slab code, and
> replaces them with straight page allocations, and that together with
> CONFIG_DEBUG_PAGEALLOC helps find bad behaviour (touching allocations
> after they are free'd etc).
> 

Yep - I'll remove that PageSlab use in there, and start testing
with these too.

thanks.

-- 
Nathan

Re: Re: [Oops] i386 mm/slab.c (cache_flusharray)

[pinotj]

Results about testing on test11 this week-end.
Things didn't go exactly as expected, unfortunately, but there are interesting results.
First, I confirm that use of patch-xfs with patch-slab (and without CONFIG_DEBUG_SLAB) makes my system boot normaly.
I don't mention it after, but I always used the patch-printk to get more verbosity in case of slab oops.

I. Test on "small" kernel (http://cercle-daejeon.homelinux.org/misc/config-small2.txt except in first part of A.2.)

 A. With XFS (and without Ext3)

  1. no patch
  I couldn't reproduce the oops and I have no explanation at this time. I changed some parameters in the config (mostly debug to have same config for all kernels) but even with the config I used before, I didn't succeed to trigger the oops again. I first thought it was CONFIG_DEBUG_SLAB that may be the main problem of this story (and explain last tests) but I got same non-oops with CONFIG_DEBUG_SLAB=y. It doesn't mean that config change corrects the problem, though. Previous test with CONFIG_PREEMPT=y/n
shown that it could be very tricky to get the oops sometimes. More tests are needed here.

  2. patch-xfs & patch-slab
  Oops during first compile. The first test, it was kswapd who complained. I thought it could be a side effect of my config (CONFIG_SWAP=n) and not enough RAM, even if it looked strange. That's why I decided to use config-small2.txt for all the tests (that I remade). In this case, I got a very similar oops, from `ld`:

  ---
  ld: page allocation failure. order:0, mode:0x8d0
  Unable to handle kernel NULL pointer dereference at virtual address 00000074
  c01d4cbd
  *pde = 00000000
  Oops: 0002 [#1]
  CPU:    0
  EIP:    0060:[_xfs_trans_alloc+149/160]    Not tainted
  EIP:    0060:[<c01d4cbd>]    Not tainted
  ---

  Full oops: http://cercle-daejeon.homelinux.org/misc/oops-small2-xfs-patched.txt
  First config used: http://cercle-daejeon.homelinux.org/misc/config-small.txt
   and oops of kswapd http://cercle-daejeon.homelinux.org/misc/oops-small-xfs-patched.txt

  3. patch-bio
  No oops triggered

 B. With Ext3 (and without XFS)

  1. no patch
  same as I.A.1
  2. patch-xfs & patch-slab
  Compilations looked good but I got a lot of errors in my logs:

  ---
  kernel: ld: page allocation failure. order:0, mode:0x50
  last message repeated 31 times
  klogd: page allocation failure. order:0, mode:0x50
  last message repeated 63 times
  kswapd0: page allocation failure. order:0, mode:0x50
  ENOMEM in journal_alloc_journal_head, retrying.
  ion failure. order:0, mode:0x50
  kswapd0: page allocation failure. order:0, mode:0x50
  last message repeated 291 times
  ---

  Full log: http://cercle-daejeon.homelinux.org/misc/error-small2-ext3-patched.txt
  NB: I made a `swapoff -a && mkswap /dev/hda3 && swapon -a` a few days before, so swap should be clean

  3. patch-bio
  As A.3, no oops triggered

II. Tests on "big" kernel, support for XFS and Ext3 (http://cercle-daejeon.homelinux.org/misc/config-big.txt)

 A. no patch
 Same as I.A.1

 B. patch-xfs & patch-slab
 Oops very similar with I.A.2
 Full oops: http://cercle-daejeon.homelinux.org/misc/oops-big-patched.txt

 C. patch-bio
 Here is the interesting thing. Till now, I didn't saw any problem with patch bio, but this time, very easily, I got kind of double triple Lutz, starting by the infamous "BUG at mm/slab.c". There is almost 800 lines.
 Full oops: http://cercle-daejeon.homelinux.org/misc/oops-big-patch-bio-full.txt
 Oops via ksymoops: Full oops: http://cercle-daejeon.homelinux.org/misc/oops-big-patch-bio.txt

III. Conclusion

Well, it's not really easy to find a pattern here. Change in configuration can modify behavior of the oops but we can not correlate these with one setting. I will try to get back the oops by changing config.
Tests in I. confirm that it's not an XFS-only problem but seems to affect page allocation for fs in general.
I hope these oops will be clearer to you. I still have no problem with test9.

Jerome Pinot
(Hope it missed nothing)

Re: [Oops] i386 mm/slab.c (cache_flusharray)

[Nathan Scott]

On Tue, Dec 09, 2003 at 01:57:35AM +0100, pinotj@club-internet.fr wrote:
> Results about testing on test11 this week-end.

thanks.

>   ---
>   ld: page allocation failure. order:0, mode:0x8d0
>   Unable to handle kernel NULL pointer dereference at virtual address 00000074
>   c01d4cbd
>   *pde = 00000000
>   Oops: 0002 [#1]
>   CPU:    0
>   EIP:    0060:[_xfs_trans_alloc+149/160]    Not tainted
>   EIP:    0060:[<c01d4cbd>]    Not tainted
>   ---

Ah, yes, I know what this is (and can reproduce it) and I don't
have a fix as yet.  This is an unrelated problem - basically,
we are doing kmem_cache_alloc with __GFP_NOFAIL set within XFS,
but the allocations are failing and returning NULL (but we are
assuming they will not).

You (and I) are hitting this more frequently now because Linus'
patch bypasses the slab allocator and is more expensive in terms
of memory used.  However, I have heard of one person who hit this
"in the wild" too, so its something we will need to address.

[ Christoph, is this failure expected?  I think you/Steve made
some changes there to use __GFP_NOFAIL and assume it wont fail?
(in 2.4 we do memory allocations differently to better handle
failures, but that code was removed...) ]

> 
>  B. With Ext3 (and without XFS)
> 
>   1. no patch
>   same as I.A.1
>   2. patch-xfs & patch-slab
>   Compilations looked good but I got a lot of errors in my logs:
> 
>   ---
>   kernel: ld: page allocation failure. order:0, mode:0x50
>   last message repeated 31 times
>   klogd: page allocation failure. order:0, mode:0x50
>   last message repeated 63 times
>   kswapd0: page allocation failure. order:0, mode:0x50
>   ENOMEM in journal_alloc_journal_head, retrying.
>   ion failure. order:0, mode:0x50
>   kswapd0: page allocation failure. order:0, mode:0x50
>   last message repeated 291 times

I expect that this is a similar issue to the above, but from
ext2/3's point of view.

> 
> Tests in I. confirm that it's not an XFS-only problem but seems to
> affect page allocation for fs in general.
> I hope these oops will be clearer to you. I still have no problem with test9.

FWIW and IIRC, we didn't push any XFS changes (nothing major
anyway, that I could foresee causing this) on to Linus in
between -test9 and -test11.

cheers.

-- 
Nathan

Re: [Oops] i386 mm/slab.c (cache_flusharray)

[Christoph Hellwig]

On Tue, Dec 09, 2003 at 01:03:22PM +1100, Nathan Scott wrote:
> [ Christoph, is this failure expected?  I think you/Steve made
> some changes there to use __GFP_NOFAIL and assume it wont fail?
> (in 2.4 we do memory allocations differently to better handle
> failures, but that code was removed...) ]

It looks like the slab allocator doesn't like __GFP_NOFAIL, we'll
probably have to revert the XFS memory allocation wrappers to the
2.4 versions.

Re: [Oops] i386 mm/slab.c (cache_flusharray)

[Nathan Scott]

On Tue, Dec 09, 2003 at 08:21:32AM +0100, Christoph Hellwig wrote:
> On Tue, Dec 09, 2003 at 01:03:22PM +1100, Nathan Scott wrote:
> > [ Christoph, is this failure expected?  I think you/Steve made
> > some changes there to use __GFP_NOFAIL and assume it wont fail?
> > (in 2.4 we do memory allocations differently to better handle
> > failures, but that code was removed...) ]
> 
> It looks like the slab allocator doesn't like __GFP_NOFAIL, we'll
> probably have to revert the XFS memory allocation wrappers to the
> 2.4 versions.
> 

OK, thanks - I'll look into it.

cheers.

-- 
Nathan

Re: [Oops] i386 mm/slab.c (cache_flusharray)

[Christoph Hellwig]

On Wed, Dec 10, 2003 at 10:58:32AM +1100, Nathan Scott wrote:
> > > [ Christoph, is this failure expected?  I think you/Steve made
> > > some changes there to use __GFP_NOFAIL and assume it wont fail?
> > > (in 2.4 we do memory allocations differently to better handle
> > > failures, but that code was removed...) ]
> > 
> > It looks like the slab allocator doesn't like __GFP_NOFAIL, we'll
> > probably have to revert the XFS memory allocation wrappers to the
> > 2.4 versions.
> > 
> 
> OK, thanks - I'll look into it.

This starts to look really fishy:

hch@bird:/repo/repo/linux-2.5/fs/jbd$ egrep -r NOFAIL *
journal.c:      new_bh = alloc_buffer_head(GFP_NOFS|__GFP_NOFAIL);
journal.c:      return kmalloc(size, flags | (retry ? __GFP_NOFAIL :
0));

both of these end up in the slab layer, just like XFS - except
that __cache_alloc still may fail.  Looks like whe're better of
fixing mm/slab.c

Re: [Oops] i386 mm/slab.c (cache_flusharray)

[Manfred Spraul]

Christoph Hellwig wrote:

>Looks like whe're better of fixing mm/slab.c
>
Maybe I'm blind, but I don't see how kmem_cache_alloc can return NULL 
with __GFP_NOFAIL:
- kmem_cache_alloc calls __cache_alloc.
- cache_alloc only return 0 if cache_alloc_refill returns 0.
- cache_alloc_refill only return 0 if cache_grow returns 0.
- cache_grow only return 0 if
    * SLAB_NO_GROW is set in flags.
    * get_free_pages(flags) fails.
    * kmem_cache_alloc(flags&SLAB_LEVEL_MASK) fails.
       SLAB_LEVEL_MASK contains __GFP_NOFAIL.

--
    Manfred

Re: [Oops] i386 mm/slab.c (cache_flusharray)

[pinotj]

I tried the "small kernel" test11 with Ext3 support instead of XFS by booting on my slack partition.

I confirm to have the same problem without XFS, oops as usual when I try to compile a kernel:

---
slab: double free detected in cache 'buffer_head' objp c605733c, objnr 9, slabp c6057000, s_mem c6057120, bufctl 12
---

[slab.c patch from Linus]

I tried the patch on the same small config (XFS and CONFIG_DEBUG_PAGEALLOC enabled) and I got oops at the beginning of boot sequence. I spent some times to write this down but I'm not so sure it's a good news. Just say me it's not a hw problem...

---
kernel BUG at mm/slab.c:1655!
invalid operand: 0000 [#1]
CPU:    0
EIP:    0060:[<c013bf35>]    Not tainted
Using defaults from ksymoops -t elf32-i386 -a i386
EFLAGS: 00010046
eax: c0243c00   ebx: 0027f448   ecx: 00000000   edx: cffb5000
esi: cffb6000   edi: cffb5000   ebp: c027bf70   esp: c027bf60
ds: 007b   es: 007b   ss: 0068
Stack: 00000286 00000000 cffb6000 cffb5000 c027bf94 c013c1a5 cffb5000 cffb6044
       cffb6000 cffb5000 cffb6000 cffb6000 c029be80 c027bfb0 c013c413 cffb6000
       00000008 00000004 00000000 000000d0 c027bfec c0283d8a cffb6000 00000090
Call Trace:
 [<c013c1a5>] do_tune_cpucache+0x1cd/0x3fc
 [<c013c413>] enable_cpucache+0x3f/0x64
 [<c0283d8a>] kmem_cache_init+0x1ae/0x280
 [<c0105000>] _stext+0x0/0x20
 [<c027c5af>] start_kernel+0xb3/0x138
Code: 0f 0b 77 06 75 2f 24 c0 8b 15 c4 bf 29 c0 eb af 8d 76 00 57


>>EIP; c013bf35 <kfree+81/b0>   <=====

>>eax; c0243c00 <bpp4tab+5f40/13436>
>>edx; cffb5000 <_end+fd0c990/3fd55990>
>>esi; cffb6000 <_end+fd0d990/3fd55990>
>>edi; cffb5000 <_end+fd0c990/3fd55990>
>>ebp; c027bf70 <init_thread_union+1f70/2000>
>>esp; c027bf60 <init_thread_union+1f60/2000>

Trace; c013c1a5 <do_tune_cpucache+1cd/3fc>
Trace; c013c413 <enable_cpucache+3f/64>
Trace; c0283d8a <kmem_cache_init+1ae/280>
Trace; c0105000 <_stext+0/0>
Trace; c027c5af <start_kernel+b3/138>

Code;  c013bf35 <kfree+81/b0>
00000000 <_EIP>:
Code;  c013bf35 <kfree+81/b0>   <=====
   0:   0f 0b                     ud2a      <=====
Code;  c013bf37 <kfree+83/b0>
   2:   77 06                     ja     a <_EIP+0xa>
Code;  c013bf39 <kfree+85/b0>
   4:   75 2f                     jne    35 <_EIP+0x35>
Code;  c013bf3b <kfree+87/b0>
   6:   24 c0                     and    $0xc0,%al
Code;  c013bf3d <kfree+89/b0>
   8:   8b 15 c4 bf 29 c0         mov    0xc029bfc4,%edx
Code;  c013bf43 <kfree+8f/b0>
   e:   eb af                     jmp    ffffffbf <_EIP+0xffffffbf>
Code;  c013bf45 <kfree+91/b0>
  10:   8d 76 00                  lea    0x0(%esi),%esi
Code;  c013bf48 <kfree+94/b0>
  13:   57                        push   %edi

 <0>Kernel panic: Attempted to kill the idle task!
---

Jerome

Re: [Oops] i386 mm/slab.c (cache_flusharray)

[Linus Torvalds]

On Thu, 4 Dec 2003 pinotj@club-internet.fr wrote:
>
> [slab.c patch from Linus]
>
> I tried the patch on the same small config (XFS and
> CONFIG_DEBUG_PAGEALLOC enabled) and I got oops at the beginning of boot
> sequence. I spent some times to write this down but I'm not so sure it's
> a good news. Just say me it's not a hw problem...

Sorry - it's not a hw problem, and in fact it's a problem with my slab
debugging patch: please don't use CONFIG_DEBUG_SLAB together with my
crappy patch. My patch wants _only_ CONFIG_DEBUG_PAGEALLOC instead of
using the SLAB debugger.

So please turn the slab debugger off, and depend on CONFIG_DEBUG_PAGEALLOC
entirely and try again,

		Linus

Re: Re: [Oops] i386 mm/slab.c (cache_flusharray)

[pinotj]

I triggered the slab oops with a very small kernel -test11 (~700KB):

---
slab: double free detected in cache 'biovec-1', objp c12c2df0, objn 122, slabp c12c2000, s_mem c12c2280, bufctl ffffffff
---

Oops occurs very quickly during compilation.

Here is the .config file (I removed unset options). Full config can be found at http://cercle-daejeon.homelinux.org/misc/config-small.txt , if you want diff with other config for example.

BTW, I don't understand why I can't remove config about game port, mouse and DOS partition. Compilation without DMA fails.

I will reduce it again by removing FB and try unset some options in "linux for embbeded systems" and in debug.

Hope it can help find the problem.

Jerome Pinot

---
CONFIG_X86=y
CONFIG_MMU=y
CONFIG_UID16=y
CONFIG_GENERIC_ISA_DMA=y

CONFIG_CLEAN_COMPILE=y
CONFIG_STANDALONE=y
CONFIG_BROKEN_ON_SMP=y

CONFIG_LOG_BUF_SHIFT=14
CONFIG_KALLSYMS=y
CONFIG_FUTEX=y
CONFIG_EPOLL=y
CONFIG_IOSCHED_NOOP=y
CONFIG_IOSCHED_AS=y
CONFIG_IOSCHED_DEADLINE=y

CONFIG_X86_PC=y
CONFIG_M386=y
CONFIG_X86_L1_CACHE_SHIFT=4
CONFIG_RWSEM_GENERIC_SPINLOCK=y
CONFIG_X86_PPRO_FENCE=y
CONFIG_X86_F00F_BUG=y
CONFIG_NOHIGHMEM=y

CONFIG_PCI=y
CONFIG_PCI_GOANY=y
CONFIG_PCI_BIOS=y
CONFIG_PCI_DIRECT=y

CONFIG_BINFMT_ELF=y

CONFIG_IDE=y
CONFIG_BLK_DEV_IDE=y

CONFIG_BLK_DEV_IDEDISK=y

CONFIG_BLK_DEV_IDEPCI=y
CONFIG_BLK_DEV_GENERIC=y
CONFIG_BLK_DEV_IDEDMA_PCI=y
CONFIG_BLK_DEV_ADMA=y
CONFIG_BLK_DEV_IDEDMA=y

CONFIG_INPUT=y

CONFIG_INPUT_MOUSEDEV=y
CONFIG_INPUT_MOUSEDEV_PSAUX=y
CONFIG_INPUT_MOUSEDEV_SCREEN_X=1024
CONFIG_INPUT_MOUSEDEV_SCREEN_Y=768

CONFIG_SOUND_GAMEPORT=y
CONFIG_SERIO=y
CONFIG_SERIO_I8042=y

CONFIG_INPUT_KEYBOARD=y
CONFIG_KEYBOARD_ATKBD=y

CONFIG_VT=y
CONFIG_VT_CONSOLE=y
CONFIG_HW_CONSOLE=y

CONFIG_FB=y
CONFIG_FB_VESA=y
CONFIG_VIDEO_SELECT=y

CONFIG_VGA_CONSOLE=y
CONFIG_DUMMY_CONSOLE=y
CONFIG_FRAMEBUFFER_CONSOLE=y
CONFIG_PCI_CONSOLE=y
CONFIG_FONTS=y
CONFIG_FONT_8x8=y
CONFIG_FONT_8x16=y

CONFIG_XFS_FS=y

CONFIG_PROC_FS=y
CONFIG_PROC_KCORE=y
CONFIG_RAMFS=y

CONFIG_MSDOS_PARTITION=y

CONFIG_DEBUG_KERNEL=y
CONFIG_DEBUG_STACKOVERFLOW=y
CONFIG_DEBUG_SLAB=y
CONFIG_DEBUG_IOVIRT=y
CONFIG_DEBUG_SPINLOCK=y
CONFIG_DEBUG_PAGEALLOC=y
CONFIG_DEBUG_INFO=y
CONFIG_DEBUG_SPINLOCK_SLEEP=y
CONFIG_FRAME_POINTER=y

CONFIG_X86_BIOS_REBOOT=y
CONFIG_PC=y
---

Re: Re: [Oops] i386 mm/slab.c (cache_flusharray)

[Linus Torvalds]

On Sat, 29 Nov 2003 pinotj@club-internet.fr wrote:
>
> I triggered the slab oops with a very small kernel -test11 (~700KB):

The only thing that looks at _all_ likely to explain the problem is

> CONFIG_XFS_FS=y

since there aren't that many XFS users I know of. It's also now the only
thing that uses buffer heads in your config, so..

I assume it's not an option to try another filesystem on this setup, but
it's entirely possible that the 2.6.x buffer-head removal has impacted XFS
negatively - although I'm a bit surprised at how easily you seem to show
problems, since XFS actually has active maintenance.

Nathan - I don't know if you follow linux-kernel, but Jerome Pinot has
been having bad slab problems for some time now. Do normal XFS users
compile with slab debugging turned on?

		Linus

Re: [Oops] i386 mm/slab.c (cache_flusharray)

[Nathan Scott]

On Mon, Dec 01, 2003 at 04:36:33PM -0800, Linus Torvalds wrote:
> 
> I assume it's not an option to try another filesystem on this setup, but
> it's entirely possible that the 2.6.x buffer-head removal has impacted XFS
> negatively - although I'm a bit surprised at how easily you seem to show
> problems, since XFS actually has active maintenance.
> 
> Nathan - I don't know if you follow linux-kernel, but Jerome Pinot has

Yep, although I try to filter out "noise" and have inadvertently
missed this discussion so far.

> been having bad slab problems for some time now. Do normal XFS users
> compile with slab debugging turned on?

Hmm - I know I do - my nightly QA testing runs with this set.
Let me dig through the archives and catch up a bit on this issue;
I'll get back to you.

thanks.

-- 
Nathan

Re: [Oops] i386 mm/slab.c (cache_flusharray)

[Nathan Scott]

Hi there,

On Tue, Dec 02, 2003 at 12:37:16PM +1100, Nathan Scott wrote:
> On Mon, Dec 01, 2003 at 04:36:33PM -0800, Linus Torvalds wrote:
> > 
> > I assume it's not an option to try another filesystem on this setup, but
> > it's entirely possible that the 2.6.x buffer-head removal has impacted XFS
> > negatively - although I'm a bit surprised at how easily you seem to show
> > problems, since XFS actually has active maintenance.
> > 
> > Nathan - I don't know if you follow linux-kernel, but Jerome Pinot has
> 
> Yep, although I try to filter out "noise" and have inadvertently
> missed this discussion so far.
> 
> > been having bad slab problems for some time now. Do normal XFS users
> > compile with slab debugging turned on?
> 
> Hmm - I know I do - my nightly QA testing runs with this set.
> Let me dig through the archives and catch up a bit on this issue;
> I'll get back to you.

OK, I've run XFS through hours and hours of very heavy stress now,
using a variety of different tests, and have tried different mount
and mkfs options as well.  And with a few kernel compiles thrown in
in the background for good measure.  Either we have quite different
hardware configs, compilers, etc; or this is something else.  This
was done with preempt enabled too (which I usually test without).

I'm not seeing anything to suggest random slab corruption, and I'm
so far unable to trip things up as easily as you're able to Jerome.
Do you have just a very small amount of memory perhaps?  I can try
running while very low on memory, but thats the only other obvious
thing I can think of atm.

cheers.

-- 
Nathan

Re: [Oops] i386 mm/slab.c (cache_flusharray)

[Mike Fedyk]

On Tue, Dec 02, 2003 at 05:44:18PM +1100, Nathan Scott wrote:
> I'm not seeing anything to suggest random slab corruption, and I'm
> so far unable to trip things up as easily as you're able to Jerome.
> Do you have just a very small amount of memory perhaps?  I can try
> running while very low on memory, but thats the only other obvious
> thing I can think of atm.

How about XFS on DM on RAID?

Re: [Oops] i386 mm/slab.c (cache_flusharray)

[Nathan Scott]

On Tue, Dec 02, 2003 at 10:05:40AM -0800, Mike Fedyk wrote:
> On Tue, Dec 02, 2003 at 05:44:18PM +1100, Nathan Scott wrote:
> > I'm not seeing anything to suggest random slab corruption, and I'm
> > so far unable to trip things up as easily as you're able to Jerome.
> > Do you have just a very small amount of memory perhaps?  I can try
> > running while very low on memory, but thats the only other obvious
> > thing I can think of atm.
> 
> How about XFS on DM on RAID?

I didn't see references to those in Jeromes original description,
so haven't been testing those in this context.  But I need to do
more testing on those subsystems anyway, so, will do.

cheers.

-- 
Nathan

Re: Re: [Oops] i386 mm/slab.c (cache_flusharray)

[pinotj]

first, some news

2.6.0-test11 makes same oops during second compilation of kernel. The vanilla kernel with PREEMPT always oops the same way. No matter, it's always reproductible.

2.6.0-test11 + Manfred's patch doesn't hang but I found a slab error in the logs that occured during a compilation. (I didn't find this for -test10, I was lucky ?)

So, there is no more way for my system to run a kernel > -test9 without problem.

>De: Manfred Spraul <manfred@colorfullife.com>
[...]
>There are several sources for the "-1": My initial guess was either a bug in slab, or a bad memory cell (only one bit difference). 
>Thus I sent him a patch that changes multiple bits. Result: It remained a single bit change, i.e it's proven that slab doesn't write BUFCTL_END into the wrong slot.

Thanks for your explanation.
Should I try with L1 and/or L2 cache disable on my computer (I don't know if it's safe) ?
I trust my hardware but it's better to get some facts.

Jerome Pinot

(between LFS/BLFS, kernel compilation and tests compilation, I will surely break kind of record about load average :-)

Re: [Oops] i386 mm/slab.c (cache_flusharray)

[Manfred Spraul]

pinotj@club-internet.fr wrote:

>Thanks for your explanation.
>Should I try with L1 and/or L2 cache disable on my computer (I don't know if it's safe) ?
>I trust my hardware but it's better to get some facts.
>
No, it wouldn't help. Something in the kernel randomly corrupts memory. 
I'm certain that it's not slab. I'm also fairly certain that it's not 
the hardware - IBM guys reproduced corruptions on both ppc64 and i386 
systems (bugzilla 1097 and 1497). The corrupted object is the slab 
structure or the bufctl entries - data near the beginning of a page. But 
I have no idea how to pinpoint it.

--
    Manfred

Re: [Oops] i386 mm/slab.c (cache_flusharray)

[Mike Fedyk]

On Thu, Nov 27, 2003 at 07:42:39PM +0100, pinotj@club-internet.fr wrote:
> first, some news
> 
> 2.6.0-test11 makes same oops during second compilation of kernel. The vanilla kernel with PREEMPT always oops the same way. No matter, it's always reproductible.
> 
> 2.6.0-test11 + Manfred's patch doesn't hang but I found a slab error in the logs that occured during a compilation. (I didn't find this for -test10, I was lucky ?)
> 
> So, there is no more way for my system to run a kernel > -test9 without problem.

Can you try some of the test9-bkN kernels and see where the problem starts?

Re: Re: [Oops] i386 mm/slab.c (cache_flusharray)

[pinotj]

Here are the results for test10 about the oops in slab.c
When I say `compilation` with no explanation, it means compilation of 2.6.0-test10 when using this same kernel, with my .config file (vmlinuz is 2.5M).

1.  2.6.0-test10 vanilla + PREEMPT_CONFIG=y + patch printk
Kernel oops during compilation, as for 2.6.0-test9 : at around 80% of the task.

---
slab: double free detected in cache 'buffer_head', objp cc2dbb58, objnr 42, slabp cc2db000, s_mem cc2db180, bufctl ffffffff.
------------[ cut here ]------------
kernel BUG at mm/slab.c:1956!
invalid operand: 0000 [#1]
CPU:    0
EIP:    0060:[free_block+357/784]    Not tainted
EIP:    0060:[<c015ad55>]    Not tainted
EFLAGS: 00010092
EIP is at free_block+0x165/0x310
eax: 00000080   ebx: 00000000   ecx: c0697854   edx: c05714f8
esi: cc2db000   edi: cc2db018   ebp: cf821c68   esp: cf821c34
ds: 007b   es: 007b   ss: 0068
Process kswapd0 (pid: 8, threadinfo=cf820000 task=cf849960)
Stack: c0504f40 c0505b3d cc2dbb58 0000002a cc2db000 cc2db180 ffffffff 0000002a
       cc2dbb58 0000000d cffdef08 c9e93180 00000010 cf821ca0 c015afda cffed800
       cffdef08 00000010 00000282 c11c89a0 00000000 00000001 cffee730 00000010
 Call Trace:
 [cache_flusharray+218/688] cache_flusharray+0xda/0x2b0
 [<c015afda>] cache_flusharray+0xda/0x2b0
 [kmem_cache_free+429/912] kmem_cache_free+0x1ad/0x390
---

full log can be found here: http://cercle-daejeon.homelinux.org/oops-full.txt
Cleaned oops (ksymoops):    http://cercle-daejeon.homelinux.org/oops.txt
Config of the kernel :      http://cercle-daejeon.homelinux.org/config.txt

NB: I don't get any oops if I compile the kernel with default settings (vmlinuz around 1.5M)

2. 2.6.0-test10 vanilla + PREEMPT_CONFIG=n + patch printk
Argh, oops at the speed of light in the beginning of compilation. Too fast to catch something in the logs.
This invalidates the first idea of bad effect of PREEMPT, it's exactly the contrary in this case.
Second try, compilation is OK, 1 times, 2 times and failed during the third time.
Again, no logs, but this time I wrote down the printk:

---
slab: double free detected in cache 'bio', objp c888fc28, objnr 42, slabp c888f000, s_mem c888f1000, bufctl ffffffff.
---

This case is not easily reproductible.

3. 2.6.0-test10 vanilla + PREEMPT_CONFIG=y + patch printk + patch magic numbers
The patch solves the problem, I can compile 4 times a kernel and do heavy work in parallele (load average around 1.2 during 2 hours) without any problems.

Conclusion: well, this confirms some facts for 2.6.0-test10:

- oops reproductible if PREEMPT_CONFIG=y each time heavy load.
The limit of load, for my system (AMD tbird 1.2GHz, 256Mo) is somewhere between the compilation of a kernel of 1.5M (default settings) and a kernel of 2.5M (custom settings). Always occurs at about 80% of compilation in this second case.

- oops occurs even if PREEMPT_CONFIG=n, but with no really reproductibility. It needs heavy load too, but it's not enough. System hangs really quickly, no logs.

Finally, I just recall the patches of Manfred used here (printk and magic number):

diff -Nru a/mm/slab.c b/mm/slab.c 2003-11-22 09:00:00 +0900
--- a/mm/slab.c         2003-11-22 08:43:02.189656536 +0900
+++ b/mm/slab.c         2003-11-22 08:45:44.158033600 +0900
@@ -1952,8 +1952,7 @@
                check_slabp(cachep, slabp);
 #if DEBUG
                if (slab_bufctl(slabp)[objnr] != BUFCTL_FREE) {
-                       printk(KERN_ERR "slab: double free detected in cache '%s', objp %p.\n",
-                                               cachep->name, objp);
+                       printk(KERN_ERR "slab: double free detected in cache '%s', objp %p, objnr %d, slabp %p, s_mem %p, bufctl %x.\n", cachep->name, objp, objnr, slabp, slabp->s_mem, slab_bufctl(slabp)[objnr]);
                        BUG();
                }
 #endif

diff -Nru a/mm/slab.c b/mm/slab.c 2003-11-22 09:00:00 +0900
--- a/mm/slab.c         2003-11-22 08:43:02.189656536 +0900
+++ b/mm/slab.c         2003-11-22 08:45:44.158033600 +0900
@@ -153,9 +153,9 @@
  * is less than 512 (PAGE_SIZE<<3), but greater than 256.
  */

-#define BUFCTL_END     0xffffFFFF
-#define BUFCTL_FREE    0xffffFFFE
-#define        SLAB_LIMIT      0xffffFFFD
+#define BUFCTL_END     0xfeffFFFF
+#define BUFCTL_FREE    0xf7ffFFFF
+#define SLAB_LIMIT     0xf0ffFFFD
 typedef unsigned int kmem_bufctl_t;

 /* Max number of objs-per-slab for caches which use off-slab slabs.

Regards,

Jerome Pinot

Re: Re: [Oops] i386 mm/slab.c (cache_flusharray)

[Linus Torvalds]

On Tue, 25 Nov 2003 pinotj@club-internet.fr wrote:
>
> 3. 2.6.0-test10 vanilla + PREEMPT_CONFIG=y + patch printk + patch magic numbers
> The patch solves the problem, I can compile 4 times a kernel and do heavy work in parallele (load average around 1.2 during 2 hours) without any problems.

Those magic numbers don't make any sense. In particular, SLAB_LIMIT is
clearly bogus both in the original version and in the "magic number
patch". The only place that uses SLAB_LIMIT is the code that decides how
many entries fit in one slab, and quite frankly, it makes no _sense_ to
have a SLAB_LIMIT that is big enough to be unsigned.

"SLAB_LIMIT" should be something in the few hundreds, maybe.

Manfred?  What is the logic behind those nonsensical numbers?

		Linus

Re: [Oops] i386 mm/slab.c (cache_flusharray)

[Manfred Spraul]

Linus Torvalds wrote:

>On Tue, 25 Nov 2003 pinotj@club-internet.fr wrote:
>  
>
>>3. 2.6.0-test10 vanilla + PREEMPT_CONFIG=y + patch printk + patch magic numbers
>>The patch solves the problem, I can compile 4 times a kernel and do heavy work in parallele (load average around 1.2 during 2 hours) without any problems.
>>    
>>
>
>Those magic numbers don't make any sense. In particular, SLAB_LIMIT is
>clearly bogus both in the original version and in the "magic number
>patch". The only place that uses SLAB_LIMIT is the code that decides how
>many entries fit in one slab, and quite frankly, it makes no _sense_ to
>have a SLAB_LIMIT that is big enough to be unsigned.
>
Object numbers  (kmem_bufctl_t) are unsigned, but some values have a 
special meaning:
"-1" is the magic value for end-of-list.
"-2" is the magic value for object in use.
All other values are valid object numbers. Right now object numbers are 
unsigned int, but initially I considered unsigned char or unsigned 
short. And then an explicit SLAB_LIMIT is necessary - with unsigned 
char, the limit would be 253 objects per slab, which could be reached if 
someone creates objects smaller than 16 bytes.

In Jerome's case, the debug checks noticed that the object-in-use 
sentinel was not in the bufctl entry during free, instead there was a 
"-1". There are several sources for the "-1": My initial guess was 
either a bug in slab, or a bad memory cell (only one bit difference). 
Thus I sent him a patch that changes multiple bits. Result: It remained 
a single bit change, i.e it's proven that slab doesn't write BUFCTL_END 
into the wrong slot.

--
    Manfred

Re: Re: Re: [Oops] i386 mm/slab.c (cache_flusharray)

[pinotj]

>> Summary: Oops reproductible when heavy load, bug in mm/slab.c
>
>Do you have CONFIG_PREEMPT on, and if so, does it go away if you compile
>without PREEMPT? We have at least one other bug that seems to be dependent
>on CONFIG_PREEMPT.
>
>		Linus

Yes, I have CONFIG_PREEMPT=y in my .config
I will try without next time.

Here is the result about what asked Manfred.
In my logs, I found:
---
Nov 21 05:46:48 gegenux kernel: slab: double free detected in cache 'buffer_head', objp c4c8e3d8, objnr 10, slabp c4c8e000,
Nov 21 05:46:49 gegenux kernel: slab: double free detected in cache 'buffer_head', objp c9a582ac, objnr 5, slabp c9a58000,
Nov 21 07:01:50 gegenux kernel: slab: double free detected in cache 'pte_chain', objp c18a6600, objnr 10, slabp c18a6000,
---

So the objnr can be different but it's in the same oops (look the time). The slabp always finish by 0xXXXXX000.

I compiled again with the patch you gave.
First compilation (kernel) no freeze, simple error of `as` but I got this in the logs:
---
slab error in cache_free_debugcheck(): cache `bio': double free, or memory outside object was overwritten
Call Trace:
 [kmem_cache_free+687/912] kmem_cache_free+0x2af/0x390
 [<c015b8cf>] kmem_cache_free+0x2af/0x390
 [mempool_free+224/544] mempool_free+0xe0/0x220
 [<c01535d0>] mempool_free+0xe0/0x220
 [mempool_free+224/544] mempool_free+0xe0/0x220
 [<c01535d0>] mempool_free+0xe0/0x220
 [kernel_map_pages+40/144] kernel_map_pages+0x28/0x90
 [<c0122bd8>] kernel_map_pages+0x28/0x90
 [bio_destructor+57/96] bio_destructor+0x39/0x60
 [<c01816f9>] bio_destructor+0x39/0x60
 [bio_put+41/64] bio_put+0x29/0x40
 [<c01818e9>] bio_put+0x29/0x40
 [end_bio_bh_io_sync+56/64] end_bio_bh_io_sync+0x38/0x40
 [<c0180e18>] end_bio_bh_io_sync+0x38/0x40
 [bio_endio+77/128] bio_endio+0x4d/0x80
--cut--
[background_writeout+0/176] background_writeout+0x0/0xb0
 [<c01568a0>] background_writeout+0x0/0xb0
 [kernel_thread_helper+5/12] kernel_thread_helper+0x5/0xc
 [<c01092a9>] kernel_thread_helper+0x5/0xc

c6fd7870: redzone 1: 0x170fc2a5, redzone 2: 0x160fc2a5.
---
System looks OK, I tried a second compilation just after and this time I got an oops:
---
slab: double free detected in cache 'buffer_head', objp cc3f9798, objnr 26, slabp cc3f9000, s_mem cc3f9180 bufctl f7ffffff.

mm/slab.c:1777: spin_lock(mm/slab.c:cffed844) already locked by mm/slab.c/1994
---cut---
No modules in ksyms, skipping objects
No ksyms, skipping lsmod
kernel BUG at mm/slab.c:1956!
invalid operand: 0000 [#1]
CPU:    0
EIP:    0060:[free_block+363/784]    Not tainted
EIP:    0060:[<c015ad6b>]    Not tainted
Using defaults from ksymoops -t elf32-i386 -a i386
EFLAGS: 00010092
eax: 0000007f   ebx: 0000000a   ecx: c06973dc   edx: c05712f8
esi: cc3f9000   edi: cc3f9018   ebp: cf821c68   esp: cf821c34
ds: 007b   es: 007b   ss: 0068
Stack: c0504d00 c05058fd cc3f9798 0000001a cc3f9000 cc3f9180 f7ffffff 0000001a
       cc3f9798 0000000b cffdef08 c3bb8180 00000010 cf821ca0 c015afea cffed800
       cffdef08 00000010 cf821ce8 c010cabc 80010c00 cd37e000 cffee730 00000010
Call Trace:
 [<c015afea>] cache_flusharray+0xda/0x2b0
 [<c010cabc>] common_interrupt+0x18/0x20
 [<c015b7cd>] kmem_cache_free+0x1ad/0x39
---cut---

You can find the complete log here:
http://cercle-daejeon.homelinux.org/oops-log.txt

Hope this will help...

Jerome

Re: [Oops] i386 mm/slab.c (cache_flusharray)

[Manfred Spraul]

pinotj@club-internet.fr wrote:

>c6fd7870: redzone 1: 0x170fc2a5, redzone 2: 0x160fc2a5.
>
Single bit error: redzone 2 must be 0x170fc2a5.

>---
>System looks OK, I tried a second compilation just after and this time I got an oops:
>---
>slab: double free detected in cache 'buffer_head', objp cc3f9798, objnr 26, slabp cc3f9000, s_mem cc3f9180 bufctl f7ffffff.
>  
>
Good.

+#define BUFCTL_END	0xfeffFFFF
+#define BUFCTL_FREE	0xf7ffFFFE
+#define	SLAB_LIMIT	0xf0ffFFFD

f7ffffff is not a valid value, slab never writes that into a bufctl. 
Someone did a ++ or "|= 1", or a hw bug.
I think the Athlon cpus have ECC for the L2 cache - could you check in 
the bios that ECC checking is enabled?

--
    Manfred

Re: Re: [Oops] i386 mm/slab.c (cache_flusharray)

[pinotj]

----Message d'origine----
>Date: Wed, 19 Nov 2003 18:09:43 -0800
>De: Andrew Morton <akpm@osdl.org>
>A: pinotj@club-internet.fr
>Copie à: linux-kernel@vger.kernel.org
>Sujet: Re: [Oops]  i386 mm/slab.c (cache_flusharray)
[...]
>Well it's interesting that it is repeatable.
>
>First thing to do is to eliminate hardware failures:
>
>1: Is the oops always the same, or does the machine crash in other ways,
>   with different backtraces?
>
>2: Try running memtest86 on that machine for 12 hours or more.
>
>3: Can the problem be reproduced on other machines?
>
>4: try a different compiler version.

First, some results about some tests (not finish yet)

0. Increase verbosity of the printk (thanks to Manfred):
(compilation of kernel)
slab: double free detected in cache 'buffer_head', objp c4c8e3d8, objnr 10,
slabp c4c8e000, s_mem c4c8e180, bufctl ffffffff.
(compilation of firebird)
slab: double free detected in cache 'pte_chain', objp c18a6600, objnr 10,
slabp c18a6000, s_mem c18a6100, bufctl ffffffff.

1. Reproductibility : yes, it oops each time I try to compile a kernel, for example, at around 75% of the task.
Always oops if I try to compile during a quite long time
One funny thing, though. I got one oops without freeze. After error of gcc, I went back to the shell but when I called ksymoops 2 commands later, everything freezed.
About the backtrace, well I'm not sure. Are you talking about what follow the `call trace` etc ? The problem is the system don't have always the time to flush everything to the log, I often got only the printk. But I always got the cache_flusharray thing in first position.

2. Test mem (not yet, I need some time). But as I said, I never had oops before, with 2.6.0-test from 4 to 9. I compiled all my LFS with 2.6.0-test9 vanilla without problem.
3. Change compiler: confirm same problem with gcc 2.95.3, 3.2.3, 3.3.1
x. ACPI: same oops with `acpi=off pci=noacpi` at boot

Summary: Oops reproductible when heavy load, bug in mm/slab.c
Don't have this problem with 2.6.0-test9 and prior
Problem appears in the last patches, before 15 november
(cset-20031115_0206) so I looked for something wrong.

I tried to remove some of the last patches (mm/ioremap.c, 
mm/filemap.c, mm/memory.c) but still got oops.
Should be another patch. Which one else can I remove to test ?

Regards,

Jerome

Re: [Oops] i386 mm/slab.c (cache_flusharray)

[Manfred Spraul]

[-- Attachment #1: Type: text/plain, Size: 1016 bytes --]

pinotj@club-internet.fr wrote:

>0. Increase verbosity of the printk (thanks to Manfred):
>(compilation of kernel)
>slab: double free detected in cache 'buffer_head', objp c4c8e3d8, objnr 10,
>slabp c4c8e000, s_mem c4c8e180, bufctl ffffffff.
>(compilation of firebird)
>slab: double free detected in cache 'pte_chain', objp c18a6600, objnr 10,
>slabp c18a6000, s_mem c18a6100, bufctl ffffffff.
>  
>
The correct value for the bufctl would be 0xfffffffe - a single bit is 
wrong, but OTHO 0xffffffff is also a valid value.
Two different caches are affected.
The addresses of the corrupted variable differ, the offset into the page 
is identical. I think that rules out bad memory. That leaves a bug in 
slab.c, a bad bit in the L1/L2 cache, or random pointer scribbling.
Jerome, could you try the attached patch? It changes the magic constants 
that are used by slab.c. And please pay attention to the objnr: Is it 
always objnr 10, slabp xxxxx000, or do you see other values as well?

--
    Manfred

--
    Manfred

[-- Attachment #2: x --]
[-- Type: text/plain, Size: 496 bytes --]

--- 2.6/mm/slab.c	2003-11-18 18:18:20.000000000 +0100
+++ build-2.6/mm/slab.c	2003-11-21 19:50:02.000000000 +0100
@@ -153,9 +153,9 @@
  * is less than 512 (PAGE_SIZE<<3), but greater than 256.
  */
 
-#define BUFCTL_END	0xffffFFFF
-#define BUFCTL_FREE	0xffffFFFE
-#define	SLAB_LIMIT	0xffffFFFD
+#define BUFCTL_END	0xfeffFFFF
+#define BUFCTL_FREE	0xf7ffFFFE
+#define	SLAB_LIMIT	0xf0ffFFFD
 typedef unsigned int kmem_bufctl_t;
 
 /* Max number of objs-per-slab for caches which use off-slab slabs.

Re: Re: [Oops] i386 mm/slab.c (cache_flusharray)

[pinotj]

>Date: Wed, 19 Nov 2003 17:07:53 -0800
>De: Andrew Morton <akpm@osdl.org>
>A: pinotj@club-internet.fr
>Copie à: linux-kernel@vger.kernel.org
>Sujet: Re: [Oops]  i386 mm/slab.c (cache_flusharray)
>
>pinotj@club-internet.fr wrote:
>>
>> kernel BUG at mm/slab.c:1957!
[...]
>
>urgh, there are several reports of this and it's always the buffer_head
>slab.  The code in there is trivial so perhaps it's just that the large
>number of buffer_heads makes them a fat target.
>
>You should have also seen the message "slab: double free detected in cache
>'buffer_head', objp 0xNNNNNNNN".

Yeah, you right, I forgot to mention it, it was just above the Oops in the logs:
---
slab: double free detected in cache 'buffer_head', objp cd09af18.
------------[ cut here ]------------
kernel BUG at mm/slab.c:1957!
---

>Don't know, sorry.

Is there any thing I can do to help figure out where does the problem comes from ? Anyway, thanks for your answer.

Regards,

Jerome Pinot

Re: [Oops] i386 mm/slab.c (cache_flusharray)

[Andrew Morton]

pinotj@club-internet.fr wrote:
>
> kernel BUG at mm/slab.c:1957!
>  ---
> 
>  >Don't know, sorry.
> 
>  Is there any thing I can do to help figure out where does the problem comes from ? 

Well it's interesting that it is repeatable.

First thing to do is to eliminate hardware failures:

1: Is the oops always the same, or does the machine crash in other ways,
   with different backtraces?

2: Try running memtest86 on that machine for 12 hours or more.

3: Can the problem be reproduced on other machines?

4: try a different compiler version.

Thanks.

[Oops] i386 mm/slab.c (cache_flusharray)

[pinotj]

Here is an Oops from the kernel 2.6.0-test9 + cset-20031115_0206.txt.gz (means all current patches till 03/11/14 [PATCH] PPC32: cancel syscall restart on signal delivery ).

Seems to come from the cache_flusharray function of mm/slab.c
It occurs in general during heavy load (compilation of kernel or xfree86...), is quite reproductible but not automatic.
Same Oops on different distro/gcc (slack 9.1 et lfs 5.0).
Arch is i386 (AMD athlon-tbird 1.2GHz)

I never had this problem with 2.6.0-test8 and before.

This oops is very annoying, I got it around 5 times a day. I'm sorry but I don't have the skill to patch this. If someone can help :-) Please CC me in answer, I'm not on the lkml.

Regards,

Jerome Pinot

---
ksymoops 2.4.9 on i686 2.6.0-test9.  Options used
     -V (default)
     -K (specified)
     -l /proc/modules (default)
     -o /mnt/lfs/lib/modules/2.6.0-test9 (specified)
     -m /mnt/lfs/boot/System.map (specified)

No modules in ksyms, skipping objects
No ksyms, skipping lsmod
kernel BUG at mm/slab.c:1957!
invalid operand: 0000 [#1]
CPU:    0
EIP:    0060:[free_block+336/752]    Not tainted
EIP:    0060:[<c015ad40>]    Not tainted
Using defaults from ksymoops -t elf32-i386 -a i386
EFLAGS: 00010096
eax: 00000045   ebx: 00000006   ecx: c0693854   edx: c056e4f8
esi: cd09a000   edi: cd09a018   ebp: cf821c68   esp: cf821c3c
ds: 007b   es: 007b   ss: 0068
Stack: c0502240 c0502e1d cd09af18 c0652a00 00000001 0000003a cd09af18 0000000f
       cffdef08 c4bcd180 00000010 cf821ca0 c015afba cffed800 cffdef08 00000010
       00000282 c1161ca0 00000000 00000001 cffee730 00000010 00010c00 c4bcd180
Call Trace:
 [<c015afba>] cache_flusharray+0xda/0x2b0
 [<c015b7ad>] kmem_cache_free+0x1ad/0x3a0
 [<c018158c>] free_buffer_head+0x2c/0x60
 [<c018158c>] free_buffer_head+0x2c/0x60
 [<c0181353>] try_to_free_buffers+0x143/0x220
 [<c0294bab>] linvfs_release_page+0x7b/0x80
 [<c017f243>] try_to_release_page+0x53/0x60
 [<c015fec7>] shrink_list+0x8b7/0xac0
 [<c010cadc>] common_interrupt+0x18/0x20
 [<c015e489>] __pagevec_release+0x29/0x40
 [<c01602d5>] shrink_cache+0x205/0x610
 [<c0161228>] shrink_zone+0x78/0xa0
 [<c01615fa>] balance_pgdat+0x17a/0x200
 [<c0161759>] kswapd+0xd9/0xf0
 [<c01274f0>] autoremove_wake_function+0x0/0x50
 [<c010c046>] ret_from_fork+0x6/0x14
 [<c01274f0>] autoremove_wake_function+0x0/0x50
 [<c0161680>] kswapd+0x0/0xf0
 [<c01092a9>] kernel_thread_helper+0x5/0xc
Code: 0f 0b a5 07 82 15 50 c0 8b 46 14 8b 4d e8 31 db 89 04 8f 89


>>EIP; c015ad40 <free_block+150/2f0>   <=====

>>ecx; c0693854 <per_cpu__runqueues+4b4/940>
>>edx; c056e4f8 <log_wait+18/20>
>>esi; cd09a000 <__crc_unregister_sound_dsp+164f0/974e1>
>>edi; cd09a018 <__crc_unregister_sound_dsp+16508/974e1>
>>ebp; cf821c68 <__crc_scsi_register_driver+6c7b9/1f2ec0>
>>esp; cf821c3c <__crc_scsi_register_driver+6c78d/1f2ec0>

Trace; c015afba <cache_flusharray+da/2b0>
Trace; c015b7ad <kmem_cache_free+1ad/3a0>
Trace; c018158c <free_buffer_head+2c/60>
Trace; c018158c <free_buffer_head+2c/60>
Trace; c0181353 <try_to_free_buffers+143/220>
Trace; c0294bab <linvfs_release_page+7b/80>
Trace; c017f243 <try_to_release_page+53/60>
Trace; c015fec7 <shrink_list+8b7/ac0>
Trace; c010cadc <common_interrupt+18/20>
Trace; c015e489 <__pagevec_release+29/40>
Trace; c01602d5 <shrink_cache+205/610>
Trace; c0161228 <shrink_zone+78/a0>
Trace; c01615fa <balance_pgdat+17a/200>
Trace; c0161759 <kswapd+d9/f0>
Trace; c01274f0 <autoremove_wake_function+0/50>
Trace; c010c046 <ret_from_fork+6/14>
Trace; c01274f0 <autoremove_wake_function+0/50>
Trace; c0161680 <kswapd+0/f0>
Trace; c01092a9 <kernel_thread_helper+5/c>

Code;  c015ad40 <free_block+150/2f0>
00000000 <_EIP>:
Code;  c015ad40 <free_block+150/2f0>   <=====
   0:   0f 0b                     ud2a      <=====
Code;  c015ad42 <free_block+152/2f0>
   2:   a5                        movsl  %ds:(%esi),%es:(%edi)
Code;  c015ad43 <free_block+153/2f0>
   3:   07                        pop    %es
Code;  c015ad44 <free_block+154/2f0>
   4:   82                        (bad)
Code;  c015ad45 <free_block+155/2f0>
   5:   15 50 c0 8b 46            adc    $0x468bc050,%eax
Code;  c015ad4a <free_block+15a/2f0>
   a:   14 8b                     adc    $0x8b,%al
Code;  c015ad4c <free_block+15c/2f0>
   c:   4d                        dec    %ebp
Code;  c015ad4d <free_block+15d/2f0>
   d:   e8 31 db 89 04            call   489db43 <_EIP+0x489db43>
Code;  c015ad52 <free_block+162/2f0>
  12:   8f 89 00 00 00 00         popl   0x0(%ecx)

Unable to handle kernel paging request at virtual address 00100104
c015ac3c
*pde = 00000000
Oops: 0002 [#2]
CPU:    0
EIP:    0060:[free_block+76/752]    Not tainted
EIP:    0060:[<c015ac3c>]    Not tainted
EFLAGS: 00010017
eax: 00100100   ebx: 00000000   ecx: cd09a810   edx: 00200200
esi: cd09a000   edi: c7430018   ebp: c558bafc   esp: c558bad0
ds: 007b   es: 007b   ss: 0068
Stack: c558badc 00000086 80010c00 ccf77000 cffdc080 0000001a cd09a810 0000000b 
       cffdef08 cd09a180 00000010 c558bb34 c015afba cffed800 cffdef08 00000010
       00000286 c11d0c68 00000000 00000001 cffee730 00000010 00010c00 cd09a180 
Call Trace:
 [<c015afba>] cache_flusharray+0xda/0x2b0
 [<c015b7ad>] kmem_cache_free+0x1ad/0x3a0
 [<c018158c>] free_buffer_head+0x2c/0x60
 [<c018158c>] free_buffer_head+0x2c/0x60
 [<c0181353>] try_to_free_buffers+0x143/0x220
 [<c0294bab>] linvfs_release_page+0x7b/0x80
 [<c017f243>] try_to_release_page+0x53/0x60
 [<c015fec7>] shrink_list+0x8b7/0xac0
 [<c0123eef>] schedule+0x36f/0x8a0
 [<c015e489>] __pagevec_release+0x29/0x40
 [<c01602d5>] shrink_cache+0x205/0x610
 [<c0127515>] autoremove_wake_function+0x25/0x50
 [<c0161228>] shrink_zone+0x78/0xa0
 [<c01612ee>] shrink_caches+0x9e/0xc0
 [<c01613b7>] try_to_free_pages+0xa7/0x170
 [<c015541a>] __alloc_pages+0x1fa/0x380
 [<c0165c17>] do_anonymous_page+0xc7/0x520
 [<c017a899>] do_sync_write+0x89/0xc0
 [<c01668f2>] handle_mm_fault+0x132/0x310
 [<c0121940>] do_page_fault+0x350/0x56a
 [<c01698de>] do_brk+0x14e/0x230
 [<c016790e>] sys_brk+0xee/0x120
 [<c01215f0>] do_page_fault+0x0/0x56a
 [<c010cb79>] error_code+0x2d/0x38
Code: 89 50 04 89 02 c7 46 04 00 02 20 00 c7 06 00 01 10 00 89 c8


>>EIP; c015ac3c <free_block+4c/2f0>   <=====

>>eax; 00100100 <__crc_prepare_to_wait_exclusive+ce3e5/1424fd>
>>ecx; cd09a810 <__crc_unregister_sound_dsp+16d00/974e1>
>>edx; 00200200 <__crc___user_walk+3d8ad/1cb584>
>>esi; cd09a000 <__crc_unregister_sound_dsp+164f0/974e1>
>>edi; c7430018 <__crc_inet_getsockopt+60fc6/20c3c2>
>>ebp; c558bafc <__crc_fat_write_inode+336fc/668510>
>>esp; c558bad0 <__crc_fat_write_inode+336d0/668510>

Trace; c015afba <cache_flusharray+da/2b0>
Trace; c015b7ad <kmem_cache_free+1ad/3a0>
Trace; c018158c <free_buffer_head+2c/60>
Trace; c018158c <free_buffer_head+2c/60>
Trace; c0181353 <try_to_free_buffers+143/220>
Trace; c0294bab <linvfs_release_page+7b/80>
Trace; c017f243 <try_to_release_page+53/60>
Trace; c015fec7 <shrink_list+8b7/ac0>
Trace; c0123eef <schedule+36f/8a0>
Trace; c015e489 <__pagevec_release+29/40>
Trace; c01602d5 <shrink_cache+205/610>
Trace; c0127515 <autoremove_wake_function+25/50>
Trace; c0161228 <shrink_zone+78/a0>
Trace; c01612ee <shrink_caches+9e/c0>
Trace; c01613b7 <try_to_free_pages+a7/170>
Trace; c015541a <__alloc_pages+1fa/380>
Trace; c0165c17 <do_anonymous_page+c7/520>
Trace; c017a899 <do_sync_write+89/c0>
Trace; c01668f2 <handle_mm_fault+132/310>
Trace; c0121940 <do_page_fault+350/56a>
Trace; c01698de <do_brk+14e/230>
Trace; c016790e <sys_brk+ee/120>
Trace; c01215f0 <do_page_fault+0/56a>
Trace; c010cb79 <error_code+2d/38>

Code;  c015ac3c <free_block+4c/2f0>
00000000 <_EIP>:
Code;  c015ac3c <free_block+4c/2f0>   <=====
   0:   89 50 04                  mov    %edx,0x4(%eax)   <=====
Code;  c015ac3f <free_block+4f/2f0>
   3:   89 02                     mov    %eax,(%edx)
Code;  c015ac41 <free_block+51/2f0>
   5:   c7 46 04 00 02 20 00      movl   $0x200200,0x4(%esi)
Code;  c015ac48 <free_block+58/2f0>
   c:   c7 06 00 01 10 00         movl   $0x100100,(%esi)
Code;  c015ac4e <free_block+5e/2f0>
  12:   89 c8                     mov    %ecx,%eax

Call Trace:
 [<c0124411>] schedule+0x891/0x8a0
 [<c0163ae1>] unmap_page_range+0x41/0x70
 [<c0163d24>] unmap_vmas+0x214/0x330
 [<c0169adb>] exit_mmap+0xcb/0x2b0
 [<c0127935>] mmput+0xb5/0x150
 [<c012e062>] do_exit+0x1b2/0x960
 [<c01215f0>] do_page_fault+0x0/0x56a
 [<c01215f0>] do_page_fault+0x0/0x56a
 [<c010d367>] die+0x227/0x230
 [<c01217d6>] do_page_fault+0x1e6/0x56a
 [<c0122d1e>] recalc_task_prio+0x8e/0x1b0
 [<c0122f9a>] try_to_wake_up+0x15a/0x290
 [<c01215f0>] do_page_fault+0x0/0x56a
 [<c010cb79>] error_code+0x2d/0x38
 [<c015ac3c>] free_block+0x4c/0x2f0
 [<c015afba>] cache_flusharray+0xda/0x2b0
 [<c015b7ad>] kmem_cache_free+0x1ad/0x3a0
 [<c018158c>] free_buffer_head+0x2c/0x60
 [<c018158c>] free_buffer_head+0x2c/0x60
 [<c0181353>] try_to_free_buffers+0x143/0x220
 [<c0294bab>] linvfs_release_page+0x7b/0x80
 [<c017f243>] try_to_release_page+0x53/0x60
 [<c015fec7>] shrink_list+0x8b7/0xac0
 [<c0123eef>] schedule+0x36f/0x8a0
 [<c015e489>] __pagevec_release+0x29/0x40
 [<c01602d5>] shrink_cache+0x205/0x610
 [<c0127515>] autoremove_wake_function+0x25/0x50
 [<c0161228>] shrink_zone+0x78/0xa0
 [<c01612ee>] shrink_caches+0x9e/0xc0
 [<c01613b7>] try_to_free_pages+0xa7/0x170
 [<c015541a>] __alloc_pages+0x1fa/0x380
 [<c0165c17>] do_anonymous_page+0xc7/0x520
 [<c017a899>] do_sync_write+0x89/0xc0
 [<c01668f2>] handle_mm_fault+0x132/0x310
 [<c0121940>] do_page_fault+0x350/0x56a
 [<c01698de>] do_brk+0x14e/0x230
 [<c016790e>] sys_brk+0xee/0x120
 [<c01215f0>] do_page_fault+0x0/0x56a
 [<c010cb79>] error_code+0x2d/0x38
Call Trace:
 [<c0124411>] schedule+0x891/0x8a0
 [<c0163ae1>] unmap_page_range+0x41/0x70
 [<c0163d24>] unmap_vmas+0x214/0x330
 [<c0169adb>] exit_mmap+0xcb/0x2b0
 [<c0127935>] mmput+0xb5/0x150
 [<c012e062>] do_exit+0x1b2/0x960
 [<c01215f0>] do_page_fault+0x0/0x56a
 [<c01215f0>] do_page_fault+0x0/0x56a
 [<c010d367>] die+0x227/0x230
 [<c01217d6>] do_page_fault+0x1e6/0x56a
 [<c0122d1e>] recalc_task_prio+0x8e/0x1b0
 [<c0122f9a>] try_to_wake_up+0x15a/0x290
 [<c01215f0>] do_page_fault+0x0/0x56a
 [<c010cb79>] error_code+0x2d/0x38
 [<c015ac3c>] free_block+0x4c/0x2f0
 [<c015afba>] cache_flusharray+0xda/0x2b0
 [<c015b7ad>] kmem_cache_free+0x1ad/0x3a0
 [<c018158c>] free_buffer_head+0x2c/0x60
 [<c018158c>] free_buffer_head+0x2c/0x60
 [<c0181353>] try_to_free_buffers+0x143/0x220
 [<c0294bab>] linvfs_release_page+0x7b/0x80
 [<c017f243>] try_to_release_page+0x53/0x60
 [<c015fec7>] shrink_list+0x8b7/0xac0
 [<c0123eef>] schedule+0x36f/0x8a0
 [<c015e489>] __pagevec_release+0x29/0x40
 [<c01602d5>] shrink_cache+0x205/0x610
 [<c0127515>] autoremove_wake_function+0x25/0x50
 [<c0161228>] shrink_zone+0x78/0xa0
 [<c01612ee>] shrink_caches+0x9e/0xc0
 [<c01613b7>] try_to_free_pages+0xa7/0x170
 [<c015541a>] __alloc_pages+0x1fa/0x380
 [<c0165c17>] do_anonymous_page+0xc7/0x520
 [<c017a899>] do_sync_write+0x89/0xc0
 [<c01668f2>] handle_mm_fault+0x132/0x310
 [<c0121940>] do_page_fault+0x350/0x56a
 [<c01698de>] do_brk+0x14e/0x230
 [<c016790e>] sys_brk+0xee/0x120
 [<c01215f0>] do_page_fault+0x0/0x56a
 [<c010cb79>] error_code+0x2d/0x38
Call Trace:
 [<c0124411>] schedule+0x891/0x8a0
 [<c0163ae1>] unmap_page_range+0x41/0x70
 [<c0163d24>] unmap_vmas+0x214/0x330
 [<c0169adb>] exit_mmap+0xcb/0x2b0
 [<c0127935>] mmput+0xb5/0x150
 [<c012e062>] do_exit+0x1b2/0x960
 [<c01215f0>] do_page_fault+0x0/0x56a
 [<c01215f0>] do_page_fault+0x0/0x56a
 [<c010d367>] die+0x227/0x230
 [<c01217d6>] do_page_fault+0x1e6/0x56a
 [<c0122d1e>] recalc_task_prio+0x8e/0x1b0
 [<c0122f9a>] try_to_wake_up+0x15a/0x290
 [<c01215f0>] do_page_fault+0x0/0x56a
 [<c010cb79>] error_code+0x2d/0x38
 [<c015ac3c>] free_block+0x4c/0x2f0
 [<c015afba>] cache_flusharray+0xda/0x2b0
 [<c015b7ad>] kmem_cache_free+0x1ad/0x3a0
 [<c018158c>] free_buffer_head+0x2c/0x60
 [<c018158c>] free_buffer_head+0x2c/0x60
 [<c0181353>] try_to_free_buffers+0x143/0x220
 [<c0294bab>] linvfs_release_page+0x7b/0x80
 [<c017f243>] try_to_release_page+0x53/0x60
 [<c015fec7>] shrink_list+0x8b7/0xac0
 [<c0123eef>] schedule+0x36f/0x8a0
 [<c015e489>] __pagevec_release+0x29/0x40
 [<c01602d5>] shrink_cache+0x205/0x610
 [<c0127515>] autoremove_wake_function+0x25/0x50
 [<c0161228>] shrink_zone+0x78/0xa0
 [<c01612ee>] shrink_caches+0x9e/0xc0
 [<c01613b7>] try_to_free_pages+0xa7/0x170
 [<c015541a>] __alloc_pages+0x1fa/0x380
 [<c0165c17>] do_anonymous_page+0xc7/0x520
 [<c017a899>] do_sync_write+0x89/0xc0
 [<c01668f2>] handle_mm_fault+0x132/0x310
 [<c0121940>] do_page_fault+0x350/0x56a
 [<c01698de>] do_brk+0x14e/0x230
 [<c016790e>] sys_brk+0xee/0x120
 [<c01215f0>] do_page_fault+0x0/0x56a
 [<c010cb79>] error_code+0x2d/0x38
Call Trace:
 [<c0126b1b>] __might_sleep+0xab/0xd0
 [<c01677b9>] remove_shared_vm_struct+0x39/0xa0
 [<c0169be1>] exit_mmap+0x1d1/0x2b0
 [<c0127935>] mmput+0xb5/0x150
 [<c012e062>] do_exit+0x1b2/0x960
 [<c01215f0>] do_page_fault+0x0/0x56a
 [<c01215f0>] do_page_fault+0x0/0x56a
 [<c010d367>] die+0x227/0x230
 [<c01217d6>] do_page_fault+0x1e6/0x56a
 [<c0122d1e>] recalc_task_prio+0x8e/0x1b0
 [<c0122f9a>] try_to_wake_up+0x15a/0x290
 [<c01215f0>] do_page_fault+0x0/0x56a
 [<c010cb79>] error_code+0x2d/0x38
 [<c015ac3c>] free_block+0x4c/0x2f0
 [<c015b7ad>] kmem_cache_free+0x1ad/0x3a0
 [<c018158c>] free_buffer_head+0x2c/0x60
 [<c018158c>] free_buffer_head+0x2c/0x60
 [<c0181353>] try_to_free_buffers+0x143/0x220
 [<c0294bab>] linvfs_release_page+0x7b/0x80
 [<c017f243>] try_to_release_page+0x53/0x60
 [<c015fec7>] shrink_list+0x8b7/0xac0
 [<c0123eef>] schedule+0x36f/0x8a0
 [<c015e489>] __pagevec_release+0x29/0x40
 [<c01602d5>] shrink_cache+0x205/0x610
 [<c0127515>] autoremove_wake_function+0x25/0x50
 [<c0161228>] shrink_zone+0x78/0xa0
 [<c01612ee>] shrink_caches+0x9e/0xc0
 [<c01613b7>] try_to_free_pages+0xa7/0x170
 [<c015541a>] __alloc_pages+0x1fa/0x380
 [<c0165c17>] do_anonymous_page+0xc7/0x520
 [<c017a899>] do_sync_write+0x89/0xc0
 [<c01668f2>] handle_mm_fault+0x132/0x310
 [<c0121940>] do_page_fault+0x350/0x56a
 [<c01698de>] do_brk+0x14e/0x230
 [<c016790e>] sys_brk+0xee/0x120
 [<c01215f0>] do_page_fault+0x0/0x56a
 [<c010cb79>] error_code+0x2d/0x38
Warning (Oops_read): Code line not seen, dumping what data is available


Trace; c0124411 <schedule+891/8a0>
Trace; c0163ae1 <unmap_page_range+41/70>
Trace; c0163d24 <unmap_vmas+214/330>
Trace; c0169adb <exit_mmap+cb/2b0>
Trace; c0127935 <mmput+b5/150>
Trace; c012e062 <do_exit+1b2/960>
Trace; c01215f0 <do_page_fault+0/56a>
Trace; c01215f0 <do_page_fault+0/56a>
Trace; c010d367 <die+227/230>
Trace; c01217d6 <do_page_fault+1e6/56a>
Trace; c0122d1e <recalc_task_prio+8e/1b0>
Trace; c0122f9a <try_to_wake_up+15a/290>
Trace; c01215f0 <do_page_fault+0/56a>
Trace; c010cb79 <error_code+2d/38>
Trace; c015ac3c <free_block+4c/2f0>
Trace; c015afba <cache_flusharray+da/2b0>
Trace; c015b7ad <kmem_cache_free+1ad/3a0>
Trace; c018158c <free_buffer_head+2c/60>
Trace; c018158c <free_buffer_head+2c/60>
Trace; c0181353 <try_to_free_buffers+143/220>
Trace; c0294bab <linvfs_release_page+7b/80>
Trace; c017f243 <try_to_release_page+53/60>
Trace; c015fec7 <shrink_list+8b7/ac0>
Trace; c0123eef <schedule+36f/8a0>
Trace; c015e489 <__pagevec_release+29/40>
Trace; c01602d5 <shrink_cache+205/610>
Trace; c0127515 <autoremove_wake_function+25/50>
Trace; c0161228 <shrink_zone+78/a0>
Trace; c01612ee <shrink_caches+9e/c0>
Trace; c01613b7 <try_to_free_pages+a7/170>
Trace; c015541a <__alloc_pages+1fa/380>
Trace; c0165c17 <do_anonymous_page+c7/520>
Trace; c017a899 <do_sync_write+89/c0>
Trace; c01668f2 <handle_mm_fault+132/310>
Trace; c0121940 <do_page_fault+350/56a>
Trace; c01698de <do_brk+14e/230>
Trace; c016790e <sys_brk+ee/120>
Trace; c01215f0 <do_page_fault+0/56a>
Trace; c010cb79 <error_code+2d/38>
Trace; c0124411 <schedule+891/8a0>
Trace; c0163ae1 <unmap_page_range+41/70>
Trace; c0163d24 <unmap_vmas+214/330>
Trace; c0169adb <exit_mmap+cb/2b0>
Trace; c0127935 <mmput+b5/150>
Trace; c012e062 <do_exit+1b2/960>
Trace; c01215f0 <do_page_fault+0/56a>
Trace; c01215f0 <do_page_fault+0/56a>
Trace; c010d367 <die+227/230>
Trace; c01217d6 <do_page_fault+1e6/56a>
Trace; c0122d1e <recalc_task_prio+8e/1b0>
Trace; c0122f9a <try_to_wake_up+15a/290>
Trace; c01215f0 <do_page_fault+0/56a>
Trace; c010cb79 <error_code+2d/38>
Trace; c015ac3c <free_block+4c/2f0>
Trace; c015afba <cache_flusharray+da/2b0>
Trace; c015b7ad <kmem_cache_free+1ad/3a0>
Trace; c018158c <free_buffer_head+2c/60>
Trace; c018158c <free_buffer_head+2c/60>
Trace; c0181353 <try_to_free_buffers+143/220>
Trace; c0294bab <linvfs_release_page+7b/80>
Trace; c017f243 <try_to_release_page+53/60>
Trace; c015fec7 <shrink_list+8b7/ac0>
Trace; c0123eef <schedule+36f/8a0>
Trace; c015e489 <__pagevec_release+29/40>
Trace; c01602d5 <shrink_cache+205/610>
Trace; c0127515 <autoremove_wake_function+25/50>
Trace; c0161228 <shrink_zone+78/a0>
Trace; c01612ee <shrink_caches+9e/c0>
Trace; c01613b7 <try_to_free_pages+a7/170>
Trace; c015541a <__alloc_pages+1fa/380>
Trace; c0165c17 <do_anonymous_page+c7/520>
Trace; c017a899 <do_sync_write+89/c0>
Trace; c01668f2 <handle_mm_fault+132/310>
Trace; c0121940 <do_page_fault+350/56a>
Trace; c01698de <do_brk+14e/230>
Trace; c016790e <sys_brk+ee/120>
Trace; c01215f0 <do_page_fault+0/56a>
Trace; c010cb79 <error_code+2d/38>
Trace; c0124411 <schedule+891/8a0>
Trace; c0163ae1 <unmap_page_range+41/70>
Trace; c0163d24 <unmap_vmas+214/330>
Trace; c0169adb <exit_mmap+cb/2b0>
Trace; c0127935 <mmput+b5/150>
Trace; c012e062 <do_exit+1b2/960>
Trace; c01215f0 <do_page_fault+0/56a>
Trace; c01215f0 <do_page_fault+0/56a>
Trace; c010d367 <die+227/230>
Trace; c01217d6 <do_page_fault+1e6/56a>
Trace; c0122d1e <recalc_task_prio+8e/1b0>
Trace; c0122f9a <try_to_wake_up+15a/290>
Trace; c01215f0 <do_page_fault+0/56a>
Trace; c010cb79 <error_code+2d/38>
Trace; c015ac3c <free_block+4c/2f0>
Trace; c015afba <cache_flusharray+da/2b0>
Trace; c015b7ad <kmem_cache_free+1ad/3a0>
Trace; c018158c <free_buffer_head+2c/60>
Trace; c018158c <free_buffer_head+2c/60>
Trace; c0181353 <try_to_free_buffers+143/220>
Trace; c0294bab <linvfs_release_page+7b/80>
Trace; c017f243 <try_to_release_page+53/60>
Trace; c015fec7 <shrink_list+8b7/ac0>
Trace; c0123eef <schedule+36f/8a0>
Trace; c015e489 <__pagevec_release+29/40>
Trace; c01602d5 <shrink_cache+205/610>
Trace; c0127515 <autoremove_wake_function+25/50>
Trace; c0161228 <shrink_zone+78/a0>
Trace; c01612ee <shrink_caches+9e/c0>
Trace; c01613b7 <try_to_free_pages+a7/170>
Trace; c015541a <__alloc_pages+1fa/380>
Trace; c0165c17 <do_anonymous_page+c7/520>
Trace; c017a899 <do_sync_write+89/c0>
Trace; c01668f2 <handle_mm_fault+132/310>
Trace; c0121940 <do_page_fault+350/56a>
Trace; c01698de <do_brk+14e/230>
Trace; c016790e <sys_brk+ee/120>
Trace; c01215f0 <do_page_fault+0/56a>
Trace; c010cb79 <error_code+2d/38>
Trace; c0126b1b <__might_sleep+ab/d0>
Trace; c01677b9 <remove_shared_vm_struct+39/a0>
Trace; c0169be1 <exit_mmap+1d1/2b0>
Trace; c0127935 <mmput+b5/150>
Trace; c012e062 <do_exit+1b2/960>
Trace; c01215f0 <do_page_fault+0/56a>
Trace; c01215f0 <do_page_fault+0/56a>
Trace; c010d367 <die+227/230>
Trace; c01217d6 <do_page_fault+1e6/56a>
Trace; c0122d1e <recalc_task_prio+8e/1b0>
Trace; c0122f9a <try_to_wake_up+15a/290>
Trace; c01215f0 <do_page_fault+0/56a>
Trace; c010cb79 <error_code+2d/38>
Trace; c015ac3c <free_block+4c/2f0>
Trace; c015b7ad <kmem_cache_free+1ad/3a0>
Trace; c018158c <free_buffer_head+2c/60>
Trace; c018158c <free_buffer_head+2c/60>
Trace; c0181353 <try_to_free_buffers+143/220>
Trace; c0294bab <linvfs_release_page+7b/80>
Trace; c017f243 <try_to_release_page+53/60>
Trace; c015fec7 <shrink_list+8b7/ac0>
Trace; c0123eef <schedule+36f/8a0>
Trace; c015e489 <__pagevec_release+29/40>
Trace; c01602d5 <shrink_cache+205/610>
Trace; c0127515 <autoremove_wake_function+25/50>
Trace; c0161228 <shrink_zone+78/a0>
Trace; c01612ee <shrink_caches+9e/c0>
Trace; c01613b7 <try_to_free_pages+a7/170>
Trace; c015541a <__alloc_pages+1fa/380>
Trace; c0165c17 <do_anonymous_page+c7/520>
Trace; c017a899 <do_sync_write+89/c0>
Trace; c01668f2 <handle_mm_fault+132/310>
Trace; c0121940 <do_page_fault+350/56a>
Trace; c01698de <do_brk+14e/230>
Trace; c016790e <sys_brk+ee/120>
Trace; c01215f0 <do_page_fault+0/56a>
Trace; c010cb79 <error_code+2d/38>

Re: [Oops] i386 mm/slab.c (cache_flusharray)

[Andrew Morton]

pinotj@club-internet.fr wrote:
>
> kernel BUG at mm/slab.c:1957!
> invalid operand: 0000 [#1]
> CPU:    0
> EIP:    0060:[free_block+336/752]    Not tainted
> EIP:    0060:[<c015ad40>]    Not tainted
> Using defaults from ksymoops -t elf32-i386 -a i386
> EFLAGS: 00010096
> eax: 00000045   ebx: 00000006   ecx: c0693854   edx: c056e4f8
> esi: cd09a000   edi: cd09a018   ebp: cf821c68   esp: cf821c3c
> ds: 007b   es: 007b   ss: 0068
> Stack: c0502240 c0502e1d cd09af18 c0652a00 00000001 0000003a cd09af18 0000000f
>        cffdef08 c4bcd180 00000010 cf821ca0 c015afba cffed800 cffdef08 00000010
>        00000282 c1161ca0 00000000 00000001 cffee730 00000010 00010c00 c4bcd180
> Call Trace:
>  [<c015afba>] cache_flusharray+0xda/0x2b0
>  [<c015b7ad>] kmem_cache_free+0x1ad/0x3a0
>  [<c018158c>] free_buffer_head+0x2c/0x60
>  [<c018158c>] free_buffer_head+0x2c/0x60

urgh, there are several reports of this and it's always the buffer_head
slab.  The code in there is trivial so perhaps it's just that the large
number of buffer_heads makes them a fat target.

You should have also seen the message "slab: double free detected in cache
'buffer_head', objp 0xNNNNNNNN".

Don't know, sorry.