Re: [ANOMALIES]: 2.4.2 - __alloc_pages: failed & mount hanging withloop device issues

Re: [ANOMALIES]: 2.4.2 - __alloc_pages: failed & mount hanging withloop device issues

[Shawn Starr]

Ok apply patch and loop patch... I'll let you know what happens in my next
email.

Mike Galbraith wrote:

> On Fri, 23 Feb 2001, Shawn Starr wrote:
>
> > Feb 23 03:31:18 coredump kernel: __alloc_pages: 3-order allocation
> > failed.
> > Feb 23 03:31:18 coredump kernel: __alloc_pages: 3-order allocation
> > failed.
> > Feb 23 03:31:18 coredump kernel: __alloc_pages: 2-order allocation
> > failed.
> > Feb 23 03:31:18 coredump kernel: __alloc_pages: 3-order allocation
> > failed.
> > Feb 23 03:31:18 coredump kernel: __alloc_pages: 3-order allocation
> > failed.
> > Feb 23 03:31:18 coredump kernel: __alloc_pages: 2-order allocation
> > failed.
> > Feb 23 03:31:18 coredump kernel: __alloc_pages: 3-order allocation
> > failed.
> >
> > The use of mkisofs and xcdroster with cdrecord seems to cause this fault
> > in kernel.log
>
> Hi,
>
> Can you try the below for the high order allocation problem?  We don't
> try very hard at all to service high order allocations under load.  If
> it helps, let me know and I'll submit it to Rik for consideration.
>
> (for loop troubles, you should try Jens' latest loop patch located in
> your favorite kernel mirror under pub/linux/kernel/people/axboe)
>
>         -Mike
>
> (patch was done against 2.4.1-ac20, but should go in ok)
> --- mm/page_alloc.c.org Fri Feb 23 13:21:54 2001
> +++ mm/page_alloc.c     Fri Feb 23 13:28:33 2001
> @@ -274,7 +274,7 @@
>  struct page * __alloc_pages(zonelist_t *zonelist, unsigned long order)
>  {
>         zone_t **zone;
> -       int direct_reclaim = 0;
> +       int direct_reclaim = 0, loop = 0;
>         unsigned int gfp_mask = zonelist->gfp_mask;
>         struct page * page;
>
> @@ -366,7 +366,7 @@
>          *   able to free some memory we can't free ourselves
>          */
>         wakeup_kswapd();
> -       if (gfp_mask & __GFP_WAIT) {
> +       if (gfp_mask & __GFP_WAIT && loop) {
>                 __set_current_state(TASK_RUNNING);
>                 current->policy |= SCHED_YIELD;
>                 schedule();
> @@ -393,6 +393,7 @@
>          *      --> try to free pages ourselves with page_launder
>          */
>         if (!(current->flags & PF_MEMALLOC)) {
> +               loop++;
>                 /*
>                  * Are we dealing with a higher order allocation?
>                  *
> @@ -440,7 +441,7 @@
>                         memory_pressure++;
>                         try_to_free_pages(gfp_mask);
>                         wakeup_bdflush(0);
> -                       if (!order)
> +                       if (!order || loop < (1 << order))
>                                 goto try_again;
>                 }
>         }

Re: [ANOMALIES]: 2.4.2 - __alloc_pages: failed - Patch failed

[Shawn Starr]

Feb 23 21:17:47 coredump kernel: __alloc_pages: 3-order allocation failed.
Feb 23 21:17:47 coredump kernel: __alloc_pages: 2-order allocation failed.
Feb 23 21:17:47 coredump kernel: __alloc_pages: 1-order allocation failed.
Feb 23 21:17:47 coredump kernel: __alloc_pages: 3-order allocation failed.
Feb 23 21:17:47 coredump kernel: __alloc_pages: 3-order allocation failed.
Feb 23 21:17:47 coredump kernel: __alloc_pages: 2-order allocation failed.
Feb 23 21:17:47 coredump kernel: __alloc_pages: 1-order allocation failed.

didnt, work, still causing this..

Shawn Starr wrote:

> Ok apply patch and loop patch... I'll let you know what happens in my next
> email.
>
> Mike Galbraith wrote:
>
> > On Fri, 23 Feb 2001, Shawn Starr wrote:
> >
> > > Feb 23 03:31:18 coredump kernel: __alloc_pages: 3-order allocation
> > > failed.
> > > Feb 23 03:31:18 coredump kernel: __alloc_pages: 3-order allocation
> > > failed.
> > > Feb 23 03:31:18 coredump kernel: __alloc_pages: 2-order allocation
> > > failed.
> > > Feb 23 03:31:18 coredump kernel: __alloc_pages: 3-order allocation
> > > failed.
> > > Feb 23 03:31:18 coredump kernel: __alloc_pages: 3-order allocation
> > > failed.
> > > Feb 23 03:31:18 coredump kernel: __alloc_pages: 2-order allocation
> > > failed.
> > > Feb 23 03:31:18 coredump kernel: __alloc_pages: 3-order allocation
> > > failed.
> > >
> > > The use of mkisofs and xcdroster with cdrecord seems to cause this fault
> > > in kernel.log
> >
> > Hi,
> >
> > Can you try the below for the high order allocation problem?  We don't
> > try very hard at all to service high order allocations under load.  If
> > it helps, let me know and I'll submit it to Rik for consideration.
> >
> > (for loop troubles, you should try Jens' latest loop patch located in
> > your favorite kernel mirror under pub/linux/kernel/people/axboe)
> >
> >         -Mike
> >
> > (patch was done against 2.4.1-ac20, but should go in ok)
> > --- mm/page_alloc.c.org Fri Feb 23 13:21:54 2001
> > +++ mm/page_alloc.c     Fri Feb 23 13:28:33 2001
> > @@ -274,7 +274,7 @@
> >  struct page * __alloc_pages(zonelist_t *zonelist, unsigned long order)
> >  {
> >         zone_t **zone;
> > -       int direct_reclaim = 0;
> > +       int direct_reclaim = 0, loop = 0;
> >         unsigned int gfp_mask = zonelist->gfp_mask;
> >         struct page * page;
> >
> > @@ -366,7 +366,7 @@
> >          *   able to free some memory we can't free ourselves
> >          */
> >         wakeup_kswapd();
> > -       if (gfp_mask & __GFP_WAIT) {
> > +       if (gfp_mask & __GFP_WAIT && loop) {
> >                 __set_current_state(TASK_RUNNING);
> >                 current->policy |= SCHED_YIELD;
> >                 schedule();
> > @@ -393,6 +393,7 @@
> >          *      --> try to free pages ourselves with page_launder
> >          */
> >         if (!(current->flags & PF_MEMALLOC)) {
> > +               loop++;
> >                 /*
> >                  * Are we dealing with a higher order allocation?
> >                  *
> > @@ -440,7 +441,7 @@
> >                         memory_pressure++;
> >                         try_to_free_pages(gfp_mask);
> >                         wakeup_bdflush(0);
> > -                       if (!order)
> > +                       if (!order || loop < (1 << order))
> >                                 goto try_again;
> >                 }
> >         }
>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

Re: [ANOMALIES]: 2.4.2 - __alloc_pages: failed & mount hanging withloop device issues

[Rik van Riel]

On Fri, 23 Feb 2001, Shawn Starr wrote:
> Mike Galbraith wrote:
> >  [patch]

Mike, this patch looks like a really good idea.

regards,

Rik
--
Virtual memory is like a game you can't win;
However, without VM there's truly nothing to lose...

		http://www.surriel.com/
http://www.conectiva.com/	http://distro.conectiva.com.br/

Re: [ANOMALIES]: 2.4.2 - __alloc_pages: failed - Patch failed

[Marcelo Tosatti]

On Fri, 23 Feb 2001, Shawn Starr wrote:

> Feb 23 21:17:47 coredump kernel: __alloc_pages: 3-order allocation failed.
> Feb 23 21:17:47 coredump kernel: __alloc_pages: 2-order allocation failed.
> Feb 23 21:17:47 coredump kernel: __alloc_pages: 1-order allocation failed.
> Feb 23 21:17:47 coredump kernel: __alloc_pages: 3-order allocation failed.
> Feb 23 21:17:47 coredump kernel: __alloc_pages: 3-order allocation failed.
> Feb 23 21:17:47 coredump kernel: __alloc_pages: 2-order allocation failed.
> Feb 23 21:17:47 coredump kernel: __alloc_pages: 1-order allocation failed.
> 
> didnt, work, still causing this..

Ok, could you please add a line with "BUG();" after the
printk("__alloc_pages: %d-order allocation failed", ..) in mm/page_alloc.c
function __alloc_pages() ?

This will make you get an oops when an allocation fails and if you decode
it (with ksymoops) we can have a pretty useful backtrace to have more clue
of what's failing.

TIA

Re: [ANOMALIES]: 2.4.2 - __alloc_pages: failed - Patch failed

[Arnaldo Carvalho de Melo]

Em Sat, Feb 24, 2001 at 05:31:55PM -0500, Shawn Starr escreveu:
> Doing so..., Im not sure hot to use ksymoops or where to get that program.
> I just usually use the sysq and dump but its ugly ;-)

http://www.kernel.org/pub/linux/utils/kernel/ksymoops/v2.4/ksymoops-2.4.0.tar.bz2

- Arnaldo

Re: [ANOMALIES]: 2.4.2 - __alloc_pages: failed - Patch failed

[Shawn Starr]

Doing so..., Im not sure hot to use ksymoops or where to get that program.
I just usually use the sysq and dump but its ugly ;-)

Shawn.

Marcelo Tosatti wrote:

> On Fri, 23 Feb 2001, Shawn Starr wrote:
>
> > Feb 23 21:17:47 coredump kernel: __alloc_pages: 3-order allocation failed.
> > Feb 23 21:17:47 coredump kernel: __alloc_pages: 2-order allocation failed.
> > Feb 23 21:17:47 coredump kernel: __alloc_pages: 1-order allocation failed.
> > Feb 23 21:17:47 coredump kernel: __alloc_pages: 3-order allocation failed.
> > Feb 23 21:17:47 coredump kernel: __alloc_pages: 3-order allocation failed.
> > Feb 23 21:17:47 coredump kernel: __alloc_pages: 2-order allocation failed.
> > Feb 23 21:17:47 coredump kernel: __alloc_pages: 1-order allocation failed.
> >
> > didnt, work, still causing this..
>
> Ok, could you please add a line with "BUG();" after the
> printk("__alloc_pages: %d-order allocation failed", ..) in mm/page_alloc.c
> function __alloc_pages() ?
>
> This will make you get an oops when an allocation fails and if you decode
> it (with ksymoops) we can have a pretty useful backtrace to have more clue
> of what's failing.
>
> TIA

Re: [ANOMALIES]: 2.4.2 - __alloc_pages: ksysoops debug info

[Shawn Starr]

[-- Attachment #1: Type: text/plain, Size: 1635 bytes --]

Ok, I copied the part from kern.log and pasted it into a separate file then ran
ksysoops and it appeared to have worked.

Any more info you need?

Shawn.

Shawn Starr wrote:

> Doing so..., Im not sure hot to use ksymoops or where to get that program.
> I just usually use the sysq and dump but its ugly ;-)
>
> Shawn.
>
> Marcelo Tosatti wrote:
>
> > On Fri, 23 Feb 2001, Shawn Starr wrote:
> >
> > > Feb 23 21:17:47 coredump kernel: __alloc_pages: 3-order allocation failed.
> > > Feb 23 21:17:47 coredump kernel: __alloc_pages: 2-order allocation failed.
> > > Feb 23 21:17:47 coredump kernel: __alloc_pages: 1-order allocation failed.
> > > Feb 23 21:17:47 coredump kernel: __alloc_pages: 3-order allocation failed.
> > > Feb 23 21:17:47 coredump kernel: __alloc_pages: 3-order allocation failed.
> > > Feb 23 21:17:47 coredump kernel: __alloc_pages: 2-order allocation failed.
> > > Feb 23 21:17:47 coredump kernel: __alloc_pages: 1-order allocation failed.
> > >
> > > didnt, work, still causing this..
> >
> > Ok, could you please add a line with "BUG();" after the
> > printk("__alloc_pages: %d-order allocation failed", ..) in mm/page_alloc.c
> > function __alloc_pages() ?
> >
> > This will make you get an oops when an allocation fails and if you decode
> > it (with ksymoops) we can have a pretty useful backtrace to have more clue
> > of what's failing.
> >
> > TIA
>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

[-- Attachment #2: oops --]
[-- Type: text/plain, Size: 2521 bytes --]

ksymoops 2.3.7 on i586 2.4.2.  Options used
     -V (default)
     -k /proc/ksyms (default)
     -l /proc/modules (default)
     -o /lib/modules/2.4.2/ (default)
     -m /boot/System.map (specified)

Feb 24 19:57:32 coredump kernel: kernel BUG at page_alloc.c:507!
Feb 24 19:57:32 coredump kernel: invalid operand: 0000
Feb 24 19:57:32 coredump kernel: CPU:    0
Feb 24 19:57:32 coredump kernel: EIP:    0010:[__alloc_pages+761/776]
Feb 24 19:57:32 coredump kernel: EFLAGS: 00013282
Feb 24 19:57:32 coredump kernel: eax: 00000020   ebx: 00000000   ecx: c1ba8000   edx: c025abe8
Feb 24 19:57:32 coredump kernel: esi: c025c000   edi: 00000003   ebp: c1d06000   esp: c1d07d38
Feb 24 19:57:32 coredump kernel: ds: 0018   es: 0018   ss: 0018
Feb 24 19:57:32 coredump kernel: Process cdda2wav (pid: 386, stackpage=c1d07000)
Feb 24 19:57:32 coredump kernel: Stack: c0219a05 c0219b93 000001fb 00000003 00008000 00008000 00000000 00000007
Feb 24 19:57:32 coredump kernel:        00000008 00000000 c025bff4 c0127fe4 c01b5079 c1d07de4 00000000 00000001
Feb 24 19:57:32 coredump kernel:        c1d07de0 00008000 00000007 c01b5142 00008000 00000000 00000001 00000000
Feb 24 19:57:32 coredump kernel: Call Trace: [__get_free_pages+20/36] [sg_low_malloc+305/408] [sg_malloc+98/280] [sg_build_indi+385/440] [sg_build_reserve+37/68] [sg_ioctl+1582/2588] [__get_free_pages+20/36]
Feb 24 19:57:32 coredump kernel: Code: 0f 0b 83 c4 0c 31 c0 5b 5e 5f 5d 83 c4 10 c3 83 fa 09 77 13
Using defaults from ksymoops -t elf32-i386 -a i386

Code;  00000000 Before first symbol
00000000 <_EIP>:
Code;  00000000 Before first symbol
   0:   0f 0b                     ud2a   
Code;  00000002 Before first symbol
   2:   83 c4 0c                  add    $0xc,%esp
Code;  00000005 Before first symbol
   5:   31 c0                     xor    %eax,%eax
Code;  00000007 Before first symbol
   7:   5b                        pop    %ebx
Code;  00000008 Before first symbol
   8:   5e                        pop    %esi
Code;  00000009 Before first symbol
   9:   5f                        pop    %edi
Code;  0000000a Before first symbol
   a:   5d                        pop    %ebp
Code;  0000000b Before first symbol
   b:   83 c4 10                  add    $0x10,%esp
Code;  0000000e Before first symbol
   e:   c3                        ret    
Code;  0000000f Before first symbol
   f:   83 fa 09                  cmp    $0x9,%edx
Code;  00000012 Before first symbol
  12:   77 13                     ja     27 <_EIP+0x27> 00000027 Before first symbol

Re: [ANOMALIES]: 2.4.2 - __alloc_pages: failed - Patch failed

[Mike Galbraith]

On Fri, 23 Feb 2001, Shawn Starr wrote:

> Feb 23 21:17:47 coredump kernel: __alloc_pages: 3-order allocation failed.
> Feb 23 21:17:47 coredump kernel: __alloc_pages: 2-order allocation failed.
> Feb 23 21:17:47 coredump kernel: __alloc_pages: 1-order allocation failed.
> Feb 23 21:17:47 coredump kernel: __alloc_pages: 3-order allocation failed.
> Feb 23 21:17:47 coredump kernel: __alloc_pages: 3-order allocation failed.
> Feb 23 21:17:47 coredump kernel: __alloc_pages: 2-order allocation failed.
> Feb 23 21:17:47 coredump kernel: __alloc_pages: 1-order allocation failed.
>
> didnt, work, still causing this..

What does free output look like after abort?  (is someone leaking or
is all memory just temporarily tied up?)

	-Mike

Re: [ANOMALIES]: 2.4.2 - __alloc_pages: failed - Patch failed

[Marcelo Tosatti]

On Mon, 26 Feb 2001, Alan Cox wrote:

> > We can add an allocation flag (__GFP_NO_CRITICAL?) which can be used by
> > sg_low_malloc() (and other non critical allocations) to fail previously
> > and not print the message. 
> 
> It is just for debugging. The message can go. If anytbing it would be more
> useful to tack Failed alloc data on the end of /proc/slabinfo

The issue is not the warn message.

Non critical allocations (such as this case of sg_low_malloc()) are trying
to get additional memory to optimize things -- we want the allocator to be
lazy and fail previously instead doing hard work. If kswapd cannot keep up
with the memory pressure, we're surely in a memory shortage state.

Its better to get out of the memory shortage instead running into OOM
because of some optimization, I guess.

Another example of such a flag is swapin readahead.

Re: [ANOMALIES]: 2.4.2 - __alloc_pages: failed - Patch failed

[Alan Cox]

> We can add an allocation flag (__GFP_NO_CRITICAL?) which can be used by
> sg_low_malloc() (and other non critical allocations) to fail previously
> and not print the message. 

It is just for debugging. The message can go. If anytbing it would be more
useful to tack Failed alloc data on the end of /proc/slabinfo

Re: [ANOMALIES]: 2.4.2 - __alloc_pages: failed - Patch failed

[Marcelo Tosatti]

On Sun, 25 Feb 2001, Mike Galbraith wrote:

> The way sg_low_malloc() tries to allocate, failure messages are
> pretty much garanteed.  It tries high order allocations (which
> are unreliable even when not stressed) and backs off until it
> succeeds.
> 
> In other words, the messages are a red herring.

Yup. And they should not be printed. 

We can add an allocation flag (__GFP_NO_CRITICAL?) which can be used by
sg_low_malloc() (and other non critical allocations) to fail previously
and not print the message. 

Linus ? 

Re: [ANOMALIES]: 2.4.2 - __alloc_pages: failed - Patch failed

[Shawn Starr]

Well, I Discovered, something strange. I put in a blank new CD-R, so these
errors are not of concern? (In which case why have the kernel log get
spewed with them if they are guaranteed to happen?)

Shawn.

--
Hugged a Tux today? (tm)

On Sun, 25 Feb 2001, Mike Galbraith wrote:

> The way sg_low_malloc() tries to allocate, failure messages are
> pretty much garanteed.  It tries high order allocations (which
> are unreliable even when not stressed) and backs off until it
> succeeds.
>
> In other words, the messages are a red herring.
>
> 	-Mike
>
>
>

Re: [ANOMALIES]: 2.4.2 - __alloc_pages: failed - Patch failed

[Mike Galbraith]

The way sg_low_malloc() tries to allocate, failure messages are
pretty much garanteed.  It tries high order allocations (which
are unreliable even when not stressed) and backs off until it
succeeds.

In other words, the messages are a red herring.

	-Mike

Re: [ANOMALIES]: 2.4.2 - __alloc_pages: failed - Patch failed

[Shawn Starr]

Right now before OOPS:
Mem:    62244K av,   61292K used,     952K free,       0K shrd,    1496K buff
Swap:  467444K av,   37344K used,  430100K free                   29528K cached

I got a lot of things running, several daemons, netscape, and other things. I put
a 400MB swap for now just to help things.

Here's what happens after oops:

Wait a second...before I didn't have the 400MB swap on, and I had about 952K of
physical ram left. Shouldn't it try and swap if it cant get enough physical
memory?

It did NOT oops with that amount of swap:

If i turn it off:

Mem:    62244K av,   61288K used,     956K free,       0K shrd,    1448K buff
Swap:   64252K av,   38024K used,   26228K free                   29880K cached

and try the xcdrgtk (X CDRoaster)

I get...hrm.. this is really strange. Now its not ooping?!

I dont know it must have to do with something somewhere I cant tell you.

Mike Galbraith wrote:

> On Sun, 25 Feb 2001, Shawn Starr wrote:
>
> > Unsure, the system remains stable after the fault though, strangely /dev/dsp
> > becomes "busy". I suspect it has to do with this somehow.. but im not sure.
> > I submitted a ksymoops dump, maybe that can help.
>
> Drop to single user and do a whopping big dd or iozone or bonnie
> and see what free reports afterward.  If much of your ram becomes
> available, it's not a leak.
>
>         -Mike