Re: [ANOMALIES]: 2.4.2 - __alloc_pages: failed - Patch failed

Re: [ANOMALIES]: 2.4.2 - __alloc_pages: failed - Patch failed

[Shawn Starr]

Right now before OOPS:
Mem:    62244K av,   61292K used,     952K free,       0K shrd,    1496K buff
Swap:  467444K av,   37344K used,  430100K free                   29528K cached

I got a lot of things running, several daemons, netscape, and other things. I put
a 400MB swap for now just to help things.

Here's what happens after oops:

Wait a second...before I didn't have the 400MB swap on, and I had about 952K of
physical ram left. Shouldn't it try and swap if it cant get enough physical
memory?

It did NOT oops with that amount of swap:

If i turn it off:

Mem:    62244K av,   61288K used,     956K free,       0K shrd,    1448K buff
Swap:   64252K av,   38024K used,   26228K free                   29880K cached

and try the xcdrgtk (X CDRoaster)

I get...hrm.. this is really strange. Now its not ooping?!

I dont know it must have to do with something somewhere I cant tell you.

Mike Galbraith wrote:

> On Sun, 25 Feb 2001, Shawn Starr wrote:
>
> > Unsure, the system remains stable after the fault though, strangely /dev/dsp
> > becomes "busy". I suspect it has to do with this somehow.. but im not sure.
> > I submitted a ksymoops dump, maybe that can help.
>
> Drop to single user and do a whopping big dd or iozone or bonnie
> and see what free reports afterward.  If much of your ram becomes
> available, it's not a leak.
>
>         -Mike

Re: [ANOMALIES]: 2.4.2 - __alloc_pages: failed - Patch failed

[Mike Galbraith]

The way sg_low_malloc() tries to allocate, failure messages are
pretty much garanteed.  It tries high order allocations (which
are unreliable even when not stressed) and backs off until it
succeeds.

In other words, the messages are a red herring.

	-Mike

Re: [ANOMALIES]: 2.4.2 - __alloc_pages: failed - Patch failed

[Shawn Starr]

Well, I Discovered, something strange. I put in a blank new CD-R, so these
errors are not of concern? (In which case why have the kernel log get
spewed with them if they are guaranteed to happen?)

Shawn.

--
Hugged a Tux today? (tm)

On Sun, 25 Feb 2001, Mike Galbraith wrote:

> The way sg_low_malloc() tries to allocate, failure messages are
> pretty much garanteed.  It tries high order allocations (which
> are unreliable even when not stressed) and backs off until it
> succeeds.
>
> In other words, the messages are a red herring.
>
> 	-Mike
>
>
>

Re: [ANOMALIES]: 2.4.2 - __alloc_pages: failed - Patch failed

[Marcelo Tosatti]

On Sun, 25 Feb 2001, Mike Galbraith wrote:

> The way sg_low_malloc() tries to allocate, failure messages are
> pretty much garanteed.  It tries high order allocations (which
> are unreliable even when not stressed) and backs off until it
> succeeds.
> 
> In other words, the messages are a red herring.

Yup. And they should not be printed. 

We can add an allocation flag (__GFP_NO_CRITICAL?) which can be used by
sg_low_malloc() (and other non critical allocations) to fail previously
and not print the message. 

Linus ? 

Re: [ANOMALIES]: 2.4.2 - __alloc_pages: failed - Patch failed

[Alan Cox]

> We can add an allocation flag (__GFP_NO_CRITICAL?) which can be used by
> sg_low_malloc() (and other non critical allocations) to fail previously
> and not print the message. 

It is just for debugging. The message can go. If anytbing it would be more
useful to tack Failed alloc data on the end of /proc/slabinfo

Re: [ANOMALIES]: 2.4.2 - __alloc_pages: failed - Patch failed

[Marcelo Tosatti]

On Mon, 26 Feb 2001, Alan Cox wrote:

> > We can add an allocation flag (__GFP_NO_CRITICAL?) which can be used by
> > sg_low_malloc() (and other non critical allocations) to fail previously
> > and not print the message. 
> 
> It is just for debugging. The message can go. If anytbing it would be more
> useful to tack Failed alloc data on the end of /proc/slabinfo

The issue is not the warn message.

Non critical allocations (such as this case of sg_low_malloc()) are trying
to get additional memory to optimize things -- we want the allocator to be
lazy and fail previously instead doing hard work. If kswapd cannot keep up
with the memory pressure, we're surely in a memory shortage state.

Its better to get out of the memory shortage instead running into OOM
because of some optimization, I guess.

Another example of such a flag is swapin readahead.

Re: [ANOMALIES]: 2.4.2 - __alloc_pages: failed - Causes more then just msgs

[Shawn Starr]

It may not be an important message but what does happen is /dev/dsp becomes
hung and no sound works after the fault. So something is definately wrong.

Shawn.

Marcelo Tosatti wrote:

> On Mon, 26 Feb 2001, Alan Cox wrote:
>
> > > We can add an allocation flag (__GFP_NO_CRITICAL?) which can be used by
> > > sg_low_malloc() (and other non critical allocations) to fail previously
> > > and not print the message.
> >
> > It is just for debugging. The message can go. If anytbing it would be more
> > useful to tack Failed alloc data on the end of /proc/slabinfo
>
> The issue is not the warn message.
>
> Non critical allocations (such as this case of sg_low_malloc()) are trying
> to get additional memory to optimize things -- we want the allocator to be
> lazy and fail previously instead doing hard work. If kswapd cannot keep up
> with the memory pressure, we're surely in a memory shortage state.
>
> Its better to get out of the memory shortage instead running into OOM
> because of some optimization, I guess.
>
> Another example of such a flag is swapin readahead.

--
Hugged a Tux today? (tm)

Re: [ANOMALIES]: 2.4.2 - __alloc_pages: failed - Causes more then just msgs

[Mike Galbraith]

On Mon, 26 Feb 2001, Shawn Starr wrote:

> It may not be an important message but what does happen is /dev/dsp becomes
> hung and no sound works after the fault. So something is definately wrong.

Do you mean it hangs without the BUG() inserted, or only after the oops?

	-Mike

Re: [ANOMALIES]: 2.4.2 - __alloc_pages: failed - Causes more then just msgs

[Shawn Starr]

When added with BUG(); it will hang /dev/dsp.

I'm not sure if it did without it. I'll be restarting with the removed
BUG(); soon.

On Tue, 27 Feb 2001, Mike Galbraith wrote:

> On Mon, 26 Feb 2001, Shawn Starr wrote:
>
> > It may not be an important message but what does happen is /dev/dsp becomes
> > hung and no sound works after the fault. So something is definately wrong.
>
> Do you mean it hangs without the BUG() inserted, or only after the oops?
>
> 	-Mike
>
>
>

Re: [ANOMALIES]: 2.4.2 - __alloc_pages: failed - Causes more then just msgs

[Mike Galbraith]

On Tue, 27 Feb 2001, Shawn Starr wrote:

> When added with BUG(); it will hang /dev/dsp.

If the device is opened and we oops before closing it, subsequent
open attempts will fail (busy).  If it hangs after a failed high
order allocation attempt without the BUG() insertion, that could
be called a driver bug.

	-Mike

Re: [ANOMALIES]: 2.4.2 - __alloc_pages: failed & mount hanging withloop device issues

[Shawn Starr]

Ok apply patch and loop patch... I'll let you know what happens in my next
email.

Mike Galbraith wrote:

> On Fri, 23 Feb 2001, Shawn Starr wrote:
>
> > Feb 23 03:31:18 coredump kernel: __alloc_pages: 3-order allocation
> > failed.
> > Feb 23 03:31:18 coredump kernel: __alloc_pages: 3-order allocation
> > failed.
> > Feb 23 03:31:18 coredump kernel: __alloc_pages: 2-order allocation
> > failed.
> > Feb 23 03:31:18 coredump kernel: __alloc_pages: 3-order allocation
> > failed.
> > Feb 23 03:31:18 coredump kernel: __alloc_pages: 3-order allocation
> > failed.
> > Feb 23 03:31:18 coredump kernel: __alloc_pages: 2-order allocation
> > failed.
> > Feb 23 03:31:18 coredump kernel: __alloc_pages: 3-order allocation
> > failed.
> >
> > The use of mkisofs and xcdroster with cdrecord seems to cause this fault
> > in kernel.log
>
> Hi,
>
> Can you try the below for the high order allocation problem?  We don't
> try very hard at all to service high order allocations under load.  If
> it helps, let me know and I'll submit it to Rik for consideration.
>
> (for loop troubles, you should try Jens' latest loop patch located in
> your favorite kernel mirror under pub/linux/kernel/people/axboe)
>
>         -Mike
>
> (patch was done against 2.4.1-ac20, but should go in ok)
> --- mm/page_alloc.c.org Fri Feb 23 13:21:54 2001
> +++ mm/page_alloc.c     Fri Feb 23 13:28:33 2001
> @@ -274,7 +274,7 @@
>  struct page * __alloc_pages(zonelist_t *zonelist, unsigned long order)
>  {
>         zone_t **zone;
> -       int direct_reclaim = 0;
> +       int direct_reclaim = 0, loop = 0;
>         unsigned int gfp_mask = zonelist->gfp_mask;
>         struct page * page;
>
> @@ -366,7 +366,7 @@
>          *   able to free some memory we can't free ourselves
>          */
>         wakeup_kswapd();
> -       if (gfp_mask & __GFP_WAIT) {
> +       if (gfp_mask & __GFP_WAIT && loop) {
>                 __set_current_state(TASK_RUNNING);
>                 current->policy |= SCHED_YIELD;
>                 schedule();
> @@ -393,6 +393,7 @@
>          *      --> try to free pages ourselves with page_launder
>          */
>         if (!(current->flags & PF_MEMALLOC)) {
> +               loop++;
>                 /*
>                  * Are we dealing with a higher order allocation?
>                  *
> @@ -440,7 +441,7 @@
>                         memory_pressure++;
>                         try_to_free_pages(gfp_mask);
>                         wakeup_bdflush(0);
> -                       if (!order)
> +                       if (!order || loop < (1 << order))
>                                 goto try_again;
>                 }
>         }

Re: [ANOMALIES]: 2.4.2 - __alloc_pages: failed - Patch failed

[Shawn Starr]

Feb 23 21:17:47 coredump kernel: __alloc_pages: 3-order allocation failed.
Feb 23 21:17:47 coredump kernel: __alloc_pages: 2-order allocation failed.
Feb 23 21:17:47 coredump kernel: __alloc_pages: 1-order allocation failed.
Feb 23 21:17:47 coredump kernel: __alloc_pages: 3-order allocation failed.
Feb 23 21:17:47 coredump kernel: __alloc_pages: 3-order allocation failed.
Feb 23 21:17:47 coredump kernel: __alloc_pages: 2-order allocation failed.
Feb 23 21:17:47 coredump kernel: __alloc_pages: 1-order allocation failed.

didnt, work, still causing this..

Shawn Starr wrote:

> Ok apply patch and loop patch... I'll let you know what happens in my next
> email.
>
> Mike Galbraith wrote:
>
> > On Fri, 23 Feb 2001, Shawn Starr wrote:
> >
> > > Feb 23 03:31:18 coredump kernel: __alloc_pages: 3-order allocation
> > > failed.
> > > Feb 23 03:31:18 coredump kernel: __alloc_pages: 3-order allocation
> > > failed.
> > > Feb 23 03:31:18 coredump kernel: __alloc_pages: 2-order allocation
> > > failed.
> > > Feb 23 03:31:18 coredump kernel: __alloc_pages: 3-order allocation
> > > failed.
> > > Feb 23 03:31:18 coredump kernel: __alloc_pages: 3-order allocation
> > > failed.
> > > Feb 23 03:31:18 coredump kernel: __alloc_pages: 2-order allocation
> > > failed.
> > > Feb 23 03:31:18 coredump kernel: __alloc_pages: 3-order allocation
> > > failed.
> > >
> > > The use of mkisofs and xcdroster with cdrecord seems to cause this fault
> > > in kernel.log
> >
> > Hi,
> >
> > Can you try the below for the high order allocation problem?  We don't
> > try very hard at all to service high order allocations under load.  If
> > it helps, let me know and I'll submit it to Rik for consideration.
> >
> > (for loop troubles, you should try Jens' latest loop patch located in
> > your favorite kernel mirror under pub/linux/kernel/people/axboe)
> >
> >         -Mike
> >
> > (patch was done against 2.4.1-ac20, but should go in ok)
> > --- mm/page_alloc.c.org Fri Feb 23 13:21:54 2001
> > +++ mm/page_alloc.c     Fri Feb 23 13:28:33 2001
> > @@ -274,7 +274,7 @@
> >  struct page * __alloc_pages(zonelist_t *zonelist, unsigned long order)
> >  {
> >         zone_t **zone;
> > -       int direct_reclaim = 0;
> > +       int direct_reclaim = 0, loop = 0;
> >         unsigned int gfp_mask = zonelist->gfp_mask;
> >         struct page * page;
> >
> > @@ -366,7 +366,7 @@
> >          *   able to free some memory we can't free ourselves
> >          */
> >         wakeup_kswapd();
> > -       if (gfp_mask & __GFP_WAIT) {
> > +       if (gfp_mask & __GFP_WAIT && loop) {
> >                 __set_current_state(TASK_RUNNING);
> >                 current->policy |= SCHED_YIELD;
> >                 schedule();
> > @@ -393,6 +393,7 @@
> >          *      --> try to free pages ourselves with page_launder
> >          */
> >         if (!(current->flags & PF_MEMALLOC)) {
> > +               loop++;
> >                 /*
> >                  * Are we dealing with a higher order allocation?
> >                  *
> > @@ -440,7 +441,7 @@
> >                         memory_pressure++;
> >                         try_to_free_pages(gfp_mask);
> >                         wakeup_bdflush(0);
> > -                       if (!order)
> > +                       if (!order || loop < (1 << order))
> >                                 goto try_again;
> >                 }
> >         }
>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

Re: [ANOMALIES]: 2.4.2 - __alloc_pages: failed - Patch failed

[Marcelo Tosatti]

On Fri, 23 Feb 2001, Shawn Starr wrote:

> Feb 23 21:17:47 coredump kernel: __alloc_pages: 3-order allocation failed.
> Feb 23 21:17:47 coredump kernel: __alloc_pages: 2-order allocation failed.
> Feb 23 21:17:47 coredump kernel: __alloc_pages: 1-order allocation failed.
> Feb 23 21:17:47 coredump kernel: __alloc_pages: 3-order allocation failed.
> Feb 23 21:17:47 coredump kernel: __alloc_pages: 3-order allocation failed.
> Feb 23 21:17:47 coredump kernel: __alloc_pages: 2-order allocation failed.
> Feb 23 21:17:47 coredump kernel: __alloc_pages: 1-order allocation failed.
> 
> didnt, work, still causing this..

Ok, could you please add a line with "BUG();" after the
printk("__alloc_pages: %d-order allocation failed", ..) in mm/page_alloc.c
function __alloc_pages() ?

This will make you get an oops when an allocation fails and if you decode
it (with ksymoops) we can have a pretty useful backtrace to have more clue
of what's failing.

TIA

Re: [ANOMALIES]: 2.4.2 - __alloc_pages: failed - Patch failed

[Shawn Starr]

Doing so..., Im not sure hot to use ksymoops or where to get that program.
I just usually use the sysq and dump but its ugly ;-)

Shawn.

Marcelo Tosatti wrote:

> On Fri, 23 Feb 2001, Shawn Starr wrote:
>
> > Feb 23 21:17:47 coredump kernel: __alloc_pages: 3-order allocation failed.
> > Feb 23 21:17:47 coredump kernel: __alloc_pages: 2-order allocation failed.
> > Feb 23 21:17:47 coredump kernel: __alloc_pages: 1-order allocation failed.
> > Feb 23 21:17:47 coredump kernel: __alloc_pages: 3-order allocation failed.
> > Feb 23 21:17:47 coredump kernel: __alloc_pages: 3-order allocation failed.
> > Feb 23 21:17:47 coredump kernel: __alloc_pages: 2-order allocation failed.
> > Feb 23 21:17:47 coredump kernel: __alloc_pages: 1-order allocation failed.
> >
> > didnt, work, still causing this..
>
> Ok, could you please add a line with "BUG();" after the
> printk("__alloc_pages: %d-order allocation failed", ..) in mm/page_alloc.c
> function __alloc_pages() ?
>
> This will make you get an oops when an allocation fails and if you decode
> it (with ksymoops) we can have a pretty useful backtrace to have more clue
> of what's failing.
>
> TIA

Re: [ANOMALIES]: 2.4.2 - __alloc_pages: failed - Patch failed

[Arnaldo Carvalho de Melo]

Em Sat, Feb 24, 2001 at 05:31:55PM -0500, Shawn Starr escreveu:
> Doing so..., Im not sure hot to use ksymoops or where to get that program.
> I just usually use the sysq and dump but its ugly ;-)

http://www.kernel.org/pub/linux/utils/kernel/ksymoops/v2.4/ksymoops-2.4.0.tar.bz2

- Arnaldo

Re: [ANOMALIES]: 2.4.2 - __alloc_pages: failed - Patch failed

[Mike Galbraith]

On Fri, 23 Feb 2001, Shawn Starr wrote:

> Feb 23 21:17:47 coredump kernel: __alloc_pages: 3-order allocation failed.
> Feb 23 21:17:47 coredump kernel: __alloc_pages: 2-order allocation failed.
> Feb 23 21:17:47 coredump kernel: __alloc_pages: 1-order allocation failed.
> Feb 23 21:17:47 coredump kernel: __alloc_pages: 3-order allocation failed.
> Feb 23 21:17:47 coredump kernel: __alloc_pages: 3-order allocation failed.
> Feb 23 21:17:47 coredump kernel: __alloc_pages: 2-order allocation failed.
> Feb 23 21:17:47 coredump kernel: __alloc_pages: 1-order allocation failed.
>
> didnt, work, still causing this..

What does free output look like after abort?  (is someone leaking or
is all memory just temporarily tied up?)

	-Mike