3-order allocation failed

3-order allocation failed

[Dagfinn Ilmari Mannsåker]

My computer is serving stuff via Apache over an ssh tunnel. Since
2.4.5-ac-something (I thint it was -ac5, not quite sure), I've been getting
lots of these messages in the kernel log (up to 1200/min, less with later
kernels, it seems):

Jul 23 21:06:01 ilm kernel: __alloc_pages: 3-order allocation failed.

This only occurs when there has been considerable traffic (over 200kB/s) for
some time, and only when it's Apache over SSH. I can push >800kB/s for hours
with samba without getting these messages. I haven't tried apache without SSH,
but I'll try that tomoroow.
The data is being served from a 36GB ReiserFS sw-raid0 array on 2 UltraWide
SCSI drives on a Tekram 390F controller.

I'm currently using a RealTek 8139 NIC (8139too driver); but it also occured
with a RealTek 8029 (ne2k-pci).

When this occurs, transfers stall after a few hundred kilobytes (but directory
indices and small documents work just fine), and X programs are unable to
connect to the X server. As soon as I stop the transfers, everything is OK.

Some system information:
Linux version 2.4.7 (ilmari@ilm.nlc.no) (gcc version 2.95.4 20010703
(Debian prerelease)) #2 Sat Jul 21 19:03:00 CEST 2001

Relevant devices:
00:09.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL-8029(AS)
        Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-
        Status: Cap- 66Mhz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
        Interrupt: pin A routed to IRQ 15
        Region 0: I/O ports at e400 [size=32]

00:0d.0 SCSI storage controller: Symbios Logic Inc. (formerly NCR) 53c875 (rev 26)
        Subsystem: Tekram Technology Co.,Ltd. DC390F Ultra Wide SCSI Controller
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR- FastB2B-
        Status: Cap+ 66Mhz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
        Latency: 32 (4250ns min, 16000ns max), cache line size 08
        Interrupt: pin A routed to IRQ 11
        Region 0: I/O ports at e800 [size=256]
        Region 1: Memory at db004000 (32-bit, non-prefetchable) [size=256]
        Region 2: Memory at db005000 (32-bit, non-prefetchable) [size=4K]
        Expansion ROM at <unassigned> [disabled] [size=64K]
        Capabilities: [40] Power Management version 1
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
                Status: D0 PME-Enable- DSel=0 DScale=0 PME-

00:11.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL-8139 (rev 10)
        Subsystem: Kingston Technologies EtheRx
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-
        Status: Cap+ 66Mhz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
        Latency: 32 (8000ns min, 16000ns max)
        Interrupt: pin A routed to IRQ 15
        Region 0: I/O ports at ec00 [size=256]
        Region 1: Memory at db006000 (32-bit, non-prefetchable) [size=256]
        Capabilities: [50] Power Management version 2
                Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1+,D2+,D3hot+,D3cold-)
                Status: D0 PME-Enable- DSel=0 DScale=0 PME-

Related packages:
ii  gcc       2.95.4-4  The GNU C compiler.
ii  libc6     2.2.3-7   GNU C Library: Shared libraries and Timezone data
ii  apache    1.3.20-1  Versatile, high-performance HTTP server
ii  ssh       2.9p2-4   Secure rlogin/rsh/rcp replacement (OpenSSH)
ii  samba     2.2.1a-1  A LanManager like file and printer server for Unix.

-- 
Dagfinn I. Mannsåker
GPG Public Key ID: 0x51ECFAC6
Fingerprint:  48BB A64D CE9B 9A06 65DF  395C D42E CDC4 51EC FAC6

Re: 3-order allocation failed

[Tigran Aivazian]

Hi Rik,

A simple test seems to show problems with page allocator.

a) take a 6G RAM machine

b) take a 70G harddisk

c) mke2fs on it

observe lots of "0-order allocation failures" while looking at
/proc/meminfo reveals that I still have 5.1G of free memory (presumably
some of it in NORMAL zone, not just all HIGH)... Isn't buddy allocator
clever enough to break up the multi-page chunks if we need single pages?
I don't know.

Regards,
Tigran

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: 3-order allocation failed

[Rik van Riel]

On Thu, 2 Nov 2000, Anthony Towns wrote:
> On Wed, Nov 01, 2000 at 01:42:17PM -0800, Dunlap, Randy wrote:
> > > Is this bug in the usb-driver (usb-uhci), in the camera's driver
> > > (cpia_usb) or in the v4l?
> > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> > Could there be a memory leak as well?  But I expect that
> > it's simply that memory is just fragmented so that enough
> > contiguous pages aren't available, like Rik said.
> 
> There is a memory leak, the memory kmalloc'ed in cpia_usb_open
> for sbuf[*].data is never kfree'd

This way you'll be running out of order-2 free memory
segments /very/ quickly ... ;)

Rik
--
"What you're running that piece of shit Gnome?!?!"
       -- Miguel de Icaza, UKUUG 2000

http://www.conectiva.com/		http://www.surriel.com/

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: 3-order allocation failed

[Andrew Morton]

"Pasi Kärkkäinen" wrote:
> 
> I added show_stack(0); to the mm/page_alloc.c :
> 
>         /* No luck.. */
>         printk(KERN_ERR "__alloc_pages: %lu-order allocation failed.\n", order)
>         show_stack(0);
>         return NULL;
> 
> Then, when the first stack-dump came to kern.log, I gave it to
> ksymoops. The result can be seen on
> 
> http://edu.joroinen.fi/~pk/ksymoops-output.
> 

Alas:

Nov  1 12:48:34 mansion kernel: c3543e80 c01e5be0 00000002 00000000 00000007 c12277c8 00000007 00000007
Nov  1 12:48:34 mansion kernel:        00000000 c02200d4 c012bb44 c01288ad c12277c8 00000246 00000007 00000000
Nov  1 12:48:34 mansion kernel:        00000001 c0128ab9 c12277c8 00000007 c6529e60 00000000 c885ed60 c886e222
Nov  1 12:48:34 mansion kernel: Call Trace: [inet_check_attr+49792/72172] [<c012bb44>] [<c01288ad>] [<c0128ab9>] [<c885ed60>] [<c886e222>] [<c885ed60>]
Nov  1 12:48:39 mansion kernel:        [<c886911c>] [<c885e185>] [<c013866d>] [<c012f782>] [<c012e9b1>] [<c012e8ea>] [<c012ebdc>] [<c010a31f>] <3>__alloc_pages: 2-order
allocation failed.

...

Trace; c886911c <[cpia]cpia_open+88/160>
Trace; c885e185 <[videodev]video_open+79/94>
Trace; c013866d <permission+95/f4>
Trace; c012f782 <chrdev_open+3e/4c>
Trace; c012e9b1 <dentry_open+bd/148>
Trace; c012e8ea <filp_open+52/5c>
Trace; c012ebdc <sys_open+38/b4>
Trace; c010a31f <system_call+33/38>
Trace; c886911c <[cpia]cpia_open+88/160>        

So your klogd tried to interpret the trace and screwed it up.  Then
ksymoops tried to interpret klogd's output and screwed it up.

Could you please change you init scripts so `klogd' is 
started with the `-x' option and then restart your logging
daemons?  There's a reasonable chance that if you do this
your klogd will segfault and stop working when it sees the
trace - I'm not sure if Debian have fixed this one.

Alternatively, if you still have that kernel,

cd /usr/src/linux
gdb vmlinux
x/10i 0xc01e5be0
x/10i 0xc02200d4
x/10i 0xc012bb44
x/10i 0xc01288ad
[etc]

That should (finally) tell us where the allocations are occurring.
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

RE: 3-order allocation failed

[Dunlap, Randy]

Wow, I bet that's been there awhile.

BTW, did you submit this patch earlier?  I don't
recall having seen it.

I'll forward it.

Thanks,
~Randy_________________________________________
|randy.dunlap_at_intel.com        503-677-5408|
|NOTE: Any views presented here are mine alone|
|& may not represent the views of my employer.|
----------------------------------------------- 

> -----Original Message-----
> From: Anthony Towns [mailto:aj@azure.humbug.org.au]
> Sent: Wednesday, November 01, 2000 2:21 PM
> To: unlisted-recipients
> Cc: 'Pasi Kärkkäinen'; Rik van Riel; Forever shall I be.;
> linux-kernel@vger.kernel.org
> Subject: Re: 3-order allocation failed
> 
> 
> On Wed, Nov 01, 2000 at 01:42:17PM -0800, Dunlap, Randy wrote:
> > > Is this bug in the usb-driver (usb-uhci), in the camera's driver
> > > (cpia_usb) or in the v4l?
> > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> > Could there be a memory leak as well?  But I expect that
> > it's simply that memory is just fragmented so that enough
> > contiguous pages aren't available, like Rik said.
> 
> There is a memory leak, the memory kmalloc'ed in cpia_usb_open for
> sbuf[*].data is never kfree'd (except when an error occurs during
> initialisation). Adding some kfree's in cpia_usb_free_resources fixed
> the problem in test7 or so, here's a patch against -test10.
> 
> --- drivers/media/video/cpia_usb.c.orig	Wed Nov  1 14:14:37 2000
> +++ drivers/media/video/cpia_usb.c	Wed Nov  1 14:16:05 2000
> @@ -432,10 +432,20 @@
>  		ucpia->sbuf[1].urb = NULL;
>  	}
>  
> +	if (ucpia->sbuf[1].data) {
> +		kfree(ucpia->sbuf[1].data);
> +		ucpia->sbuf[1].data = NULL;
> +	}
> + 
>  	if (ucpia->sbuf[0].urb) {
>  		usb_unlink_urb(ucpia->sbuf[0].urb);
>  		usb_free_urb(ucpia->sbuf[0].urb);
>  		ucpia->sbuf[0].urb = NULL;
> +	}
> +
> +	if (ucpia->sbuf[0].data) {
> +		kfree(ucpia->sbuf[0].data);
> +		ucpia->sbuf[0].data = NULL;
>  	}
>  }
> 
> HTH.
> 
> Cheers,
> aj 
> 
> -- 
> Anthony Towns <aj@humbug.org.au> <http://azure.humbug.org.au/~aj/>
> I don't speak for anyone save myself. GPG signed mail preferred.

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: 3-order allocation failed

[Anthony Towns]

On Wed, Nov 01, 2000 at 01:42:17PM -0800, Dunlap, Randy wrote:
> > Is this bug in the usb-driver (usb-uhci), in the camera's driver
> > (cpia_usb) or in the v4l?
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> Could there be a memory leak as well?  But I expect that
> it's simply that memory is just fragmented so that enough
> contiguous pages aren't available, like Rik said.

There is a memory leak, the memory kmalloc'ed in cpia_usb_open for
sbuf[*].data is never kfree'd (except when an error occurs during
initialisation). Adding some kfree's in cpia_usb_free_resources fixed
the problem in test7 or so, here's a patch against -test10.

--- drivers/media/video/cpia_usb.c.orig	Wed Nov  1 14:14:37 2000
+++ drivers/media/video/cpia_usb.c	Wed Nov  1 14:16:05 2000
@@ -432,10 +432,20 @@
 		ucpia->sbuf[1].urb = NULL;
 	}
 
+	if (ucpia->sbuf[1].data) {
+		kfree(ucpia->sbuf[1].data);
+		ucpia->sbuf[1].data = NULL;
+	}
+ 
 	if (ucpia->sbuf[0].urb) {
 		usb_unlink_urb(ucpia->sbuf[0].urb);
 		usb_free_urb(ucpia->sbuf[0].urb);
 		ucpia->sbuf[0].urb = NULL;
+	}
+
+	if (ucpia->sbuf[0].data) {
+		kfree(ucpia->sbuf[0].data);
+		ucpia->sbuf[0].data = NULL;
 	}
 }

HTH.

Cheers,
aj 

-- 
Anthony Towns <aj@humbug.org.au> <http://azure.humbug.org.au/~aj/>
I don't speak for anyone save myself. GPG signed mail preferred.

  ``We reject: kings, presidents, and voting.
                 We believe in: rough consensus and working code.''
                                      -- Dave Clark
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

RE: 3-order allocation failed

[Dunlap, Randy]

Hi,

> From: Pasi Kärkkäinen [mailto:pk@edu.joroinen.fi]
> 
> On Thu, 26 Oct 2000, Rik van Riel wrote:
> 
> > On Thu, 26 Oct 2000, Forever shall I be. wrote:
> > > On Thu, Oct 26, 2000 at 02:57:30PM +0300, Pasi Kärkkäinen wrote:
> > 
> > > > __alloc_pages: 2-order allocation failed.
> > > > __alloc_pages: 2-order allocation failed.
> > > > __alloc_pages: 5-order allocation failed.
> > > > __alloc_pages: 4-order allocation failed.
> > > > __alloc_pages: 3-order allocation failed.
> > > > __alloc_pages: 2-order allocation failed.
> > > > __alloc_pages: 5-order allocation failed.
> > > > 
> > > > Any ideas?
> > > 
> > > I'm getting __alloc_pages: 7-order allocation failed.
> > > all the time in 2.4.0-test9 on my "pIII (Katmai)".. kernel's
> > > compiled with 2.95.2 + bounds, without -fbounds-checking
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Are there many places in the kernel that
do that (alloc 2^8 = 256 pages) .. after init?
Do you know where/what it is?  Sound driver maybe?

> > It means something in the system is trying to allocate a
> > large continuous area of memory that isn't available...
> > 
> > The printk is basically a debug output indicating that we
> > don't have the large physically contiguous area available
> > that's being requested.
> > 
> > Basically everything bigger than order-1 (2 contiguous
> > pages) is unreliable at runtime. Orders 2 and 3 should
> > usually be available (if you only allocate very few of
> > them) and higher orders should not be relied upon.
> > 
> > If somebody is seeing a lot of these messages, it means
> > that some driver in the system is asking unreasonable
> > things from the VM subsystem ;)
> > 
> > (and buffer allocations are failing)
> > 
> 
> I got those order-x messages when I was running a script, that looked
> something like this:
> 
> 	streamer -s 320x240 -o webcam.jpg
> 	sleep 5
> 
> It worked fine for about 20 minutes, and after I started to get those
> messages and the camera didn't work anymore.
> 
> Solution: I'm now using a program, that is "using the camera all the
> time" and it just saves the frames with 5 seconds delay.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Makes sense.  Good, practical workaround solution.

I looked thru cpia_usb_open() and everything that it
calls or causes in your scenario, and the only
order-2 allocation that I see is in cpia_usb_open().

If order-1 allocs are much better than order-2 allocs,
like Rik said, then you could change
drivers/media/video/cpia_usb.c line 41 (in 2.4.0-test10)
to be: #define FRAMES_PER_DESC	8
instead of 10.  That would make the kmalloc()'s in
cpia_usb_open() be order-1 instead of order-2.
All of the other kmalloc()s in the USB subsystem in this
scenarios are already small (less than 1 page).
Please let me/us know how this works if you try it.


> The script I was running previously used streamer, that 
> initializes and 
> opens the v4l-device everytime I run it.
> 
> Is this bug in the usb-driver (usb-uhci), in the camera's driver
> (cpia_usb) or in the v4l?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Could there be a memory leak as well?  But I expect that
it's simply that memory is just fragmented so that enough
contiguous pages aren't available, like Rik said.

~Randy

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: 3-order allocation failed

[Pasi Kärkkäinen]

On Thu, 26 Oct 2000, Rik van Riel wrote:

> On Thu, 26 Oct 2000, Forever shall I be. wrote:
> > On Thu, Oct 26, 2000 at 02:57:30PM +0300, Pasi Kärkkäinen wrote:
> 
> > > __alloc_pages: 2-order allocation failed.
> > > __alloc_pages: 2-order allocation failed.
> > > __alloc_pages: 5-order allocation failed.
> > > __alloc_pages: 4-order allocation failed.
> > > __alloc_pages: 3-order allocation failed.
> > > __alloc_pages: 2-order allocation failed.
> > > __alloc_pages: 5-order allocation failed.
> > > 
> > > Any ideas?
> > 
> > I'm getting __alloc_pages: 7-order allocation failed.
> > all the time in 2.4.0-test9 on my "pIII (Katmai)".. kernel's
> > compiled with 2.95.2 + bounds, without -fbounds-checking
> 
> It means something in the system is trying to allocate a
> large continuous area of memory that isn't available...
> 
> The printk is basically a debug output indicating that we
> don't have the large physically contiguous area available
> that's being requested.
> 
> Basically everything bigger than order-1 (2 contiguous
> pages) is unreliable at runtime. Orders 2 and 3 should
> usually be available (if you only allocate very few of
> them) and higher orders should not be relied upon.
> 
> If somebody is seeing a lot of these messages, it means
> that some driver in the system is asking unreasonable
> things from the VM subsystem ;)
> 
> (and buffer allocations are failing)
> 

I added show_stack(0); to the mm/page_alloc.c :

        /* No luck.. */
        printk(KERN_ERR "__alloc_pages: %lu-order allocation failed.\n", order)
        show_stack(0);
        return NULL;

Then, when the first stack-dump came to kern.log, I gave it to
ksymoops. The result can be seen on

http://edu.joroinen.fi/~pk/ksymoops-output.

Hope that helps someone. If not, I may do further tests.

- Pasi Kärkkäinen
       
                                   ^
                                .     .
                                 Linux
                              /    -    \
                             Choice.of.the
                           .Next.Generation.

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: 3-order allocation failed

[Pasi Kärkkäinen]

On Thu, 26 Oct 2000, Rik van Riel wrote:

> On Thu, 26 Oct 2000, Forever shall I be. wrote:
> > On Thu, Oct 26, 2000 at 02:57:30PM +0300, Pasi Kärkkäinen wrote:
> 
> > > __alloc_pages: 2-order allocation failed.
> > > __alloc_pages: 2-order allocation failed.
> > > __alloc_pages: 5-order allocation failed.
> > > __alloc_pages: 4-order allocation failed.
> > > __alloc_pages: 3-order allocation failed.
> > > __alloc_pages: 2-order allocation failed.
> > > __alloc_pages: 5-order allocation failed.
> > > 
> > > Any ideas?
> > 
> > I'm getting __alloc_pages: 7-order allocation failed.
> > all the time in 2.4.0-test9 on my "pIII (Katmai)".. kernel's
> > compiled with 2.95.2 + bounds, without -fbounds-checking
> 
> It means something in the system is trying to allocate a
> large continuous area of memory that isn't available...
> 
> The printk is basically a debug output indicating that we
> don't have the large physically contiguous area available
> that's being requested.
> 
> Basically everything bigger than order-1 (2 contiguous
> pages) is unreliable at runtime. Orders 2 and 3 should
> usually be available (if you only allocate very few of
> them) and higher orders should not be relied upon.
> 
> If somebody is seeing a lot of these messages, it means
> that some driver in the system is asking unreasonable
> things from the VM subsystem ;)
> 
> (and buffer allocations are failing)
> 

I got those order-x messages when I was running a script, that looked
something like this:

	streamer -s 320x240 -o webcam.jpg
	sleep 5

It worked fine for about 20 minutes, and after I started to get those
messages and the camera didn't work anymore.

Solution: I'm now using a program, that is "using the camera all the
time" and it just saves the frames with 5 seconds delay.

The script I was running previously used streamer, that initializes and 
opens the v4l-device everytime I run it.

Is this bug in the usb-driver (usb-uhci), in the camera's driver
(cpia_usb) or in the v4l?

- Pasi Kärkkäinen
       
                                   ^
                                .     .
                                 Linux
                              /    -    \
                             Choice.of.the
                           .Next.Generation.

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: 3-order allocation failed

[Rik van Riel]

On Thu, 26 Oct 2000, Forever shall I be. wrote:
> On Thu, Oct 26, 2000 at 02:57:30PM +0300, Pasi Kärkkäinen wrote:

> > __alloc_pages: 2-order allocation failed.
> > __alloc_pages: 2-order allocation failed.
> > __alloc_pages: 5-order allocation failed.
> > __alloc_pages: 4-order allocation failed.
> > __alloc_pages: 3-order allocation failed.
> > __alloc_pages: 2-order allocation failed.
> > __alloc_pages: 5-order allocation failed.
> > 
> > Any ideas?
> 
> I'm getting __alloc_pages: 7-order allocation failed.
> all the time in 2.4.0-test9 on my "pIII (Katmai)".. kernel's
> compiled with 2.95.2 + bounds, without -fbounds-checking

It means something in the system is trying to allocate a
large continuous area of memory that isn't available...

The printk is basically a debug output indicating that we
don't have the large physically contiguous area available
that's being requested.

Basically everything bigger than order-1 (2 contiguous
pages) is unreliable at runtime. Orders 2 and 3 should
usually be available (if you only allocate very few of
them) and higher orders should not be relied upon.

If somebody is seeing a lot of these messages, it means
that some driver in the system is asking unreasonable
things from the VM subsystem ;)

(and buffer allocations are failing)

regards,

Rik
--
"What you're running that piece of shit Gnome?!?!"
       -- Miguel de Icaza, UKUUG 2000

http://www.conectiva.com/		http://www.surriel.com/

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/