* Kernel 2.6.9 Page Allocation Failures w/TSO+rollup.patch
@ 2004-10-25 10:48 Justin Piszcz
2004-10-25 11:33 ` Nick Piggin
0 siblings, 1 reply; 12+ messages in thread
From: Justin Piszcz @ 2004-10-25 10:48 UTC (permalink / raw)
To: linux-kernel; +Cc: akpm
I guess people who get this should just stick with 2.6.8.1?
$ dmesg
nfsd: page allocation failure. order:0, mode:0x20
[<c013923c>] __alloc_pages+0x21c/0x350
[<c0139388>] __get_free_pages+0x18/0x40
[<c013c9ef>] kmem_getpages+0x1f/0xc0
[<c013d730>] cache_grow+0xc0/0x1a0
[<c013d9db>] cache_alloc_refill+0x1cb/0x210
[<c013de41>] __kmalloc+0x71/0x80
[<c036f583>] alloc_skb+0x53/0x100
[<c031fb18>] e1000_alloc_rx_buffers+0x48/0xf0
[<c031f81e>] e1000_clean_rx_irq+0x18e/0x440
[<c0106a2f>] handle_IRQ_event+0x6f/0x80
[<c031f3fb>] e1000_clean+0x5b/0x100
[<c0375c0a>] net_rx_action+0x6a/0xf0
[<c011daa1>] __do_softirq+0x41/0x90
[<c011db17>] do_softirq+0x27/0x30
[<c0106ebc>] do_IRQ+0x10c/0x130
[<c01049c8>] common_interrupt+0x18/0x20
[<c013007b>] simplify_symbols+0x5b/0x110
[<c014019a>] shrink_list+0x30a/0x4b0
[<c01404a3>] shrink_cache+0x163/0x380
[<c0140c42>] shrink_zone+0xa2/0xd0
[<c0140cc3>] shrink_caches+0x53/0x70
[<c0140d8f>] try_to_free_pages+0xaf/0x1b0
[<c0139287>] __alloc_pages+0x267/0x350
[<c0136763>] generic_file_buffered_write+0x123/0x660
[<c013918b>] __alloc_pages+0x16b/0x350
[<c013d557>] alloc_slabmgmt+0x57/0x70
[<c027c156>] xfs_trans_unlocked_item+0x56/0x60
[<c02929cf>] xfs_write+0x78f/0xc00
[<c028de01>] linvfs_writev+0x101/0x140
[<c0116c60>] autoremove_wake_function+0x0/0x60
[<c02a5e52>] copy_from_user+0x42/0x70
[<c01542fa>] do_readv_writev+0x28a/0x2b0
[<c028dfe0>] linvfs_open+0x0/0x90
[<c01049c8>] common_interrupt+0x18/0x20
[<c0153bc0>] do_sync_write+0x0/0x110
[<c028e03a>] linvfs_open+0x5a/0x90
[<c0152da2>] dentry_open+0xd2/0x270
[<c01543d8>] vfs_writev+0x58/0x60
[<c01caf26>] nfsd_write+0xf6/0x390
[<c01049c8>] common_interrupt+0x18/0x20
[<c01d33bb>] nfsd3_proc_write+0xbb/0x120
[<c01c66c3>] nfsd_dispatch+0xa3/0x250
[<c041f4e1>] svc_process+0x6e1/0x7f0
[<c01c6463>] nfsd+0x203/0x3c0
[<c01c6260>] nfsd+0x0/0x3c0
[<c010207d>] kernel_thread_helper+0x5/0x18
I am not sure what else to do except stay with 2.6.8.1 as it did not have
these problems.
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Kernel 2.6.9 Page Allocation Failures w/TSO+rollup.patch
2004-10-25 10:48 Kernel 2.6.9 Page Allocation Failures w/TSO+rollup.patch Justin Piszcz
@ 2004-10-25 11:33 ` Nick Piggin
2004-10-25 12:03 ` Justin Piszcz
2004-10-27 21:40 ` Kernel 2.6.9 Multiple Page Allocation Failures (Part 2) Justin Piszcz
0 siblings, 2 replies; 12+ messages in thread
From: Nick Piggin @ 2004-10-25 11:33 UTC (permalink / raw)
To: Justin Piszcz; +Cc: linux-kernel, akpm
Justin Piszcz wrote:
> I guess people who get this should just stick with 2.6.8.1?
>
Does it cause any noticable problems? If not, then stay with
2.6.9.
However, it would be nice to get to the bottom of it. It might
just be happening by chance on 2.6.9 but not 2.6.8.1 though...
Anyway, how often are you getting the messages? How many
ethernet cards in the system?
Can you run a kernel with sysrq support, and do `SysRq+M`
(close to when the allocation failure happens if possible, but
otherwise on a normally running system after it has been up
for a while). Then send in the dmesg.
Thanks,
Nick
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Kernel 2.6.9 Page Allocation Failures w/TSO+rollup.patch
2004-10-25 11:33 ` Nick Piggin
@ 2004-10-25 12:03 ` Justin Piszcz
2004-10-27 21:40 ` Kernel 2.6.9 Multiple Page Allocation Failures (Part 2) Justin Piszcz
1 sibling, 0 replies; 12+ messages in thread
From: Justin Piszcz @ 2004-10-25 12:03 UTC (permalink / raw)
To: Nick Piggin; +Cc: linux-kernel, akpm
It does not cause any noticable problems that I know of, I guess it is
just a bit disturbing.
> Anyway, how often are you getting the messages?
It depends what I am doing, sometimes they happen after 20-30 minutes,
othertimes it takes a day or two.
> How many ethernet cards in the system?
There are 4 ethernet cards in the system.
1) 3c905b (on-board)
2) 3c905b (PCI)
3) 3c900 Combo (PCI)
4) Intel 82541GI/PI (PCI)
> Can you run a kernel with sysrq support, and do `SysRq+M`
> (close to when the allocation failure happens if possible, but
> otherwise on a normally running system after it has been up
> for a while). Then send in the dmesg.
Ok, I will try this.
On Mon, 25 Oct 2004, Nick Piggin wrote:
> Justin Piszcz wrote:
>> I guess people who get this should just stick with 2.6.8.1?
>>
>
> Does it cause any noticable problems? If not, then stay with
> 2.6.9.
>
> However, it would be nice to get to the bottom of it. It might
> just be happening by chance on 2.6.9 but not 2.6.8.1 though...
>
> Anyway, how often are you getting the messages? How many
> ethernet cards in the system?
>
> Can you run a kernel with sysrq support, and do `SysRq+M`
> (close to when the allocation failure happens if possible, but
> otherwise on a normally running system after it has been up
> for a while). Then send in the dmesg.
>
> Thanks,
> Nick
>
^ permalink raw reply [flat|nested] 12+ messages in thread
* Kernel 2.6.9 Multiple Page Allocation Failures (Part 2)
2004-10-25 11:33 ` Nick Piggin
2004-10-25 12:03 ` Justin Piszcz
@ 2004-10-27 21:40 ` Justin Piszcz
2004-10-27 21:58 ` Andrew Morton
2004-10-28 0:33 ` Lee Revell
1 sibling, 2 replies; 12+ messages in thread
From: Justin Piszcz @ 2004-10-27 21:40 UTC (permalink / raw)
To: Nick Piggin; +Cc: linux-kernel, akpm
After a couple days the errors have shown up again.
Can anyone help me debug this problem further?
Is there any chance Linus will freeze 2.6 and make the current development
tree 2.7? It seems like ever since around 2.6.8 things have been getting
progressively worse (page allocation failures/nvidia
breakage/XFS-oops-when-copying-over-nfs-when-the-file-is-being-written-to)?
$ uptime
17:34:32 up 2 days, 8:53, 29 users, load average: 0.29, 0.38, 0.36
jpiszcz@p500:/usr/src/linux$ grep -i sysrq .config
CONFIG_MAGIC_SYSRQ=y
I tried various combinations of sysrq+M but it did not seem to do anything
special?
Also, all of the failures below, could they possibly be causing data
corruption?
Once again, box = Dell GX1 p3 500mhz / 768MB ram (Intel 440BX/ZX/DX)
chipset.
$ lspci
00:00.0 Host bridge: Intel Corp. 440BX/ZX/DX - 82443BX/ZX/DX Host bridge
(rev 03)
00:01.0 PCI bridge: Intel Corp. 440BX/ZX/DX - 82443BX/ZX/DX AGP bridge
(rev 03)
00:07.0 ISA bridge: Intel Corp. 82371AB/EB/MB PIIX4 ISA (rev 02)
00:07.1 IDE interface: Intel Corp. 82371AB/EB/MB PIIX4 IDE (rev 01)
00:07.2 USB Controller: Intel Corp. 82371AB/EB/MB PIIX4 USB (rev 01)
00:07.3 Bridge: Intel Corp. 82371AB/EB/MB PIIX4 ACPI (rev 02)
00:0d.0 Unknown mass storage controller: Promise Technology, Inc. 20269
(rev 02)
00:0e.0 Unknown mass storage controller: Promise Technology, Inc. 20267
(rev 02)
00:0f.0 PCI bridge: Digital Equipment Corporation DECchip 21152 (rev 03)
00:11.0 Ethernet controller: 3Com Corporation 3c905B 100BaseTX [Cyclone]
(rev 24)
01:00.0 VGA compatible controller: ATI Technologies Inc 3D Rage Pro AGP
1X/2X (rev 5c)
02:09.0 Ethernet controller: Intel Corp. 82541GI/PI Gigabit Ethernet
Controller
02:0a.0 Ethernet controller: 3Com Corporation 3c905B 100BaseTX [Cyclone]
(rev 30)
02:0b.0 Ethernet controller: 3Com Corporation 3c900 Combo [Boomerang]
$ cat /proc/interrupts
CPU0
0: 204979514 XT-PIC timer
1: 2484 XT-PIC i8042
2: 0 XT-PIC cascade
5: 0 XT-PIC Crystal audio controller
8: 1 XT-PIC rtc
9: 0 XT-PIC acpi
10: 3866403 XT-PIC ide2, ide3
11: 129079320 XT-PIC ide4, ide5, eth0, eth1, eth2, eth3
12: 16198 XT-PIC i8042
15: 50 XT-PIC ide1
NMI: 0
LOC: 0
ERR: 1022
MIS: 0
$ cat /proc/ioports
0000-001f : dma1
0020-0021 : pic1
0040-0043 : timer0
0050-0053 : timer1
0060-006f : keyboard
0070-0077 : rtc
0080-008f : dma page reg
00a0-00a1 : pic2
00c0-00df : dma2
00f0-00ff : fpu
0170-0177 : ide1
0184-0187 : MPU-401 UART
02f8-02ff : serial
0376-0376 : ide1
03c0-03df : vga+
03f8-03ff : serial
0530-0533 : Crystal audio controller
0800-083f : 0000:00:07.3
0800-0803 : PM1a_EVT_BLK
0804-0805 : PM1a_CNT_BLK
0808-080b : PM_TMR
080c-080f : GPE0_BLK
0840-085f : 0000:00:07.3
0850-0857 : piix4-smbus
0cf8-0cff : PCI conf1
c880-c8ff : 0000:00:11.0
c880-c8ff : 0000:00:11.0
cc40-cc7f : 0000:00:0e.0
cc40-cc47 : ide4
cc48-cc4f : ide5
cc50-cc7f : PDC20267
cca0-ccaf : 0000:00:0d.0
cca0-cca7 : ide2
cca8-ccaf : ide3
ccb0-ccb7 : 0000:00:0e.0
ccb0-ccb7 : ide5
ccb8-ccbb : 0000:00:0d.0
ccba-ccba : ide3
ccbc-ccbf : 0000:00:0e.0
ccbe-ccbe : ide5
ccc0-ccc7 : 0000:00:0d.0
ccc0-ccc7 : ide3
ccc8-cccf : 0000:00:0e.0
ccc8-cccf : ide4
ccd0-ccd3 : 0000:00:0d.0
ccd2-ccd2 : ide2
ccd4-ccd7 : 0000:00:0e.0
ccd6-ccd6 : ide4
ccd8-ccdf : 0000:00:0d.0
ccd8-ccdf : ide2
cce0-ccff : 0000:00:07.2
d000-dfff : PCI Bus #02
dc00-dc7f : 0000:02:0a.0
dc00-dc7f : 0000:02:0a.0
dc80-dcbf : 0000:02:0b.0
dc80-dcbf : 0000:02:0b.0
dcc0-dcff : 0000:02:09.0
dcc0-dcff : e1000
e000-efff : PCI Bus #01
ec00-ecff : 0000:01:00.0
ffa0-ffaf : 0000:00:07.1
ffa8-ffaf : ide1
[<c01ca635>] nfsd_open+0x45/0x190
[<c01cae1e>] nfsd_write+0x4e/0x390
[<c0375f7a>] net_rx_action+0x6a/0xf0
[<c036f9e2>] skb_drop_fraglist+0x42/0x50
[<c036faa6>] skb_release_data+0x96/0xc0
[<c01d335b>] nfsd3_proc_write+0xbb/0x120
[<c01c6663>] nfsd_dispatch+0xa3/0x250
[<c041f841>] svc_process+0x6e1/0x7f0
[<c01c6403>] nfsd+0x203/0x3c0
[<c01c6200>] nfsd+0x0/0x3c0
[<c010207d>] kernel_thread_helper+0x5/0x18
swapper: page allocation failure. order:0, mode:0x20
[<c0139227>] __alloc_pages+0x247/0x3b0
[<c02d9471>] add_interrupt_randomness+0x31/0x40
[<c01393a8>] __get_free_pages+0x18/0x40
[<c013ca2f>] kmem_getpages+0x1f/0xc0
[<c013d770>] cache_grow+0xc0/0x1a0
[<c013da1b>] cache_alloc_refill+0x1cb/0x210
[<c013de81>] __kmalloc+0x71/0x80
[<c036f8f3>] alloc_skb+0x53/0x100
[<c031fe88>] e1000_alloc_rx_buffers+0x48/0xf0
[<c031fb8e>] e1000_clean_rx_irq+0x18e/0x440
[<c031f76b>] e1000_clean+0x5b/0x100
[<c0375f7a>] net_rx_action+0x6a/0xf0
[<c011daa1>] __do_softirq+0x41/0x90
[<c011db17>] do_softirq+0x27/0x30
[<c0106ebc>] do_IRQ+0x10c/0x130
[<c01049c8>] common_interrupt+0x18/0x20
[<c0101df0>] default_idle+0x0/0x40
[<c0101e13>] default_idle+0x23/0x40
[<c0101e9f>] cpu_idle+0x2f/0x50
[<c04f2967>] start_kernel+0x157/0x180
[<c04f2540>] unknown_bootoption+0x0/0x180
nfsd: page allocation failure. order:0, mode:0x20
[<c0139227>] __alloc_pages+0x247/0x3b0
[<c01393a8>] __get_free_pages+0x18/0x40
[<c013ca2f>] kmem_getpages+0x1f/0xc0
[<c013d770>] cache_grow+0xc0/0x1a0
[<c013da1b>] cache_alloc_refill+0x1cb/0x210
[<c013de81>] __kmalloc+0x71/0x80
[<c036f8f3>] alloc_skb+0x53/0x100
[<c031fe88>] e1000_alloc_rx_buffers+0x48/0xf0
[<c031fb8e>] e1000_clean_rx_irq+0x18e/0x440
[<c038abd0>] ip_rcv_finish+0x0/0x2d0
[<c031f76b>] e1000_clean+0x5b/0x100
[<c0375f7a>] net_rx_action+0x6a/0xf0
[<c011daa1>] __do_softirq+0x41/0x90
[<c011db17>] do_softirq+0x27/0x30
[<c0106ebc>] do_IRQ+0x10c/0x130
[<c01049c8>] common_interrupt+0x18/0x20
[<c011007b>] setup_pit_timer+0x2b/0x60
[<c0124d23>] sigprocmask+0x3/0xe0
[<c01c63f3>] nfsd+0x1f3/0x3c0
[<c01c6200>] nfsd+0x0/0x3c0
[<c010207d>] kernel_thread_helper+0x5/0x18
nfsd: page allocation failure. order:0, mode:0x20
[<c0139227>] __alloc_pages+0x247/0x3b0
[<c01393a8>] __get_free_pages+0x18/0x40
[<c013ca2f>] kmem_getpages+0x1f/0xc0
[<c013d770>] cache_grow+0xc0/0x1a0
[<c013da1b>] cache_alloc_refill+0x1cb/0x210
[<c013de81>] __kmalloc+0x71/0x80
[<c036f8f3>] alloc_skb+0x53/0x100
[<c031fe88>] e1000_alloc_rx_buffers+0x48/0xf0
[<c031fb8e>] e1000_clean_rx_irq+0x18e/0x440
[<c038abd0>] ip_rcv_finish+0x0/0x2d0
[<c031f76b>] e1000_clean+0x5b/0x100
[<c0375f7a>] net_rx_action+0x6a/0xf0
[<c011daa1>] __do_softirq+0x41/0x90
[<c011db17>] do_softirq+0x27/0x30
[<c0106ebc>] do_IRQ+0x10c/0x130
[<c01049c8>] common_interrupt+0x18/0x20
[<c011007b>] setup_pit_timer+0x2b/0x60
[<c0124d23>] sigprocmask+0x3/0xe0
[<c01c63f3>] nfsd+0x1f3/0x3c0
[<c01c6200>] nfsd+0x0/0x3c0
[<c010207d>] kernel_thread_helper+0x5/0x18
nfsd: page allocation failure. order:0, mode:0x20
[<c0139227>] __alloc_pages+0x247/0x3b0
[<c01393a8>] __get_free_pages+0x18/0x40
[<c013ca2f>] kmem_getpages+0x1f/0xc0
[<c013d770>] cache_grow+0xc0/0x1a0
[<c013da1b>] cache_alloc_refill+0x1cb/0x210
[<c013de81>] __kmalloc+0x71/0x80
[<c036f8f3>] alloc_skb+0x53/0x100
[<c031fe88>] e1000_alloc_rx_buffers+0x48/0xf0
[<c031fb8e>] e1000_clean_rx_irq+0x18e/0x440
[<c01049c8>] common_interrupt+0x18/0x20
[<c031f76b>] e1000_clean+0x5b/0x100
[<c0375f7a>] net_rx_action+0x6a/0xf0
[<c011daa1>] __do_softirq+0x41/0x90
[<c011db17>] do_softirq+0x27/0x30
[<c0106ebc>] do_IRQ+0x10c/0x130
[<c01049c8>] common_interrupt+0x18/0x20
[<c011007b>] setup_pit_timer+0x2b/0x60
[<c0124d23>] sigprocmask+0x3/0xe0
[<c01c63f3>] nfsd+0x1f3/0x3c0
[<c01c6200>] nfsd+0x0/0x3c0
[<c010207d>] kernel_thread_helper+0x5/0x18
nfsd: page allocation failure. order:0, mode:0x20
[<c0139227>] __alloc_pages+0x247/0x3b0
[<c01393a8>] __get_free_pages+0x18/0x40
[<c013ca2f>] kmem_getpages+0x1f/0xc0
[<c013d770>] cache_grow+0xc0/0x1a0
[<c013da1b>] cache_alloc_refill+0x1cb/0x210
[<c013de81>] __kmalloc+0x71/0x80
[<c036f8f3>] alloc_skb+0x53/0x100
[<c032c184>] boomerang_rx+0x254/0x490
[<c032b869>] boomerang_interrupt+0xb9/0x430
[<c0339798>] ide_intr+0x188/0x1e0
[<c01069fd>] handle_IRQ_event+0x3d/0x80
[<c0106e3f>] do_IRQ+0x8f/0x130
[<c01049c8>] common_interrupt+0x18/0x20
[<c013de66>] __kmalloc+0x56/0x80
[<c036f8f3>] alloc_skb+0x53/0x100
[<c031fe88>] e1000_alloc_rx_buffers+0x48/0xf0
[<c031fb8e>] e1000_clean_rx_irq+0x18e/0x440
[<c01049c8>] common_interrupt+0x18/0x20
[<c031f76b>] e1000_clean+0x5b/0x100
[<c0375f7a>] net_rx_action+0x6a/0xf0
[<c011daa1>] __do_softirq+0x41/0x90
[<c011db17>] do_softirq+0x27/0x30
[<c0106ebc>] do_IRQ+0x10c/0x130
[<c01049c8>] common_interrupt+0x18/0x20
[<c011007b>] setup_pit_timer+0x2b/0x60
[<c0124d23>] sigprocmask+0x3/0xe0
[<c01c63f3>] nfsd+0x1f3/0x3c0
[<c01c6200>] nfsd+0x0/0x3c0
[<c010207d>] kernel_thread_helper+0x5/0x18
eth2: memory shortage
nfsd: page allocation failure. order:0, mode:0x20
[<c0139227>] __alloc_pages+0x247/0x3b0
[<c01393a8>] __get_free_pages+0x18/0x40
[<c013ca2f>] kmem_getpages+0x1f/0xc0
[<c013d770>] cache_grow+0xc0/0x1a0
[<c013da1b>] cache_alloc_refill+0x1cb/0x210
[<c013de81>] __kmalloc+0x71/0x80
[<c036f8f3>] alloc_skb+0x53/0x100
[<c032c184>] boomerang_rx+0x254/0x490
[<c032b869>] boomerang_interrupt+0xb9/0x430
[<c0339798>] ide_intr+0x188/0x1e0
[<c01069fd>] handle_IRQ_event+0x3d/0x80
[<c0106e3f>] do_IRQ+0x8f/0x130
[<c01049c8>] common_interrupt+0x18/0x20
[<c013de66>] __kmalloc+0x56/0x80
[<c036f8f3>] alloc_skb+0x53/0x100
[<c031fe88>] e1000_alloc_rx_buffers+0x48/0xf0
[<c031fb8e>] e1000_clean_rx_irq+0x18e/0x440
[<c01049c8>] common_interrupt+0x18/0x20
[<c031f76b>] e1000_clean+0x5b/0x100
[<c0375f7a>] net_rx_action+0x6a/0xf0
[<c011daa1>] __do_softirq+0x41/0x90
[<c011db17>] do_softirq+0x27/0x30
[<c0106ebc>] do_IRQ+0x10c/0x130
[<c01049c8>] common_interrupt+0x18/0x20
[<c011007b>] setup_pit_timer+0x2b/0x60
[<c0124d23>] sigprocmask+0x3/0xe0
[<c01c63f3>] nfsd+0x1f3/0x3c0
[<c01c6200>] nfsd+0x0/0x3c0
[<c010207d>] kernel_thread_helper+0x5/0x18
nfsd: page allocation failure. order:0, mode:0x20
[<c0139227>] __alloc_pages+0x247/0x3b0
[<c01393a8>] __get_free_pages+0x18/0x40
[<c013ca2f>] kmem_getpages+0x1f/0xc0
[<c013d770>] cache_grow+0xc0/0x1a0
[<c013da1b>] cache_alloc_refill+0x1cb/0x210
[<c013de81>] __kmalloc+0x71/0x80
[<c036f8f3>] alloc_skb+0x53/0x100
[<c032c184>] boomerang_rx+0x254/0x490
[<c032b869>] boomerang_interrupt+0xb9/0x430
[<c0339798>] ide_intr+0x188/0x1e0
[<c01069fd>] handle_IRQ_event+0x3d/0x80
[<c0106e3f>] do_IRQ+0x8f/0x130
[<c01049c8>] common_interrupt+0x18/0x20
[<c013de66>] __kmalloc+0x56/0x80
[<c036f8f3>] alloc_skb+0x53/0x100
[<c031fe88>] e1000_alloc_rx_buffers+0x48/0xf0
[<c031fb8e>] e1000_clean_rx_irq+0x18e/0x440
[<c01049c8>] common_interrupt+0x18/0x20
[<c031f76b>] e1000_clean+0x5b/0x100
[<c0375f7a>] net_rx_action+0x6a/0xf0
[<c011daa1>] __do_softirq+0x41/0x90
[<c011db17>] do_softirq+0x27/0x30
[<c0106ebc>] do_IRQ+0x10c/0x130
[<c01049c8>] common_interrupt+0x18/0x20
[<c011007b>] setup_pit_timer+0x2b/0x60
[<c0124d23>] sigprocmask+0x3/0xe0
[<c01c63f3>] nfsd+0x1f3/0x3c0
[<c01c6200>] nfsd+0x0/0x3c0
[<c010207d>] kernel_thread_helper+0x5/0x18
nfsd: page allocation failure. order:0, mode:0x20
[<c0139227>] __alloc_pages+0x247/0x3b0
[<c01393a8>] __get_free_pages+0x18/0x40
[<c013ca2f>] kmem_getpages+0x1f/0xc0
[<c013d770>] cache_grow+0xc0/0x1a0
[<c013da1b>] cache_alloc_refill+0x1cb/0x210
[<c013de81>] __kmalloc+0x71/0x80
[<c036f8f3>] alloc_skb+0x53/0x100
[<c031fe88>] e1000_alloc_rx_buffers+0x48/0xf0
[<c031fb8e>] e1000_clean_rx_irq+0x18e/0x440
[<c031f76b>] e1000_clean+0x5b/0x100
[<c0375f7a>] net_rx_action+0x6a/0xf0
[<c011daa1>] __do_softirq+0x41/0x90
[<c011db17>] do_softirq+0x27/0x30
[<c0106ebc>] do_IRQ+0x10c/0x130
[<c01049c8>] common_interrupt+0x18/0x20
[<c011007b>] setup_pit_timer+0x2b/0x60
[<c0124d23>] sigprocmask+0x3/0xe0
[<c01c63f3>] nfsd+0x1f3/0x3c0
[<c01c6200>] nfsd+0x0/0x3c0
[<c010207d>] kernel_thread_helper+0x5/0x18
lftp: page allocation failure. order:0, mode:0x20
[<c0139227>] __alloc_pages+0x247/0x3b0
[<c01393a8>] __get_free_pages+0x18/0x40
[<c013ca2f>] kmem_getpages+0x1f/0xc0
[<c013d770>] cache_grow+0xc0/0x1a0
[<c013da1b>] cache_alloc_refill+0x1cb/0x210
[<c013de81>] __kmalloc+0x71/0x80
[<c036f8f3>] alloc_skb+0x53/0x100
[<c031fe88>] e1000_alloc_rx_buffers+0x48/0xf0
[<c031fb8e>] e1000_clean_rx_irq+0x18e/0x440
[<c031f76b>] e1000_clean+0x5b/0x100
[<c0375f7a>] net_rx_action+0x6a/0xf0
[<c011daa1>] __do_softirq+0x41/0x90
[<c011db17>] do_softirq+0x27/0x30
[<c0106ebc>] do_IRQ+0x10c/0x130
[<c01049c8>] common_interrupt+0x18/0x20
kswapd0: page allocation failure. order:0, mode:0x20
[<c0139227>] __alloc_pages+0x247/0x3b0
[<c01393a8>] __get_free_pages+0x18/0x40
[<c013ca2f>] kmem_getpages+0x1f/0xc0
[<c013d770>] cache_grow+0xc0/0x1a0
[<c013da1b>] cache_alloc_refill+0x1cb/0x210
[<c013de81>] __kmalloc+0x71/0x80
[<c036f8f3>] alloc_skb+0x53/0x100
[<c031fe88>] e1000_alloc_rx_buffers+0x48/0xf0
[<c031fb8e>] e1000_clean_rx_irq+0x18e/0x440
[<c038abd0>] ip_rcv_finish+0x0/0x2d0
[<c031f76b>] e1000_clean+0x5b/0x100
[<c0375f7a>] net_rx_action+0x6a/0xf0
[<c011daa1>] __do_softirq+0x41/0x90
[<c011db17>] do_softirq+0x27/0x30
[<c0106ebc>] do_IRQ+0x10c/0x130
[<c01049c8>] common_interrupt+0x18/0x20
[<c030941a>] __make_request+0x4ea/0x560
[<c0309563>] generic_make_request+0xd3/0x190
[<c0137b85>] mempool_alloc+0x85/0x190
[<c0116c60>] autoremove_wake_function+0x0/0x60
[<c0309671>] submit_bio+0x51/0xf0
[<c01049c8>] common_interrupt+0x18/0x20
[<c015915b>] bio_alloc+0x17b/0x1f0
[<c0155ab0>] end_buffer_async_write+0x0/0x110
[<c0158925>] submit_bh+0xe5/0x140
[<c0289fa5>] xfs_submit_page+0xb5/0x100
[<c028a16a>] xfs_convert_page+0x17a/0x2b0
[<c01345b7>] find_trylock_page+0x27/0x60
[<c0289aef>] xfs_probe_delalloc_page+0x1f/0xa0
[<c028a30f>] xfs_cluster_write+0x6f/0x90
[<c028a633>] xfs_page_state_convert+0x303/0x6e0
[<c028b0f4>] linvfs_writepage+0x74/0x130
[<c013fe89>] pageout+0xb9/0x100
[<c0134290>] wake_up_page+0x10/0x30
[<c0140165>] shrink_list+0x295/0x4b0
[<c01404e3>] shrink_cache+0x163/0x380
[<c01150cd>] wake_up_process+0x1d/0x20
[<c028d1f5>] pagebuf_daemon_wakeup+0x15/0x20
[<c013fbc8>] shrink_slab+0x98/0x1a0
[<c0140c82>] shrink_zone+0xa2/0xd0
[<c01410c0>] balance_pgdat+0x1e0/0x2c0
[<c014125f>] kswapd+0xbf/0xe0
[<c0116c60>] autoremove_wake_function+0x0/0x60
[<c0103f32>] ret_from_fork+0x6/0x14
[<c0116c60>] autoremove_wake_function+0x0/0x60
[<c01411a0>] kswapd+0x0/0xe0
[<c010207d>] kernel_thread_helper+0x5/0x18
kswapd0: page allocation failure. order:0, mode:0x20
[<c0139227>] __alloc_pages+0x247/0x3b0
[<c01393a8>] __get_free_pages+0x18/0x40
[<c013ca2f>] kmem_getpages+0x1f/0xc0
[<c013d770>] cache_grow+0xc0/0x1a0
[<c013da1b>] cache_alloc_refill+0x1cb/0x210
[<c013de81>] __kmalloc+0x71/0x80
[<c036f8f3>] alloc_skb+0x53/0x100
[<c032c184>] boomerang_rx+0x254/0x490
[<c032b869>] boomerang_interrupt+0xb9/0x430
[<c0339798>] ide_intr+0x188/0x1e0
[<c01069fd>] handle_IRQ_event+0x3d/0x80
[<c0106e3f>] do_IRQ+0x8f/0x130
[<c01049c8>] common_interrupt+0x18/0x20
[<c013de66>] __kmalloc+0x56/0x80
[<c036f8f3>] alloc_skb+0x53/0x100
[<c031fe88>] e1000_alloc_rx_buffers+0x48/0xf0
[<c031fb8e>] e1000_clean_rx_irq+0x18e/0x440
[<c038abd0>] ip_rcv_finish+0x0/0x2d0
[<c031f76b>] e1000_clean+0x5b/0x100
[<c0375f7a>] net_rx_action+0x6a/0xf0
[<c011daa1>] __do_softirq+0x41/0x90
[<c011db17>] do_softirq+0x27/0x30
[<c0106ebc>] do_IRQ+0x10c/0x130
[<c01049c8>] common_interrupt+0x18/0x20
[<c030941a>] __make_request+0x4ea/0x560
[<c0309563>] generic_make_request+0xd3/0x190
[<c0137b85>] mempool_alloc+0x85/0x190
[<c0116c60>] autoremove_wake_function+0x0/0x60
[<c0309671>] submit_bio+0x51/0xf0
[<c01049c8>] common_interrupt+0x18/0x20
[<c015915b>] bio_alloc+0x17b/0x1f0
[<c0155ab0>] end_buffer_async_write+0x0/0x110
[<c0158925>] submit_bh+0xe5/0x140
[<c0289fa5>] xfs_submit_page+0xb5/0x100
[<c028a16a>] xfs_convert_page+0x17a/0x2b0
[<c01345b7>] find_trylock_page+0x27/0x60
[<c0289aef>] xfs_probe_delalloc_page+0x1f/0xa0
[<c028a30f>] xfs_cluster_write+0x6f/0x90
[<c028a633>] xfs_page_state_convert+0x303/0x6e0
[<c028b0f4>] linvfs_writepage+0x74/0x130
[<c013fe89>] pageout+0xb9/0x100
[<c0134290>] wake_up_page+0x10/0x30
[<c0140165>] shrink_list+0x295/0x4b0
[<c01404e3>] shrink_cache+0x163/0x380
[<c01150cd>] wake_up_process+0x1d/0x20
[<c028d1f5>] pagebuf_daemon_wakeup+0x15/0x20
[<c013fbc8>] shrink_slab+0x98/0x1a0
[<c0140c82>] shrink_zone+0xa2/0xd0
[<c01410c0>] balance_pgdat+0x1e0/0x2c0
[<c014125f>] kswapd+0xbf/0xe0
[<c0116c60>] autoremove_wake_function+0x0/0x60
[<c0103f32>] ret_from_fork+0x6/0x14
[<c0116c60>] autoremove_wake_function+0x0/0x60
[<c01411a0>] kswapd+0x0/0xe0
[<c010207d>] kernel_thread_helper+0x5/0x18
eth2: memory shortage
kswapd0: page allocation failure. order:0, mode:0x20
[<c0139227>] __alloc_pages+0x247/0x3b0
[<c01393a8>] __get_free_pages+0x18/0x40
[<c013ca2f>] kmem_getpages+0x1f/0xc0
[<c013d770>] cache_grow+0xc0/0x1a0
[<c013da1b>] cache_alloc_refill+0x1cb/0x210
[<c013de81>] __kmalloc+0x71/0x80
[<c036f8f3>] alloc_skb+0x53/0x100
[<c032c184>] boomerang_rx+0x254/0x490
[<c032b869>] boomerang_interrupt+0xb9/0x430
[<c0339798>] ide_intr+0x188/0x1e0
[<c01069fd>] handle_IRQ_event+0x3d/0x80
[<c0106e3f>] do_IRQ+0x8f/0x130
[<c01049c8>] common_interrupt+0x18/0x20
[<c013de66>] __kmalloc+0x56/0x80
[<c036f8f3>] alloc_skb+0x53/0x100
[<c031fe88>] e1000_alloc_rx_buffers+0x48/0xf0
[<c031fb8e>] e1000_clean_rx_irq+0x18e/0x440
[<c038abd0>] ip_rcv_finish+0x0/0x2d0
[<c031f76b>] e1000_clean+0x5b/0x100
[<c0375f7a>] net_rx_action+0x6a/0xf0
[<c011daa1>] __do_softirq+0x41/0x90
[<c011db17>] do_softirq+0x27/0x30
[<c0106ebc>] do_IRQ+0x10c/0x130
[<c01049c8>] common_interrupt+0x18/0x20
[<c030941a>] __make_request+0x4ea/0x560
[<c0309563>] generic_make_request+0xd3/0x190
[<c0137b85>] mempool_alloc+0x85/0x190
[<c0116c60>] autoremove_wake_function+0x0/0x60
[<c0309671>] submit_bio+0x51/0xf0
[<c01049c8>] common_interrupt+0x18/0x20
[<c015915b>] bio_alloc+0x17b/0x1f0
[<c0155ab0>] end_buffer_async_write+0x0/0x110
[<c0158925>] submit_bh+0xe5/0x140
[<c0289fa5>] xfs_submit_page+0xb5/0x100
[<c028a16a>] xfs_convert_page+0x17a/0x2b0
[<c01345b7>] find_trylock_page+0x27/0x60
[<c0289aef>] xfs_probe_delalloc_page+0x1f/0xa0
[<c028a30f>] xfs_cluster_write+0x6f/0x90
[<c028a633>] xfs_page_state_convert+0x303/0x6e0
[<c028b0f4>] linvfs_writepage+0x74/0x130
[<c013fe89>] pageout+0xb9/0x100
[<c0134290>] wake_up_page+0x10/0x30
[<c0140165>] shrink_list+0x295/0x4b0
[<c01404e3>] shrink_cache+0x163/0x380
[<c01150cd>] wake_up_process+0x1d/0x20
[<c028d1f5>] pagebuf_daemon_wakeup+0x15/0x20
[<c013fbc8>] shrink_slab+0x98/0x1a0
[<c0140c82>] shrink_zone+0xa2/0xd0
[<c01410c0>] balance_pgdat+0x1e0/0x2c0
[<c014125f>] kswapd+0xbf/0xe0
[<c0116c60>] autoremove_wake_function+0x0/0x60
[<c0103f32>] ret_from_fork+0x6/0x14
[<c0116c60>] autoremove_wake_function+0x0/0x60
[<c01411a0>] kswapd+0x0/0xe0
[<c010207d>] kernel_thread_helper+0x5/0x18
On Mon, 25 Oct 2004, Nick Piggin wrote:
> Justin Piszcz wrote:
>> I guess people who get this should just stick with 2.6.8.1?
>>
>
> Does it cause any noticable problems? If not, then stay with
> 2.6.9.
>
> However, it would be nice to get to the bottom of it. It might
> just be happening by chance on 2.6.9 but not 2.6.8.1 though...
>
> Anyway, how often are you getting the messages? How many
> ethernet cards in the system?
>
> Can you run a kernel with sysrq support, and do `SysRq+M`
> (close to when the allocation failure happens if possible, but
> otherwise on a normally running system after it has been up
> for a while). Then send in the dmesg.
>
> Thanks,
> Nick
>
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Kernel 2.6.9 Multiple Page Allocation Failures (Part 2)
2004-10-27 21:58 ` Andrew Morton
@ 2004-10-27 21:55 ` Justin Piszcz
2004-10-27 22:19 ` Francois Romieu
0 siblings, 1 reply; 12+ messages in thread
From: Justin Piszcz @ 2004-10-27 21:55 UTC (permalink / raw)
To: Andrew Morton; +Cc: nickpiggin, linux-kernel
Ok, thanks, one last question,
I do not explicitly set ethtool* tso, however I use dhcpcd on this
interface, does that set TSO on the interface? I have never used TSO (that
I am aware of) and I am wondering if it is something else?
On Wed, 27 Oct 2004, Andrew Morton wrote:
> Justin Piszcz <jpiszcz@lucidpixels.com> wrote:
>>
>> swapper: page allocation failure. order:0, mode:0x20
>> [<c0139227>] __alloc_pages+0x247/0x3b0
>> [<c02d9471>] add_interrupt_randomness+0x31/0x40
>> [<c01393a8>] __get_free_pages+0x18/0x40
>> [<c013ca2f>] kmem_getpages+0x1f/0xc0
>> [<c013d770>] cache_grow+0xc0/0x1a0
>> [<c013da1b>] cache_alloc_refill+0x1cb/0x210
>> [<c013de81>] __kmalloc+0x71/0x80
>> [<c036f8f3>] alloc_skb+0x53/0x100
>> [<c031fe88>] e1000_alloc_rx_buffers+0x48/0xf0
>> [<c031fb8e>] e1000_clean_rx_irq+0x18e/0x440
>> [<c031f76b>] e1000_clean+0x5b/0x100
>> [<c0375f7a>] net_rx_action+0x6a/0xf0
>> [<c011daa1>] __do_softirq+0x41/0x90
>> [<c011db17>] do_softirq+0x27/0x30
>> [<c0106ebc>] do_IRQ+0x10c/0x130
>
> This should be harmless - networking will recover. The TSO fix was
> merged a week or so ago.
>
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Kernel 2.6.9 Multiple Page Allocation Failures (Part 2)
2004-10-27 21:40 ` Kernel 2.6.9 Multiple Page Allocation Failures (Part 2) Justin Piszcz
@ 2004-10-27 21:58 ` Andrew Morton
2004-10-27 21:55 ` Justin Piszcz
2004-10-28 0:33 ` Lee Revell
1 sibling, 1 reply; 12+ messages in thread
From: Andrew Morton @ 2004-10-27 21:58 UTC (permalink / raw)
To: Justin Piszcz; +Cc: nickpiggin, linux-kernel
Justin Piszcz <jpiszcz@lucidpixels.com> wrote:
>
> swapper: page allocation failure. order:0, mode:0x20
> [<c0139227>] __alloc_pages+0x247/0x3b0
> [<c02d9471>] add_interrupt_randomness+0x31/0x40
> [<c01393a8>] __get_free_pages+0x18/0x40
> [<c013ca2f>] kmem_getpages+0x1f/0xc0
> [<c013d770>] cache_grow+0xc0/0x1a0
> [<c013da1b>] cache_alloc_refill+0x1cb/0x210
> [<c013de81>] __kmalloc+0x71/0x80
> [<c036f8f3>] alloc_skb+0x53/0x100
> [<c031fe88>] e1000_alloc_rx_buffers+0x48/0xf0
> [<c031fb8e>] e1000_clean_rx_irq+0x18e/0x440
> [<c031f76b>] e1000_clean+0x5b/0x100
> [<c0375f7a>] net_rx_action+0x6a/0xf0
> [<c011daa1>] __do_softirq+0x41/0x90
> [<c011db17>] do_softirq+0x27/0x30
> [<c0106ebc>] do_IRQ+0x10c/0x130
This should be harmless - networking will recover. The TSO fix was
merged a week or so ago.
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Kernel 2.6.9 Multiple Page Allocation Failures (Part 2)
2004-10-27 21:55 ` Justin Piszcz
@ 2004-10-27 22:19 ` Francois Romieu
2004-10-27 22:23 ` Justin Piszcz
0 siblings, 1 reply; 12+ messages in thread
From: Francois Romieu @ 2004-10-27 22:19 UTC (permalink / raw)
To: Justin Piszcz; +Cc: Andrew Morton, nickpiggin, linux-kernel
Justin Piszcz <jpiszcz@lucidpixels.com> :
[...]
> I do not explicitly set ethtool* tso, however I use dhcpcd on this
> interface, does that set TSO on the interface? I have never used TSO (that
Hint: ethtool -k ethX
--
Ueimor
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Kernel 2.6.9 Multiple Page Allocation Failures (Part 2)
2004-10-27 22:19 ` Francois Romieu
@ 2004-10-27 22:23 ` Justin Piszcz
0 siblings, 0 replies; 12+ messages in thread
From: Justin Piszcz @ 2004-10-27 22:23 UTC (permalink / raw)
To: Francois Romieu; +Cc: Andrew Morton, nickpiggin, linux-kernel
Ah, you are correct:
root@p500:~# ethtool -k eth0
Offload parameters for eth0:
rx-checksumming: on
tx-checksumming: on
scatter-gather: on
tcp segmentation offload: on
root@p500:~# ethtool -k eth1
Offload parameters for eth1:
Cannot get device rx csum settings: Operation not supported
Cannot get device tx csum settings: Operation not supported
Cannot get device scatter-gather settings: Operation not supported
Cannot get device tcp segmentation offload settings: Operation not
supported
no offload info available
root@p500:~# ethtool -k eth2
Offload parameters for eth2:
Cannot get device rx csum settings: Operation not supported
Cannot get device tx csum settings: Operation not supported
Cannot get device scatter-gather settings: Operation not supported
Cannot get device tcp segmentation offload settings: Operation not
supported
no offload info available
root@p500:~# ethtool -k eth3
Offload parameters for eth3:
Cannot get device rx csum settings: Operation not supported
Cannot get device tx csum settings: Operation not supported
Cannot get device scatter-gather settings: Operation not supported
Cannot get device tcp segmentation offload settings: Operation not
supported
no offload info available
root@p500:~#
On Thu, 28 Oct 2004, Francois Romieu wrote:
> Justin Piszcz <jpiszcz@lucidpixels.com> :
> [...]
>> I do not explicitly set ethtool* tso, however I use dhcpcd on this
>> interface, does that set TSO on the interface? I have never used TSO (that
>
> Hint: ethtool -k ethX
>
> --
> Ueimor
>
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Kernel 2.6.9 Multiple Page Allocation Failures (Part 2)
2004-10-27 21:40 ` Kernel 2.6.9 Multiple Page Allocation Failures (Part 2) Justin Piszcz
2004-10-27 21:58 ` Andrew Morton
@ 2004-10-28 0:33 ` Lee Revell
1 sibling, 0 replies; 12+ messages in thread
From: Lee Revell @ 2004-10-28 0:33 UTC (permalink / raw)
To: Justin Piszcz; +Cc: Nick Piggin, linux-kernel, akpm
On Wed, 2004-10-27 at 17:40 -0400, Justin Piszcz wrote:
> Is there any chance Linus will freeze 2.6 and make the current development
> tree 2.7? It seems like ever since around 2.6.8 things have been getting
> progressively worse (page allocation failures/nvidia
> breakage/XFS-oops-when-copying-over-nfs-when-the-file-is-being-written-to)?
This not the kernel's problem when nvidia breaks. The kernel developers
make NO EFFORT to support binary only modules! Please, talk to nvidia
if this is a problem for you.
Lee
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Kernel 2.6.9 Multiple Page Allocation Failures (Part 2)
2004-11-04 12:17 ` Marcelo Tosatti
@ 2004-11-04 18:18 ` Stefan Schmidt
0 siblings, 0 replies; 12+ messages in thread
From: Stefan Schmidt @ 2004-11-04 18:18 UTC (permalink / raw)
To: Marcelo Tosatti; +Cc: linux-kernel
On Thu, Nov 04, 2004 at 10:17:22AM -0200, Marcelo Tosatti wrote:
> What is min_free_kbytes default on your machine?
I think it was 768, definitely around 700-800.
2.6.9 said:
Dentry cache hash table entries: 131072 (order: 7, 524288 bytes)
Inode-cache hash table entries: 65536 (order: 6, 262144 bytes)
Memory: 4058200k/4095936k available (2005k kernel code, 36816k reserved, 995k
data, 196k init, 3178432k highmem)
> > I tried the following kernels: 2.6.9-mm1, 2.6.10-rc1-bk12, 2.6.9-rc3-bk6,
> > 2.6.9-rc3-bk5 all of which froze at some point presenting me only with the
> > above page allocation failure. (no more sysrq)
> This should be harmless as Andrew said - it would be helpful if you could
> plug a serial cable to the box - this last oops on the picture doesnt say
> much.
Well right now the machine is running 2.4.28-rc1 with the 3w-9nnn patch by
Adam Radford from this list and i would like to see it run stable for about a
day before i give 2.6 another try. I think i'll have a terminal server hooked
up by then.
> How intense is the network traffic you're generating?
I was around 60-80 mbit/s each direction at i think 16k interrupts/s.
With 2.4.28-rc1 this is currently at 180mbit/s 27kpps up, 116mbit/s 24kpps down
still swapping a bit but no kernel messages on this, just around 1.7 rx
errors/s.
> 2.6.7 was stable under the same load?
No, sorry to give you this impression, 2.6.7 is just what some of my collegues
and i consider the more stable 2.6 kernel under heavy i/o load.
> Something is definately screwed, and there are quite an amount of
> similar reports.
Can i tell people its ok to see nf_hook_slow in the stack trace as it's
vm-related? A collegue was quite bluffed when i showed him. ;)
> XFS also seems to be very memory hungry...
I have 8 XFS-Filesystems in use here with several thousand files from some k to
your 'usual' 4GB DVD-image. XFS built as a module at first and then inline but
that did not change anything off course. 2x200 + 6x250GB that is.
Stefan
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Kernel 2.6.9 Multiple Page Allocation Failures (Part 2)
2004-11-03 22:24 Stefan Schmidt
@ 2004-11-04 12:17 ` Marcelo Tosatti
2004-11-04 18:18 ` Stefan Schmidt
0 siblings, 1 reply; 12+ messages in thread
From: Marcelo Tosatti @ 2004-11-04 12:17 UTC (permalink / raw)
To: Stefan Schmidt; +Cc: linux-kernel
On Wed, Nov 03, 2004 at 11:24:48PM +0100, Stefan Schmidt wrote:
> Andrew Morton <akpm@osdl.org> wrote:
> > > swapper: page allocation failure. order:0, mode:0x20
> > This should be harmless - networking will recover. The TSO fix was
> > merged a week or so ago.
> Ahem,
> i don't think its all that harmless:
> My machine got hundreds of the same in a row even after raising
> vm.min_free_kbytes to 8192 and playing with vm.* in one direction and then in
> the other one.
What is min_free_kbytes default on your machine?
> Essentially the application took 2.2GB and cache was around
> 1.7GB of 4GB available memory. The machine swapping heavily at ~3m intervals
> and setting swappiness to zero did not help.
> I tried the following kernels: 2.6.9-mm1, 2.6.10-rc1-bk12, 2.6.9-rc3-bk6,
> 2.6.9-rc3-bk5 all of which froze at some point presenting me only with the
> above page allocation failure. (no more sysrq)
> 2.6.9 gave me this in the end: http://zaphods.net/~zaphodb/recursive_die.png
This should be harmless as Andrew said - it would be helpful if you could
plug a serial cable to the box - this last oops on the picture doesnt say
much.
How intense is the network traffic you're generating?
> Which was the point i decided to give 2.4 a try and guess what: This is
> running faster and at higher packet rates than the 2.6 kernels achived and
> is stable for the last three hours which at this load is more than the
> abovementioned 2.6 kernels managed.
> I got the impression there is another 2.6 vm-instability which made it in the
> kernel after 2.6.7.
2.6.7 was stable under the same load?
2.4 keeps a higher amount of free pages around, but you should be able
to tune that with vm.min_free_kbytes...
For some reason the VM is not being able to keep enough pages free.
But then it shouldnt matter - if the tg3 driver can't allocate memory
it should drop the packet and not lockup.
Something is definately screwed, and there are quite an amount of
similar reports.
XFS also seems to be very memory hungry...
> The machine is a Dual Opteron on Tyan K8S 4GB ECC-RAM running debian/unstable
> (yes thats 32-bit only) from a 3ware 9000 Controller (SATA) - 8 250GB Disks
> configured as single disks, no hw raid. Root-FS resides on SoftwareRAID,
> Application uses xfs filesystems on the disks.
>
> I attached some vmstat and slapinfo output during the swapping-phase. Dont
> remember the exact kernel here but it was 2.6. (yes, the hostname is kernel.
> ;)
>
> Stefan
> Nov 2 17:12:26 kernel kernel: swapper: page allocation failure. order:0, mode:0x20
> Nov 2 17:12:26 kernel kernel: [__alloc_pages+417/800] __alloc_pages+0x1a1/0x320
> Nov 2 17:12:26 kernel kernel: [__get_free_pages+24/48] __get_free_pages+0x18/0x30
> Nov 2 17:12:26 kernel kernel: [kmem_getpages+24/192] kmem_getpages+0x18/0xc0
> Nov 2 17:12:26 kernel kernel: [cache_grow+157/304] cache_grow+0x9d/0x130
> Nov 2 17:12:26 kernel kernel: [nf_hook_slow+178/240] nf_hook_slow+0xb2/0xf0
> Nov 2 17:12:26 kernel kernel: [cache_alloc_refill+363/544] cache_alloc_refill+0x16b/0x220
> Nov 2 17:12:26 kernel kernel: [__kmalloc+122/144] __kmalloc+0x7a/0x90
> Nov 2 17:12:26 kernel kernel: [alloc_skb+50/208] alloc_skb+0x32/0xd0
> Nov 2 17:12:26 kernel kernel: [pg0+945331568/1068909568] tg3_alloc_rx_skb+0x70/0x130 [tg3]
> Nov 2 17:12:26 kernel kernel: [pg0+945332742/1068909568] tg3_rx+0x206/0x3b0 [tg3]
> Nov 2 17:12:26 kernel kernel: [pg0+945333307/1068909568] tg3_poll+0x8b/0x130 [tg3]
> Nov 2 17:12:26 kernel kernel: [net_rx_action+112/240] net_rx_action+0x70/0xf0
> Nov 2 17:12:26 kernel kernel: [__do_softirq+184/208] __do_softirq+0xb8/0xd0
> Nov 2 17:12:26 kernel kernel: [do_softirq+45/48] do_softirq+0x2d/0x30
> Nov 2 17:12:26 kernel kernel: [do_IRQ+43/64] do_IRQ+0x2b/0x40
> Nov 2 17:12:26 kernel kernel: [common_interrupt+24/32] common_interrupt+0x18/0x20
> Nov 2 17:12:26 kernel kernel: [default_idle+0/64] default_idle+0x0/0x40
> Nov 2 17:12:26 kernel kernel: [default_idle+44/64] default_idle+0x2c/0x40
> Nov 2 17:12:26 kernel kernel: [cpu_idle+51/64] cpu_idle+0x33/0x40
> Nov 2 17:12:26 kernel kernel: [start_kernel+332/368] start_kernel+0x14c/0x170
> Nov 2 17:12:26 kernel kernel: [unknown_bootoption+0/416] unknown_bootoption+0x0/0x1a0
>
>
> procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
> r b swpd free buff cache si so bi bo in cs us sy id wa
> 1 1 132964 3272 6420 1681888 0 0 1401 7668 15763 691 4 23 40 33
> 1 1 132964 3100 6380 1685256 0 0 1928 11176 16051 705 4 27 39 30
> 0 1 132964 3292 6212 1685288 0 0 1788 7575 16031 720 5 23 51 21
> 0 2 132964 4016 6432 1683368 0 0 1752 9379 16221 652 4 26 46 24
> 0 2 134180 197092 520 1496176 26 175 2473 12866 16302 1630 5 27 21 48
> 1 0 145608 89844 1476 1604432 56 1182 2672 9028 16248 822 4 26 37 32
> 1 0 145608 15148 2016 1675404 20 0 2127 4868 16193 612 4 26 48 22
> 0 1 146816 3152 2432 1688956 171 122 1872 6761 14757 583 2 15 29 53
> 0 2 146816 3972 2736 1693380 298 0 1982 7910 15626 666 3 23 48 27
> 1 0 146816 4028 2796 1689412 170 0 1759 7339 15586 597 2 18 35 45
> 0 1 146816 4136 2984 1690960 127 0 2091 7396 16146 557 3 26 43 27
> 0 3 146816 4320 3460 1685736 86 0 2042 7381 15981 597 3 23 32 41
> 0 1 146816 16820 3556 1672304 73 0 1082 9960 15361 777 2 18 34 46
> 0 1 146816 3348 3704 1686584 40 0 1762 8153 16247 633 4 26 42 28
> 0 2 146816 5700 3912 1683600 26 0 1973 9772 16120 669 4 24 42 30
> 0 0 146816 6096 4084 1674224 32 0 1478 7469 15378 569 2 18 33 47
> 0 3 146816 4284 4148 1680812 15 0 1052 8376 15717 576 3 24 43 30
> 0 1 146816 3572 4412 1684472 29 0 1660 10212 15740 597 3 22 38 37
> 0 0 146816 6220 4576 1680260 24 0 2256 8223 16015 656 4 25 47 25
> 0 1 146816 3448 4436 1683892 11 0 1526 8507 15820 628 4 21 38 37
> 0 0 146816 3152 4552 1687964 10 0 1609 9256 16244 665 4 25 52 18
> 0 0 146816 4212 4660 1683816 5 0 1524 8207 16066 545 3 25 42 30
>
> labinfo - version: 2.1
> # name <active_objs> <num_objs> <objsize> <objperslab> <pagesperslab> : tunables <batchcount> <limit> <sharedfactor> : slabdata <active_slabs> <num_slabs> <sharedavail>
> ip_conntrack_expect 0 0 128 31 1 : tunables 120 60 8 : slabdata 0 0 0
> ip_conntrack 22024 23748 320 12 1 : tunables 54 27 8 : slabdata 1979 1979 85
> fib6_nodes 5 119 32 119 1 : tunables 120 60 8 : slabdata 1 1 0
> ip6_dst_cache 4 15 256 15 1 : tunables 120 60 8 : slabdata 1 1 0
> ndisc_cache 1 20 192 20 1 : tunables 120 60 8 : slabdata 1 1 0
> rawv6_sock 4 11 704 11 2 : tunables 54 27 8 : slabdata 1 1 0
> udpv6_sock 1 6 640 6 1 : tunables 54 27 8 : slabdata 1 1 0
> tcpv6_sock 5 6 1216 3 1 : tunables 24 12 8 : slabdata 2 2 0
> unix_sock 23 27 448 9 1 : tunables 54 27 8 : slabdata 3 3 0
> tcp_tw_bucket 6281 7192 128 31 1 : tunables 120 60 8 : slabdata 232 232 120
> tcp_bind_bucket 6 226 16 226 1 : tunables 120 60 8 : slabdata 1 1 0
> tcp_open_request 214 310 128 31 1 : tunables 120 60 8 : slabdata 10 10 0
> inet_peer_cache 2 61 64 61 1 : tunables 120 60 8 : slabdata 1 1 0
> secpath_cache 0 0 128 31 1 : tunables 120 60 8 : slabdata 0 0 0
> xfrm_dst_cache 0 0 256 15 1 : tunables 120 60 8 : slabdata 0 0 0
> ip_fib_alias 9 226 16 226 1 : tunables 120 60 8 : slabdata 1 1 0
> ip_fib_hash 9 119 32 119 1 : tunables 120 60 8 : slabdata 1 1 0
> ip_dst_cache 11007 13515 256 15 1 : tunables 120 60 8 : slabdata 901 901 0
> arp_cache 1 20 192 20 1 : tunables 120 60 8 : slabdata 1 1 0
> raw_sock 3 7 512 7 1 : tunables 54 27 8 : slabdata 1 1 0
> udp_sock 3 8 512 8 1 : tunables 54 27 8 : slabdata 1 1 0
> tcp_sock 7192 7910 1088 7 2 : tunables 24 12 8 : slabdata 1130 1130 72
> flow_cache 0 0 128 31 1 : tunables 120 60 8 : slabdata 0 0 0
> dm-snapshot-in 128 162 48 81 1 : tunables 120 60 8 : slabdata 2 2 0
> dm-snapshot-ex 0 0 16 226 1 : tunables 120 60 8 : slabdata 0 0 0
> dm-crypt_io 0 0 76 52 1 : tunables 120 60 8 : slabdata 0 0 0
> dm_tio 0 0 16 226 1 : tunables 120 60 8 : slabdata 0 0 0
> dm_io 0 0 16 226 1 : tunables 120 60 8 : slabdata 0 0 0
> scsi_cmd_cache 100 100 384 10 1 : tunables 54 27 8 : slabdata 10 10 27
> cfq_ioc_pool 0 0 24 156 1 : tunables 120 60 8 : slabdata 0 0 0
> cfq_pool 0 0 104 38 1 : tunables 120 60 8 : slabdata 0 0 0
> crq_pool 0 0 52 75 1 : tunables 120 60 8 : slabdata 0 0 0
> deadline_drq 0 0 48 81 1 : tunables 120 60 8 : slabdata 0 0 0
> as_arq 515 845 60 65 1 : tunables 120 60 8 : slabdata 13 13 300
> xfs_chashlist 583 740 20 185 1 : tunables 120 60 8 : slabdata 4 4 0
> xfs_ili 1873 2184 140 28 1 : tunables 120 60 8 : slabdata 78 78 0
> xfs_ifork 0 0 56 70 1 : tunables 120 60 8 : slabdata 0 0 0
> xfs_efi_item 15 15 260 15 1 : tunables 54 27 8 : slabdata 1 1 0
> xfs_efd_item 15 15 260 15 1 : tunables 54 27 8 : slabdata 1 1 0
> xfs_buf_item 639 783 148 27 1 : tunables 120 60 8 : slabdata 29 29 284
> xfs_dabuf 10 226 16 226 1 : tunables 120 60 8 : slabdata 1 1 0
> xfs_da_state 0 0 336 12 1 : tunables 54 27 8 : slabdata 0 0 0
> xfs_trans 260 260 596 13 2 : tunables 54 27 8 : slabdata 20 20 27
> xfs_inode 2212 3135 368 11 1 : tunables 54 27 8 : slabdata 285 285 0
> xfs_btree_cur 120 120 132 30 1 : tunables 120 60 8 : slabdata 4 4 0
> xfs_bmap_free_item 24 290 12 290 1 : tunables 120 60 8 : slabdata 1 1 0
> xfs_buf_t 690 870 256 15 1 : tunables 120 60 8 : slabdata 58 58 224
> linvfs_icache 2003 2673 344 11 1 : tunables 54 27 8 : slabdata 243 243 0
> hugetlbfs_inode_cache 1 12 312 12 1 : tunables 54 27 8 : slabdata 1 1 0
> ext2_inode_cache 0 0 448 9 1 : tunables 54 27 8 : slabdata 0 0 0
> ext2_xattr 0 0 44 88 1 : tunables 120 60 8 : slabdata 0 0 0
> journal_handle 135 135 28 135 1 : tunables 120 60 8 : slabdata 1 1 0
> journal_head 849 2835 48 81 1 : tunables 120 60 8 : slabdata 35 35 60
> revoke_table 2 290 12 290 1 : tunables 120 60 8 : slabdata 1 1 0
> revoke_record 109 226 16 226 1 : tunables 120 60 8 : slabdata 1 1 0
> ext3_inode_cache 414 707 508 7 1 : tunables 54 27 8 : slabdata 101 101 0
> ext3_xattr 0 0 44 88 1 : tunables 120 60 8 : slabdata 0 0 0
> dnotify_cache 0 0 20 185 1 : tunables 120 60 8 : slabdata 0 0 0
> eventpoll_pwq 0 0 36 107 1 : tunables 120 60 8 : slabdata 0 0 0
> eventpoll_epi 0 0 128 31 1 : tunables 120 60 8 : slabdata 0 0 0
> kioctx 0 0 192 20 1 : tunables 120 60 8 : slabdata 0 0 0
> kiocb 0 0 128 31 1 : tunables 120 60 8 : slabdata 0 0 0
> fasync_cache 0 0 16 226 1 : tunables 120 60 8 : slabdata 0 0 0
> shmem_inode_cache 7 9 408 9 1 : tunables 54 27 8 : slabdata 1 1 0
> posix_timers_cache 0 0 104 38 1 : tunables 120 60 8 : slabdata 0 0 0
> uid_cache 3 61 64 61 1 : tunables 120 60 8 : slabdata 1 1 0
> sgpool-128 32 33 2560 3 2 : tunables 24 12 8 : slabdata 11 11 0
> sgpool-64 32 33 1280 3 1 : tunables 24 12 8 : slabdata 11 11 0
> sgpool-32 102 102 640 6 1 : tunables 54 27 8 : slabdata 17 17 27
> sgpool-16 96 96 320 12 1 : tunables 54 27 8 : slabdata 8 8 0
> sgpool-8 200 200 192 20 1 : tunables 120 60 8 : slabdata 10 10 60
> blkdev_ioc 84 135 28 135 1 : tunables 120 60 8 : slabdata 1 1 0
> blkdev_queue 10 20 372 10 1 : tunables 54 27 8 : slabdata 2 2 0
> blkdev_requests 471 840 140 28 1 : tunables 120 60 8 : slabdata 30 30 240
> biovec-(256) 256 256 3072 2 2 : tunables 24 12 8 : slabdata 128 128 0
> biovec-128 256 260 1536 5 2 : tunables 24 12 8 : slabdata 52 52 0
> biovec-64 320 320 768 5 1 : tunables 54 27 8 : slabdata 64 64 0
> biovec-16 350 380 192 20 1 : tunables 120 60 8 : slabdata 19 19 0
> biovec-4 374 427 64 61 1 : tunables 120 60 8 : slabdata 7 7 0
> biovec-1 1456 3164 16 226 1 : tunables 120 60 8 : slabdata 14 14 480
> bio 1418 2196 64 61 1 : tunables 120 60 8 : slabdata 36 36 480
> file_lock_cache 23 43 92 43 1 : tunables 120 60 8 : slabdata 1 1 0
> sock_inode_cache 7064 7830 384 10 1 : tunables 54 27 8 : slabdata 783 783 135
> skbuff_head_cache 14636 23220 192 20 1 : tunables 120 60 8 : slabdata 1161 1161 420
> sock 3 10 384 10 1 : tunables 54 27 8 : slabdata 1 1 0
> proc_inode_cache 321 360 328 12 1 : tunables 54 27 8 : slabdata 30 30 0
> sigqueue 19 27 148 27 1 : tunables 120 60 8 : slabdata 1 1 0
> radix_tree_node 22347 28602 276 14 1 : tunables 54 27 8 : slabdata 2043 2043 27
> bdev_cache 30 45 448 9 1 : tunables 54 27 8 : slabdata 5 5 0
> mnt_cache 26 62 128 31 1 : tunables 120 60 8 : slabdata 2 2 0
> inode_cache 1040 1128 312 12 1 : tunables 54 27 8 : slabdata 94 94 0
> dentry_cache 10067 15400 140 28 1 : tunables 120 60 8 : slabdata 550 550 240
> filp 8189 9100 192 20 1 : tunables 120 60 8 : slabdata 455 455 60
> names_cache 26 26 4096 1 1 : tunables 24 12 8 : slabdata 26 26 0
> idr_layer_cache 112 116 136 29 1 : tunables 120 60 8 : slabdata 4 4 0
> buffer_head 345712 472959 48 81 1 : tunables 120 60 8 : slabdata 5839 5839 480
> mm_struct 91 91 576 7 1 : tunables 54 27 8 : slabdata 13 13 0
> vm_area_struct 1994 2205 88 45 1 : tunables 120 60 8 : slabdata 49 49 0
> fs_cache 86 183 64 61 1 : tunables 120 60 8 : slabdata 3 3 0
> files_cache 87 99 448 9 1 : tunables 54 27 8 : slabdata 11 11 0
> signal_cache 111 165 256 15 1 : tunables 120 60 8 : slabdata 11 11 0
> sighand_cache 104 108 1344 3 1 : tunables 24 12 8 : slabdata 36 36 0
> task_struct 111 111 1280 3 1 : tunables 24 12 8 : slabdata 37 37 0
> anon_vma 635 870 12 290 1 : tunables 120 60 8 : slabdata 3 3 0
> pgd 115 238 32 119 1 : tunables 120 60 8 : slabdata 2 2 0
> pmd 180 180 4096 1 1 : tunables 24 12 8 : slabdata 180 180 0
> size-131072(DMA) 0 0 131072 1 32 : tunables 8 4 0 : slabdata 0 0 0
> size-131072 0 0 131072 1 32 : tunables 8 4 0 : slabdata 0 0 0
> size-65536(DMA) 0 0 65536 1 16 : tunables 8 4 0 : slabdata 0 0 0
> size-65536 19 19 65536 1 16 : tunables 8 4 0 : slabdata 19 19 0
> size-32768(DMA) 0 0 32768 1 8 : tunables 8 4 0 : slabdata 0 0 0
> size-32768 139 139 32768 1 8 : tunables 8 4 0 : slabdata 139 139 0
> size-16384(DMA) 0 0 16384 1 4 : tunables 8 4 0 : slabdata 0 0 0
> size-16384 151 153 16384 1 4 : tunables 8 4 0 : slabdata 151 153 0
> size-8192(DMA) 0 0 8192 1 2 : tunables 8 4 0 : slabdata 0 0 0
> size-8192 365 365 8192 1 2 : tunables 8 4 0 : slabdata 365 365 0
> size-4096(DMA) 0 0 4096 1 1 : tunables 24 12 8 : slabdata 0 0 0
> size-4096 770 770 4096 1 1 : tunables 24 12 8 : slabdata 770 770 48
> size-2048(DMA) 0 0 2048 2 1 : tunables 24 12 8 : slabdata 0 0 0
> size-2048 14460 14556 2048 2 1 : tunables 24 12 8 : slabdata 7278 7278 12
> size-1024(DMA) 0 0 1024 4 1 : tunables 54 27 8 : slabdata 0 0 0
> size-1024 888 888 1024 4 1 : tunables 54 27 8 : slabdata 222 222 54
> size-512(DMA) 0 0 512 8 1 : tunables 54 27 8 : slabdata 0 0 0
> size-512 886 1352 512 8 1 : tunables 54 27 8 : slabdata 169 169 108
> size-256(DMA) 0 0 256 15 1 : tunables 120 60 8 : slabdata 0 0 0
> size-256 270 300 256 15 1 : tunables 120 60 8 : slabdata 20 20 0
> size-192(DMA) 0 0 192 20 1 : tunables 120 60 8 : slabdata 0 0 0
> size-192 156 180 192 20 1 : tunables 120 60 8 : slabdata 9 9 0
> size-128(DMA) 0 0 128 31 1 : tunables 120 60 8 : slabdata 0 0 0
> size-128 1921 2635 128 31 1 : tunables 120 60 8 : slabdata 85 85 0
> size-64(DMA) 0 0 64 61 1 : tunables 120 60 8 : slabdata 0 0 0
> size-64 7293 13603 64 61 1 : tunables 120 60 8 : slabdata 223 223 480
> size-32(DMA) 0 0 32 119 1 : tunables 120 60 8 : slabdata 0 0 0
> size-32 10802 19516 32 119 1 : tunables 120 60 8 : slabdata 164 164 120
> kmem_cache 160 160 192 20 1 : tunables 120 60 8 : slabdata 8 8 0
>
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Kernel 2.6.9 Multiple Page Allocation Failures (Part 2)
@ 2004-11-03 22:24 Stefan Schmidt
2004-11-04 12:17 ` Marcelo Tosatti
0 siblings, 1 reply; 12+ messages in thread
From: Stefan Schmidt @ 2004-11-03 22:24 UTC (permalink / raw)
To: linux-kernel
[-- Attachment #1: Type: text/plain, Size: 1638 bytes --]
Andrew Morton <akpm@osdl.org> wrote:
> > swapper: page allocation failure. order:0, mode:0x20
> This should be harmless - networking will recover. The TSO fix was
> merged a week or so ago.
Ahem,
i don't think its all that harmless:
My machine got hundreds of the same in a row even after raising
vm.min_free_kbytes to 8192 and playing with vm.* in one direction and then in
the other one. Essentially the application took 2.2GB and cache was around
1.7GB of 4GB available memory. The machine swapping heavily at ~3m intervals
and setting swappiness to zero did not help.
I tried the following kernels: 2.6.9-mm1, 2.6.10-rc1-bk12, 2.6.9-rc3-bk6,
2.6.9-rc3-bk5 all of which froze at some point presenting me only with the
above page allocation failure. (no more sysrq)
2.6.9 gave me this in the end: http://zaphods.net/~zaphodb/recursive_die.png
Which was the point i decided to give 2.4 a try and guess what: This is
running faster and at higher packet rates than the 2.6 kernels achived and
is stable for the last three hours which at this load is more than the
abovementioned 2.6 kernels managed.
I got the impression there is another 2.6 vm-instability which made it in the
kernel after 2.6.7.
The machine is a Dual Opteron on Tyan K8S 4GB ECC-RAM running debian/unstable
(yes thats 32-bit only) from a 3ware 9000 Controller (SATA) - 8 250GB Disks
configured as single disks, no hw raid. Root-FS resides on SoftwareRAID,
Application uses xfs filesystems on the disks.
I attached some vmstat and slapinfo output during the swapping-phase. Dont
remember the exact kernel here but it was 2.6. (yes, the hostname is kernel.
;)
Stefan
[-- Attachment #2: kernel.slab --]
[-- Type: text/plain, Size: 18343 bytes --]
Nov 2 17:12:26 kernel kernel: swapper: page allocation failure. order:0, mode:0x20
Nov 2 17:12:26 kernel kernel: [__alloc_pages+417/800] __alloc_pages+0x1a1/0x320
Nov 2 17:12:26 kernel kernel: [__get_free_pages+24/48] __get_free_pages+0x18/0x30
Nov 2 17:12:26 kernel kernel: [kmem_getpages+24/192] kmem_getpages+0x18/0xc0
Nov 2 17:12:26 kernel kernel: [cache_grow+157/304] cache_grow+0x9d/0x130
Nov 2 17:12:26 kernel kernel: [nf_hook_slow+178/240] nf_hook_slow+0xb2/0xf0
Nov 2 17:12:26 kernel kernel: [cache_alloc_refill+363/544] cache_alloc_refill+0x16b/0x220
Nov 2 17:12:26 kernel kernel: [__kmalloc+122/144] __kmalloc+0x7a/0x90
Nov 2 17:12:26 kernel kernel: [alloc_skb+50/208] alloc_skb+0x32/0xd0
Nov 2 17:12:26 kernel kernel: [pg0+945331568/1068909568] tg3_alloc_rx_skb+0x70/0x130 [tg3]
Nov 2 17:12:26 kernel kernel: [pg0+945332742/1068909568] tg3_rx+0x206/0x3b0 [tg3]
Nov 2 17:12:26 kernel kernel: [pg0+945333307/1068909568] tg3_poll+0x8b/0x130 [tg3]
Nov 2 17:12:26 kernel kernel: [net_rx_action+112/240] net_rx_action+0x70/0xf0
Nov 2 17:12:26 kernel kernel: [__do_softirq+184/208] __do_softirq+0xb8/0xd0
Nov 2 17:12:26 kernel kernel: [do_softirq+45/48] do_softirq+0x2d/0x30
Nov 2 17:12:26 kernel kernel: [do_IRQ+43/64] do_IRQ+0x2b/0x40
Nov 2 17:12:26 kernel kernel: [common_interrupt+24/32] common_interrupt+0x18/0x20
Nov 2 17:12:26 kernel kernel: [default_idle+0/64] default_idle+0x0/0x40
Nov 2 17:12:26 kernel kernel: [default_idle+44/64] default_idle+0x2c/0x40
Nov 2 17:12:26 kernel kernel: [cpu_idle+51/64] cpu_idle+0x33/0x40
Nov 2 17:12:26 kernel kernel: [start_kernel+332/368] start_kernel+0x14c/0x170
Nov 2 17:12:26 kernel kernel: [unknown_bootoption+0/416] unknown_bootoption+0x0/0x1a0
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
1 1 132964 3272 6420 1681888 0 0 1401 7668 15763 691 4 23 40 33
1 1 132964 3100 6380 1685256 0 0 1928 11176 16051 705 4 27 39 30
0 1 132964 3292 6212 1685288 0 0 1788 7575 16031 720 5 23 51 21
0 2 132964 4016 6432 1683368 0 0 1752 9379 16221 652 4 26 46 24
0 2 134180 197092 520 1496176 26 175 2473 12866 16302 1630 5 27 21 48
1 0 145608 89844 1476 1604432 56 1182 2672 9028 16248 822 4 26 37 32
1 0 145608 15148 2016 1675404 20 0 2127 4868 16193 612 4 26 48 22
0 1 146816 3152 2432 1688956 171 122 1872 6761 14757 583 2 15 29 53
0 2 146816 3972 2736 1693380 298 0 1982 7910 15626 666 3 23 48 27
1 0 146816 4028 2796 1689412 170 0 1759 7339 15586 597 2 18 35 45
0 1 146816 4136 2984 1690960 127 0 2091 7396 16146 557 3 26 43 27
0 3 146816 4320 3460 1685736 86 0 2042 7381 15981 597 3 23 32 41
0 1 146816 16820 3556 1672304 73 0 1082 9960 15361 777 2 18 34 46
0 1 146816 3348 3704 1686584 40 0 1762 8153 16247 633 4 26 42 28
0 2 146816 5700 3912 1683600 26 0 1973 9772 16120 669 4 24 42 30
0 0 146816 6096 4084 1674224 32 0 1478 7469 15378 569 2 18 33 47
0 3 146816 4284 4148 1680812 15 0 1052 8376 15717 576 3 24 43 30
0 1 146816 3572 4412 1684472 29 0 1660 10212 15740 597 3 22 38 37
0 0 146816 6220 4576 1680260 24 0 2256 8223 16015 656 4 25 47 25
0 1 146816 3448 4436 1683892 11 0 1526 8507 15820 628 4 21 38 37
0 0 146816 3152 4552 1687964 10 0 1609 9256 16244 665 4 25 52 18
0 0 146816 4212 4660 1683816 5 0 1524 8207 16066 545 3 25 42 30
labinfo - version: 2.1
# name <active_objs> <num_objs> <objsize> <objperslab> <pagesperslab> : tunables <batchcount> <limit> <sharedfactor> : slabdata <active_slabs> <num_slabs> <sharedavail>
ip_conntrack_expect 0 0 128 31 1 : tunables 120 60 8 : slabdata 0 0 0
ip_conntrack 22024 23748 320 12 1 : tunables 54 27 8 : slabdata 1979 1979 85
fib6_nodes 5 119 32 119 1 : tunables 120 60 8 : slabdata 1 1 0
ip6_dst_cache 4 15 256 15 1 : tunables 120 60 8 : slabdata 1 1 0
ndisc_cache 1 20 192 20 1 : tunables 120 60 8 : slabdata 1 1 0
rawv6_sock 4 11 704 11 2 : tunables 54 27 8 : slabdata 1 1 0
udpv6_sock 1 6 640 6 1 : tunables 54 27 8 : slabdata 1 1 0
tcpv6_sock 5 6 1216 3 1 : tunables 24 12 8 : slabdata 2 2 0
unix_sock 23 27 448 9 1 : tunables 54 27 8 : slabdata 3 3 0
tcp_tw_bucket 6281 7192 128 31 1 : tunables 120 60 8 : slabdata 232 232 120
tcp_bind_bucket 6 226 16 226 1 : tunables 120 60 8 : slabdata 1 1 0
tcp_open_request 214 310 128 31 1 : tunables 120 60 8 : slabdata 10 10 0
inet_peer_cache 2 61 64 61 1 : tunables 120 60 8 : slabdata 1 1 0
secpath_cache 0 0 128 31 1 : tunables 120 60 8 : slabdata 0 0 0
xfrm_dst_cache 0 0 256 15 1 : tunables 120 60 8 : slabdata 0 0 0
ip_fib_alias 9 226 16 226 1 : tunables 120 60 8 : slabdata 1 1 0
ip_fib_hash 9 119 32 119 1 : tunables 120 60 8 : slabdata 1 1 0
ip_dst_cache 11007 13515 256 15 1 : tunables 120 60 8 : slabdata 901 901 0
arp_cache 1 20 192 20 1 : tunables 120 60 8 : slabdata 1 1 0
raw_sock 3 7 512 7 1 : tunables 54 27 8 : slabdata 1 1 0
udp_sock 3 8 512 8 1 : tunables 54 27 8 : slabdata 1 1 0
tcp_sock 7192 7910 1088 7 2 : tunables 24 12 8 : slabdata 1130 1130 72
flow_cache 0 0 128 31 1 : tunables 120 60 8 : slabdata 0 0 0
dm-snapshot-in 128 162 48 81 1 : tunables 120 60 8 : slabdata 2 2 0
dm-snapshot-ex 0 0 16 226 1 : tunables 120 60 8 : slabdata 0 0 0
dm-crypt_io 0 0 76 52 1 : tunables 120 60 8 : slabdata 0 0 0
dm_tio 0 0 16 226 1 : tunables 120 60 8 : slabdata 0 0 0
dm_io 0 0 16 226 1 : tunables 120 60 8 : slabdata 0 0 0
scsi_cmd_cache 100 100 384 10 1 : tunables 54 27 8 : slabdata 10 10 27
cfq_ioc_pool 0 0 24 156 1 : tunables 120 60 8 : slabdata 0 0 0
cfq_pool 0 0 104 38 1 : tunables 120 60 8 : slabdata 0 0 0
crq_pool 0 0 52 75 1 : tunables 120 60 8 : slabdata 0 0 0
deadline_drq 0 0 48 81 1 : tunables 120 60 8 : slabdata 0 0 0
as_arq 515 845 60 65 1 : tunables 120 60 8 : slabdata 13 13 300
xfs_chashlist 583 740 20 185 1 : tunables 120 60 8 : slabdata 4 4 0
xfs_ili 1873 2184 140 28 1 : tunables 120 60 8 : slabdata 78 78 0
xfs_ifork 0 0 56 70 1 : tunables 120 60 8 : slabdata 0 0 0
xfs_efi_item 15 15 260 15 1 : tunables 54 27 8 : slabdata 1 1 0
xfs_efd_item 15 15 260 15 1 : tunables 54 27 8 : slabdata 1 1 0
xfs_buf_item 639 783 148 27 1 : tunables 120 60 8 : slabdata 29 29 284
xfs_dabuf 10 226 16 226 1 : tunables 120 60 8 : slabdata 1 1 0
xfs_da_state 0 0 336 12 1 : tunables 54 27 8 : slabdata 0 0 0
xfs_trans 260 260 596 13 2 : tunables 54 27 8 : slabdata 20 20 27
xfs_inode 2212 3135 368 11 1 : tunables 54 27 8 : slabdata 285 285 0
xfs_btree_cur 120 120 132 30 1 : tunables 120 60 8 : slabdata 4 4 0
xfs_bmap_free_item 24 290 12 290 1 : tunables 120 60 8 : slabdata 1 1 0
xfs_buf_t 690 870 256 15 1 : tunables 120 60 8 : slabdata 58 58 224
linvfs_icache 2003 2673 344 11 1 : tunables 54 27 8 : slabdata 243 243 0
hugetlbfs_inode_cache 1 12 312 12 1 : tunables 54 27 8 : slabdata 1 1 0
ext2_inode_cache 0 0 448 9 1 : tunables 54 27 8 : slabdata 0 0 0
ext2_xattr 0 0 44 88 1 : tunables 120 60 8 : slabdata 0 0 0
journal_handle 135 135 28 135 1 : tunables 120 60 8 : slabdata 1 1 0
journal_head 849 2835 48 81 1 : tunables 120 60 8 : slabdata 35 35 60
revoke_table 2 290 12 290 1 : tunables 120 60 8 : slabdata 1 1 0
revoke_record 109 226 16 226 1 : tunables 120 60 8 : slabdata 1 1 0
ext3_inode_cache 414 707 508 7 1 : tunables 54 27 8 : slabdata 101 101 0
ext3_xattr 0 0 44 88 1 : tunables 120 60 8 : slabdata 0 0 0
dnotify_cache 0 0 20 185 1 : tunables 120 60 8 : slabdata 0 0 0
eventpoll_pwq 0 0 36 107 1 : tunables 120 60 8 : slabdata 0 0 0
eventpoll_epi 0 0 128 31 1 : tunables 120 60 8 : slabdata 0 0 0
kioctx 0 0 192 20 1 : tunables 120 60 8 : slabdata 0 0 0
kiocb 0 0 128 31 1 : tunables 120 60 8 : slabdata 0 0 0
fasync_cache 0 0 16 226 1 : tunables 120 60 8 : slabdata 0 0 0
shmem_inode_cache 7 9 408 9 1 : tunables 54 27 8 : slabdata 1 1 0
posix_timers_cache 0 0 104 38 1 : tunables 120 60 8 : slabdata 0 0 0
uid_cache 3 61 64 61 1 : tunables 120 60 8 : slabdata 1 1 0
sgpool-128 32 33 2560 3 2 : tunables 24 12 8 : slabdata 11 11 0
sgpool-64 32 33 1280 3 1 : tunables 24 12 8 : slabdata 11 11 0
sgpool-32 102 102 640 6 1 : tunables 54 27 8 : slabdata 17 17 27
sgpool-16 96 96 320 12 1 : tunables 54 27 8 : slabdata 8 8 0
sgpool-8 200 200 192 20 1 : tunables 120 60 8 : slabdata 10 10 60
blkdev_ioc 84 135 28 135 1 : tunables 120 60 8 : slabdata 1 1 0
blkdev_queue 10 20 372 10 1 : tunables 54 27 8 : slabdata 2 2 0
blkdev_requests 471 840 140 28 1 : tunables 120 60 8 : slabdata 30 30 240
biovec-(256) 256 256 3072 2 2 : tunables 24 12 8 : slabdata 128 128 0
biovec-128 256 260 1536 5 2 : tunables 24 12 8 : slabdata 52 52 0
biovec-64 320 320 768 5 1 : tunables 54 27 8 : slabdata 64 64 0
biovec-16 350 380 192 20 1 : tunables 120 60 8 : slabdata 19 19 0
biovec-4 374 427 64 61 1 : tunables 120 60 8 : slabdata 7 7 0
biovec-1 1456 3164 16 226 1 : tunables 120 60 8 : slabdata 14 14 480
bio 1418 2196 64 61 1 : tunables 120 60 8 : slabdata 36 36 480
file_lock_cache 23 43 92 43 1 : tunables 120 60 8 : slabdata 1 1 0
sock_inode_cache 7064 7830 384 10 1 : tunables 54 27 8 : slabdata 783 783 135
skbuff_head_cache 14636 23220 192 20 1 : tunables 120 60 8 : slabdata 1161 1161 420
sock 3 10 384 10 1 : tunables 54 27 8 : slabdata 1 1 0
proc_inode_cache 321 360 328 12 1 : tunables 54 27 8 : slabdata 30 30 0
sigqueue 19 27 148 27 1 : tunables 120 60 8 : slabdata 1 1 0
radix_tree_node 22347 28602 276 14 1 : tunables 54 27 8 : slabdata 2043 2043 27
bdev_cache 30 45 448 9 1 : tunables 54 27 8 : slabdata 5 5 0
mnt_cache 26 62 128 31 1 : tunables 120 60 8 : slabdata 2 2 0
inode_cache 1040 1128 312 12 1 : tunables 54 27 8 : slabdata 94 94 0
dentry_cache 10067 15400 140 28 1 : tunables 120 60 8 : slabdata 550 550 240
filp 8189 9100 192 20 1 : tunables 120 60 8 : slabdata 455 455 60
names_cache 26 26 4096 1 1 : tunables 24 12 8 : slabdata 26 26 0
idr_layer_cache 112 116 136 29 1 : tunables 120 60 8 : slabdata 4 4 0
buffer_head 345712 472959 48 81 1 : tunables 120 60 8 : slabdata 5839 5839 480
mm_struct 91 91 576 7 1 : tunables 54 27 8 : slabdata 13 13 0
vm_area_struct 1994 2205 88 45 1 : tunables 120 60 8 : slabdata 49 49 0
fs_cache 86 183 64 61 1 : tunables 120 60 8 : slabdata 3 3 0
files_cache 87 99 448 9 1 : tunables 54 27 8 : slabdata 11 11 0
signal_cache 111 165 256 15 1 : tunables 120 60 8 : slabdata 11 11 0
sighand_cache 104 108 1344 3 1 : tunables 24 12 8 : slabdata 36 36 0
task_struct 111 111 1280 3 1 : tunables 24 12 8 : slabdata 37 37 0
anon_vma 635 870 12 290 1 : tunables 120 60 8 : slabdata 3 3 0
pgd 115 238 32 119 1 : tunables 120 60 8 : slabdata 2 2 0
pmd 180 180 4096 1 1 : tunables 24 12 8 : slabdata 180 180 0
size-131072(DMA) 0 0 131072 1 32 : tunables 8 4 0 : slabdata 0 0 0
size-131072 0 0 131072 1 32 : tunables 8 4 0 : slabdata 0 0 0
size-65536(DMA) 0 0 65536 1 16 : tunables 8 4 0 : slabdata 0 0 0
size-65536 19 19 65536 1 16 : tunables 8 4 0 : slabdata 19 19 0
size-32768(DMA) 0 0 32768 1 8 : tunables 8 4 0 : slabdata 0 0 0
size-32768 139 139 32768 1 8 : tunables 8 4 0 : slabdata 139 139 0
size-16384(DMA) 0 0 16384 1 4 : tunables 8 4 0 : slabdata 0 0 0
size-16384 151 153 16384 1 4 : tunables 8 4 0 : slabdata 151 153 0
size-8192(DMA) 0 0 8192 1 2 : tunables 8 4 0 : slabdata 0 0 0
size-8192 365 365 8192 1 2 : tunables 8 4 0 : slabdata 365 365 0
size-4096(DMA) 0 0 4096 1 1 : tunables 24 12 8 : slabdata 0 0 0
size-4096 770 770 4096 1 1 : tunables 24 12 8 : slabdata 770 770 48
size-2048(DMA) 0 0 2048 2 1 : tunables 24 12 8 : slabdata 0 0 0
size-2048 14460 14556 2048 2 1 : tunables 24 12 8 : slabdata 7278 7278 12
size-1024(DMA) 0 0 1024 4 1 : tunables 54 27 8 : slabdata 0 0 0
size-1024 888 888 1024 4 1 : tunables 54 27 8 : slabdata 222 222 54
size-512(DMA) 0 0 512 8 1 : tunables 54 27 8 : slabdata 0 0 0
size-512 886 1352 512 8 1 : tunables 54 27 8 : slabdata 169 169 108
size-256(DMA) 0 0 256 15 1 : tunables 120 60 8 : slabdata 0 0 0
size-256 270 300 256 15 1 : tunables 120 60 8 : slabdata 20 20 0
size-192(DMA) 0 0 192 20 1 : tunables 120 60 8 : slabdata 0 0 0
size-192 156 180 192 20 1 : tunables 120 60 8 : slabdata 9 9 0
size-128(DMA) 0 0 128 31 1 : tunables 120 60 8 : slabdata 0 0 0
size-128 1921 2635 128 31 1 : tunables 120 60 8 : slabdata 85 85 0
size-64(DMA) 0 0 64 61 1 : tunables 120 60 8 : slabdata 0 0 0
size-64 7293 13603 64 61 1 : tunables 120 60 8 : slabdata 223 223 480
size-32(DMA) 0 0 32 119 1 : tunables 120 60 8 : slabdata 0 0 0
size-32 10802 19516 32 119 1 : tunables 120 60 8 : slabdata 164 164 120
kmem_cache 160 160 192 20 1 : tunables 120 60 8 : slabdata 8 8 0
^ permalink raw reply [flat|nested] 12+ messages in thread
end of thread, other threads:[~2004-11-04 18:21 UTC | newest]
Thread overview: 12+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2004-10-25 10:48 Kernel 2.6.9 Page Allocation Failures w/TSO+rollup.patch Justin Piszcz
2004-10-25 11:33 ` Nick Piggin
2004-10-25 12:03 ` Justin Piszcz
2004-10-27 21:40 ` Kernel 2.6.9 Multiple Page Allocation Failures (Part 2) Justin Piszcz
2004-10-27 21:58 ` Andrew Morton
2004-10-27 21:55 ` Justin Piszcz
2004-10-27 22:19 ` Francois Romieu
2004-10-27 22:23 ` Justin Piszcz
2004-10-28 0:33 ` Lee Revell
2004-11-03 22:24 Stefan Schmidt
2004-11-04 12:17 ` Marcelo Tosatti
2004-11-04 18:18 ` Stefan Schmidt
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.