* [2.6.24-rc8] page allocation failure...
@ 2008-02-14 20:40 Daniel J Blueman
2008-02-17 13:20 ` [2.6.25-rc2, 2.6.24-rc8] " Daniel J Blueman
0 siblings, 1 reply; 5+ messages in thread
From: Daniel J Blueman @ 2008-02-14 20:40 UTC (permalink / raw)
To: Linux Kernel
One of my x86-64 (1GB) systems experienced order 1 page alloc failure
after ~4 days of uptime and 9 times again in the hours since.
I understand the behaviour is that the reclaim kthread should have had
a couple of free pages around for this type allocation, but obviously
didn't (a number of times).
What other information may help with this?
Daniel
--- dmesg
swapper: page allocation failure. order:1, mode:0x4020
Pid: 0, comm: swapper Not tainted 2.6.24-rc8-117 #1
Call Trace:
<IRQ> [<ffffffff80258696>] __alloc_pages+0x336/0x390
[<ffffffff8047e677>] __netdev_alloc_skb+0x17/0x40
[<ffffffff80272825>] __slab_alloc+0x145/0x3d0
[<ffffffff8047e677>] __netdev_alloc_skb+0x17/0x40
[<ffffffff80273526>] __kmalloc_track_caller+0xf6/0x100
[<ffffffff8047d8cf>] __alloc_skb+0x6f/0x150
[<ffffffff8047e677>] __netdev_alloc_skb+0x17/0x40
[<ffffffff8040b3aa>] e1000_alloc_rx_buffers+0x17a/0x3a0
[<ffffffff804b2643>] ip_local_deliver_finish+0x83/0x1a0
[<ffffffff8040c00d>] e1000_clean_rx_irq+0x34d/0x550
[<ffffffff8040b031>] e1000_intr_msi+0x81/0x110
[<ffffffff8024e5f4>] handle_IRQ_event+0x34/0x70
[<ffffffff80250229>] handle_edge_irq+0xc9/0x150
[<ffffffff8020eeab>] do_IRQ+0x7b/0x100
[<ffffffff8020ad80>] mwait_idle+0x0/0x50
[<ffffffff8020c011>] ret_from_intr+0x0/0xa
<EOI> [<ffffffff802197a0>] lapic_next_event+0x0/0x10
[<ffffffff8020adc2>] mwait_idle+0x42/0x50
[<ffffffff8020ad05>] cpu_idle+0x75/0xa0
[<ffffffff806a8b3a>] start_kernel+0x25a/0x2e0
[<ffffffff806a8117>] _sinittext+0x117/0x120
Mem-info:
DMA per-cpu:
CPU 0: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
CPU 1: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
DMA32 per-cpu:
CPU 0: Hot: hi: 186, btch: 31 usd: 155 Cold: hi: 62, btch: 15 usd: 60
CPU 1: Hot: hi: 186, btch: 31 usd: 14 Cold: hi: 62, btch: 15 usd: 42
Active:114794 inactive:53917 dirty:8137 writeback:5025 unstable:0
free:3549 slab:79629 mapped:4643 pagetables:1017 bounce:0
DMA free:3968kB min:40kB low:48kB high:60kB active:72kB inactive:120kB
present:10236kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 992 992 992
DMA32 free:10228kB min:4008kB low:5008kB high:6012kB active:459104kB
inactive:215548kB present:1015936kB pages_scanned:0 all_unreclaimable?
no
lowmem_reserve[]: 0 0 0 0
DMA: 324*4kB 0*8kB 1*16kB 1*32kB 1*64kB 0*128kB 0*256kB 1*512kB
0*1024kB 1*2048kB 0*4096kB = 3968kB
DMA32: 2378*4kB 1*8kB 1*16kB 1*32kB 0*64kB 1*128kB 0*256kB 1*512kB
0*1024kB 0*2048kB 0*4096kB = 10208kB
Swap cache: add 750, delete 488, find 196/269, race 0+0
Free swap = 247720kB
Total swap = 248996kB
Free swap: 247720kB
261600 pages of RAM
5401 reserved pages
55912 pages shared
262 pages swap cached
--- /proc/slabinfo
slabinfo - version: 2.1
# name <active_objs> <num_objs> <objsize> <objperslab>
<pagesperslab> : tunables <limit> <batchcount> <sharedfactor> :
slabdata <active_slabs> <num_slabs> <sharedavail>
rpc_inode_cache 10 10 768 5 1 : tunables 0 0
0 : slabdata 2 2 0
xt_hashlimit 0 0 88 46 1 : tunables 0 0
0 : slabdata 0 0 0
nf_conntrack_expect 18 18 224 18 1 : tunables 0
0 0 : slabdata 1 1 0
nf_conntrack 63 70 280 14 1 : tunables 0 0
0 : slabdata 5 5 0
cfq_io_context 77 78 152 26 1 : tunables 0 0
0 : slabdata 3 3 0
cfq_queue 89 90 136 30 1 : tunables 0 0
0 : slabdata 3 3 0
bsg_cmd 0 0 312 13 1 : tunables 0 0
0 : slabdata 0 0 0
mqueue_inode_cache 9 9 832 9 2 : tunables 0 0
0 : slabdata 1 1 0
xfs_inode 186190 186984 512 8 1 : tunables 0 0
0 : slabdata 23373 23373 0
xfs_efi_item 22 22 352 11 1 : tunables 0 0
0 : slabdata 2 2 0
xfs_efd_item 22 22 360 11 1 : tunables 0 0
0 : slabdata 2 2 0
xfs_buf_item 58 66 184 22 1 : tunables 0 0
0 : slabdata 3 3 0
xfs_da_state 16 16 488 8 1 : tunables 0 0
0 : slabdata 2 2 0
xfs_vnode 186180 186865 576 7 1 : tunables 0 0
0 : slabdata 26695 26695 0
nfsd4_delegations 461 465 272 15 1 : tunables 0 0
0 : slabdata 31 31 0
nfsd4_stateowners 45 45 424 9 1 : tunables 0 0
0 : slabdata 5 5 0
isofs_inode_cache 0 0 576 7 1 : tunables 0 0
0 : slabdata 0 0 0
ext2_inode_cache 0 0 672 6 1 : tunables 0 0
0 : slabdata 0 0 0
journal_handle 340 340 24 170 1 : tunables 0 0
0 : slabdata 2 2 0
journal_head 85 126 96 42 1 : tunables 0 0
0 : slabdata 3 3 0
revoke_table 512 512 16 256 1 : tunables 0 0
0 : slabdata 2 2 0
revoke_record 256 256 32 128 1 : tunables 0 0
0 : slabdata 2 2 0
ext4_inode_cache 0 0 736 5 1 : tunables 0 0
0 : slabdata 0 0 0
ext3_inode_cache 5322 5940 688 11 2 : tunables 0 0
0 : slabdata 540 540 0
kioctx 50 60 384 10 1 : tunables 0 0
0 : slabdata 6 6 0
fasync_cache 340 340 24 170 1 : tunables 0 0
0 : slabdata 2 2 0
shmem_inode_cache 831 835 720 5 1 : tunables 0 0
0 : slabdata 167 167 0
nsproxy 146 146 56 73 1 : tunables 0 0
0 : slabdata 2 2 0
ip_dst_cache 132 132 320 12 1 : tunables 0 0
0 : slabdata 11 11 0
TCP 26 30 1600 5 2 : tunables 0 0
0 : slabdata 6 6 0
eventpoll_pwq 544 560 72 56 1 : tunables 0 0
0 : slabdata 10 10 0
scsi_io_context 0 0 112 36 1 : tunables 0 0
0 : slabdata 0 0 0
blkdev_queue 12 12 1648 4 2 : tunables 0 0
0 : slabdata 3 3 0
blkdev_requests 36 42 288 14 1 : tunables 0 0
0 : slabdata 3 3 0
sock_inode_cache 84 90 640 6 1 : tunables 0 0
0 : slabdata 15 15 0
skbuff_fclone_cache 20 27 448 9 1 : tunables 0
0 0 : slabdata 3 3 0
file_lock_cache 501 506 184 22 1 : tunables 0 0
0 : slabdata 23 23 0
Acpi-Parse 204 204 40 102 1 : tunables 0 0
0 : slabdata 2 2 0
proc_inode_cache 385 392 560 7 1 : tunables 0 0
0 : slabdata 56 56 0
sigqueue 50 50 160 25 1 : tunables 0 0
0 : slabdata 2 2 0
radix_tree_node 13307 16590 560 7 1 : tunables 0 0
0 : slabdata 2370 2370 0
bdev_cache 12 15 768 5 1 : tunables 0 0
0 : slabdata 3 3 0
sysfs_dir_cache 4641 4641 80 51 1 : tunables 0 0
0 : slabdata 91 91 0
inode_cache 71 91 528 7 1 : tunables 0 0
0 : slabdata 13 13 0
dentry 95161 97780 200 20 1 : tunables 0 0
0 : slabdata 4889 4889 0
names_cache 8 8 4096 2 2 : tunables 0 0
0 : slabdata 4 4 0
idr_layer_cache 137 140 536 7 1 : tunables 0 0
0 : slabdata 20 20 0
buffer_head 58093 59748 104 39 1 : tunables 0 0
0 : slabdata 1532 1532 0
mm_struct 64 70 768 5 1 : tunables 0 0
0 : slabdata 14 14 0
vm_area_struct 2462 2592 168 24 1 : tunables 0 0
0 : slabdata 108 108 0
files_cache 75 88 704 11 2 : tunables 0 0
0 : slabdata 8 8 0
signal_cache 109 117 832 9 2 : tunables 0 0
0 : slabdata 13 13 0
sighand_cache 88 90 2112 3 2 : tunables 0 0
0 : slabdata 30 30 0
task_struct 102 108 1808 4 2 : tunables 0 0
0 : slabdata 27 27 0
anon_vma 806 896 32 128 1 : tunables 0 0
0 : slabdata 7 7 0
pid_namespace 0 0 2104 3 2 : tunables 0 0
0 : slabdata 0 0 0
kmalloc-2048 592 624 2048 4 2 : tunables 0 0
0 : slabdata 156 156 0
kmalloc-1024 267 276 1024 4 1 : tunables 0 0
0 : slabdata 69 69 0
kmalloc-512 213 232 512 8 1 : tunables 0 0
0 : slabdata 29 29 0
kmalloc-256 524 656 256 16 1 : tunables 0 0
0 : slabdata 41 41 0
kmalloc-128 932 1152 128 32 1 : tunables 0 0
0 : slabdata 36 36 0
kmalloc-64 27166 31104 64 64 1 : tunables 0 0
0 : slabdata 486 486 0
kmalloc-32 7180 10880 32 128 1 : tunables 0 0
0 : slabdata 85 85 0
kmalloc-16 1886 2816 16 256 1 : tunables 0 0
0 : slabdata 11 11 0
kmalloc-8 2538 2560 8 512 1 : tunables 0 0
0 : slabdata 5 5 0
kmalloc-192 2205 2667 192 21 1 : tunables 0 0
0 : slabdata 127 127 0
kmalloc-96 3024 3192 96 42 1 : tunables 0 0
0 : slabdata 76 76 0
--- /proc/meminfo
MemTotal: 1024796 kB
MemFree: 55160 kB
Buffers: 12856 kB
Cached: 315856 kB
SwapCached: 1052 kB
Active: 461848 kB
Inactive: 242108 kB
SwapTotal: 248996 kB
SwapFree: 247724 kB
Dirty: 28 kB
Writeback: 0 kB
AnonPages: 375348 kB
Mapped: 18200 kB
Slab: 248164 kB
SReclaimable: 230716 kB
SUnreclaim: 17448 kB
PageTables: 4032 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
CommitLimit: 761392 kB
Committed_AS: 541628 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 264820 kB
VmallocChunk: 34359473499 kB
--- /proc/zoneinfo
Node 0, zone DMA
pages free 1259
min 10
low 12
high 15
scanned 0 (a: 2 i: 16)
spanned 4096
present 2559
nr_free_pages 1259
nr_inactive 66
nr_active 7
nr_anon_pages 0
nr_mapped 0
nr_file_pages 73
nr_dirty 0
nr_writeback 0
nr_slab_reclaimable 1296
nr_slab_unreclaimable 65
nr_page_table_pages 0
nr_unstable 0
nr_bounce 0
nr_vmscan_write 787
protection: (0, 992, 992, 992)
pagesets
cpu: 0 pcp: 0
count: 0
high: 0
batch: 1
cpu: 0 pcp: 1
count: 0
high: 0
batch: 1
vm stats threshold: 4
cpu: 1 pcp: 0
count: 0
high: 0
batch: 1
cpu: 1 pcp: 1
count: 0
high: 0
batch: 1
vm stats threshold: 4
all_unreclaimable: 0
prev_priority: 12
start_pfn: 0
Node 0, zone DMA32
pages free 12514
min 1002
low 1252
high 1503
scanned 0 (a: 0 i: 0)
spanned 257504
present 253984
nr_free_pages 12514
nr_inactive 60499
nr_active 115446
nr_anon_pages 93837
nr_mapped 4550
nr_file_pages 82403
nr_dirty 7
nr_writeback 0
nr_slab_reclaimable 56383
nr_slab_unreclaimable 4290
nr_page_table_pages 1008
nr_unstable 0
nr_bounce 0
nr_vmscan_write 7348
protection: (0, 0, 0, 0)
pagesets
cpu: 0 pcp: 0
count: 102
high: 186
batch: 31
cpu: 0 pcp: 1
count: 64
high: 62
batch: 15
vm stats threshold: 16
cpu: 1 pcp: 0
count: 102
high: 186
batch: 31
cpu: 1 pcp: 1
count: 65
high: 62
batch: 15
vm stats threshold: 16
all_unreclaimable: 0
prev_priority: 12
start_pfn: 4096
--
Daniel J Blueman
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [2.6.25-rc2, 2.6.24-rc8] page allocation failure...
2008-02-14 20:40 [2.6.24-rc8] page allocation failure Daniel J Blueman
@ 2008-02-17 13:20 ` Daniel J Blueman
2008-02-18 12:58 ` Andrew Morton
0 siblings, 1 reply; 5+ messages in thread
From: Daniel J Blueman @ 2008-02-17 13:20 UTC (permalink / raw)
To: Linux Kernel
I'm still hitting this with e1000e on 2.6.25-rc2, 10 times again.
It's clearly non-fatal, but then do we expect it to occur?
Daniel
--- [dmesg]
[ 1250.822786] swapper: page allocation failure. order:3, mode:0x4020
[ 1250.822786] Pid: 0, comm: swapper Not tainted 2.6.25-rc2-119 #2
[ 1250.822786]
[ 1250.822786] Call Trace:
[ 1250.822786] <IRQ> [<ffffffff8025fe9e>] __alloc_pages+0x34e/0x3a0
[ 1250.822786] [<ffffffff8048c6df>] ? __netdev_alloc_skb+0x1f/0x40
[ 1250.822786] [<ffffffff8027acc2>] __slab_alloc+0x102/0x3d0
[ 1250.822786] [<ffffffff8048c6df>] ? __netdev_alloc_skb+0x1f/0x40
[ 1250.822786] [<ffffffff8027b8cb>] __kmalloc_track_caller+0x7b/0xc0
[ 1250.822786] [<ffffffff8048b74f>] __alloc_skb+0x6f/0x160
[ 1250.822786] [<ffffffff8048c6df>] __netdev_alloc_skb+0x1f/0x40
[ 1250.822786] [<ffffffff8042652d>] e1000_alloc_rx_buffers+0x1ed/0x260
[ 1250.822786] [<ffffffff80426b5a>] e1000_clean_rx_irq+0x22a/0x330
[ 1250.822786] [<ffffffff80422981>] e1000_clean+0x1e1/0x540
[ 1250.822786] [<ffffffff8024b7a5>] ? tick_program_event+0x45/0x70
[ 1250.822786] [<ffffffff804930ba>] net_rx_action+0x9a/0x150
[ 1250.822786] [<ffffffff802336b4>] __do_softirq+0x74/0xf0
[ 1250.822786] [<ffffffff8020c5fc>] call_softirq+0x1c/0x30
[ 1250.822786] [<ffffffff8020eaad>] do_softirq+0x3d/0x80
[ 1250.822786] [<ffffffff80233635>] irq_exit+0x85/0x90
[ 1250.822786] [<ffffffff8020eba5>] do_IRQ+0x85/0x100
[ 1250.822786] [<ffffffff8020a5b0>] ? mwait_idle+0x0/0x50
[ 1250.822786] [<ffffffff8020b981>] ret_from_intr+0x0/0xa
[ 1250.822786] <EOI> [<ffffffff8020a5f5>] ? mwait_idle+0x45/0x50
[ 1250.822786] [<ffffffff80209a92>] ? enter_idle+0x22/0x30
[ 1250.822786] [<ffffffff8020a534>] ? cpu_idle+0x74/0xa0
[ 1250.822786] [<ffffffff80527825>] ? rest_init+0x55/0x60
[ 1250.822786]
[ 1250.822786] Mem-info:
[ 1250.822786] DMA per-cpu:
[ 1250.822786] CPU 0: hi: 0, btch: 1 usd: 0
[ 1250.822786] CPU 1: hi: 0, btch: 1 usd: 0
[ 1250.822786] DMA32 per-cpu:
[ 1250.822786] CPU 0: hi: 186, btch: 31 usd: 179
[ 1250.822786] CPU 1: hi: 186, btch: 31 usd: 159
[ 1250.822786] Active:59792 inactive:67236 dirty:4775 writeback:4779 unstable:0
[ 1250.822786] free:2232 slab:122927 mapped:3846 pagetables:715 bounce:0
[ 1250.822786] DMA free:3984kB min:36kB low:44kB high:52kB active:4kB
inactive:560kB present:10076kB pages_scanned:0 all_unreclaimable? no
[ 1250.822786] lowmem_reserve[]: 0 992 992 992
[ 1250.822786] DMA32 free:4944kB min:4008kB low:5008kB high:6012kB
active:239164kB inactive:268384kB present:1015936kB pages_scanned:0
all_unreclaimable? no
[ 1250.822786] lowmem_reserve[]: 0 0 0 0
[ 1250.822786] DMA: 6*4kB 1*8kB 1*16kB 1*32kB 1*64kB 0*128kB 1*256kB
1*512kB 1*1024kB 1*2048kB 0*4096kB = 3984kB
[ 1250.822786] DMA32: 836*4kB 148*8kB 18*16kB 0*32kB 1*64kB 1*128kB
0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 5008kB
[ 1250.822786] 88530 total pagecache pages
[ 1250.822786] Swap cache: add 33, delete 32, find 0/0
[ 1250.822786] Free swap = 248872kB
[ 1250.822786] Total swap = 248996kB
[ 1250.822786] Free swap: 248872kB
[ 1250.822786] 261600 pages of RAM
[ 1250.822786] 5448 reserved pages
[ 1250.822786] 55715 pages shared
[ 1250.822786] 1 pages swap cached
On Feb 14, 2008 8:40 PM, Daniel J Blueman <daniel.blueman@gmail.com> wrote:
> One of my x86-64 (1GB) systems experienced order 1 page alloc failure
> after ~4 days of uptime and 9 times again in the hours since.
>
> I understand the behaviour is that the reclaim kthread should have had
> a couple of free pages around for this type allocation, but obviously
> didn't (a number of times).
>
> What other information may help with this?
>
> Daniel
>
> --- dmesg
> swapper: page allocation failure. order:1, mode:0x4020
> Pid: 0, comm: swapper Not tainted 2.6.24-rc8-117 #1
>
> Call Trace:
> <IRQ> [<ffffffff80258696>] __alloc_pages+0x336/0x390
> [<ffffffff8047e677>] __netdev_alloc_skb+0x17/0x40
> [<ffffffff80272825>] __slab_alloc+0x145/0x3d0
> [<ffffffff8047e677>] __netdev_alloc_skb+0x17/0x40
> [<ffffffff80273526>] __kmalloc_track_caller+0xf6/0x100
> [<ffffffff8047d8cf>] __alloc_skb+0x6f/0x150
> [<ffffffff8047e677>] __netdev_alloc_skb+0x17/0x40
> [<ffffffff8040b3aa>] e1000_alloc_rx_buffers+0x17a/0x3a0
> [<ffffffff804b2643>] ip_local_deliver_finish+0x83/0x1a0
> [<ffffffff8040c00d>] e1000_clean_rx_irq+0x34d/0x550
> [<ffffffff8040b031>] e1000_intr_msi+0x81/0x110
> [<ffffffff8024e5f4>] handle_IRQ_event+0x34/0x70
> [<ffffffff80250229>] handle_edge_irq+0xc9/0x150
> [<ffffffff8020eeab>] do_IRQ+0x7b/0x100
> [<ffffffff8020ad80>] mwait_idle+0x0/0x50
> [<ffffffff8020c011>] ret_from_intr+0x0/0xa
> <EOI> [<ffffffff802197a0>] lapic_next_event+0x0/0x10
> [<ffffffff8020adc2>] mwait_idle+0x42/0x50
> [<ffffffff8020ad05>] cpu_idle+0x75/0xa0
> [<ffffffff806a8b3a>] start_kernel+0x25a/0x2e0
> [<ffffffff806a8117>] _sinittext+0x117/0x120
>
> Mem-info:
> DMA per-cpu:
> CPU 0: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
> CPU 1: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
> DMA32 per-cpu:
> CPU 0: Hot: hi: 186, btch: 31 usd: 155 Cold: hi: 62, btch: 15 usd: 60
> CPU 1: Hot: hi: 186, btch: 31 usd: 14 Cold: hi: 62, btch: 15 usd: 42
> Active:114794 inactive:53917 dirty:8137 writeback:5025 unstable:0
> free:3549 slab:79629 mapped:4643 pagetables:1017 bounce:0
> DMA free:3968kB min:40kB low:48kB high:60kB active:72kB inactive:120kB
> present:10236kB pages_scanned:0 all_unreclaimable? no
> lowmem_reserve[]: 0 992 992 992
> DMA32 free:10228kB min:4008kB low:5008kB high:6012kB active:459104kB
> inactive:215548kB present:1015936kB pages_scanned:0 all_unreclaimable?
> no
> lowmem_reserve[]: 0 0 0 0
> DMA: 324*4kB 0*8kB 1*16kB 1*32kB 1*64kB 0*128kB 0*256kB 1*512kB
> 0*1024kB 1*2048kB 0*4096kB = 3968kB
> DMA32: 2378*4kB 1*8kB 1*16kB 1*32kB 0*64kB 1*128kB 0*256kB 1*512kB
> 0*1024kB 0*2048kB 0*4096kB = 10208kB
> Swap cache: add 750, delete 488, find 196/269, race 0+0
> Free swap = 247720kB
> Total swap = 248996kB
> Free swap: 247720kB
> 261600 pages of RAM
> 5401 reserved pages
> 55912 pages shared
> 262 pages swap cached
>
> --- /proc/slabinfo
> slabinfo - version: 2.1
> # name <active_objs> <num_objs> <objsize> <objperslab>
> <pagesperslab> : tunables <limit> <batchcount> <sharedfactor> :
> slabdata <active_slabs> <num_slabs> <sharedavail>
> rpc_inode_cache 10 10 768 5 1 : tunables 0 0
> 0 : slabdata 2 2 0
> xt_hashlimit 0 0 88 46 1 : tunables 0 0
> 0 : slabdata 0 0 0
> nf_conntrack_expect 18 18 224 18 1 : tunables 0
> 0 0 : slabdata 1 1 0
> nf_conntrack 63 70 280 14 1 : tunables 0 0
> 0 : slabdata 5 5 0
> cfq_io_context 77 78 152 26 1 : tunables 0 0
> 0 : slabdata 3 3 0
> cfq_queue 89 90 136 30 1 : tunables 0 0
> 0 : slabdata 3 3 0
> bsg_cmd 0 0 312 13 1 : tunables 0 0
> 0 : slabdata 0 0 0
> mqueue_inode_cache 9 9 832 9 2 : tunables 0 0
> 0 : slabdata 1 1 0
> xfs_inode 186190 186984 512 8 1 : tunables 0 0
> 0 : slabdata 23373 23373 0
> xfs_efi_item 22 22 352 11 1 : tunables 0 0
> 0 : slabdata 2 2 0
> xfs_efd_item 22 22 360 11 1 : tunables 0 0
> 0 : slabdata 2 2 0
> xfs_buf_item 58 66 184 22 1 : tunables 0 0
> 0 : slabdata 3 3 0
> xfs_da_state 16 16 488 8 1 : tunables 0 0
> 0 : slabdata 2 2 0
> xfs_vnode 186180 186865 576 7 1 : tunables 0 0
> 0 : slabdata 26695 26695 0
> nfsd4_delegations 461 465 272 15 1 : tunables 0 0
> 0 : slabdata 31 31 0
> nfsd4_stateowners 45 45 424 9 1 : tunables 0 0
> 0 : slabdata 5 5 0
> isofs_inode_cache 0 0 576 7 1 : tunables 0 0
> 0 : slabdata 0 0 0
> ext2_inode_cache 0 0 672 6 1 : tunables 0 0
> 0 : slabdata 0 0 0
> journal_handle 340 340 24 170 1 : tunables 0 0
> 0 : slabdata 2 2 0
> journal_head 85 126 96 42 1 : tunables 0 0
> 0 : slabdata 3 3 0
> revoke_table 512 512 16 256 1 : tunables 0 0
> 0 : slabdata 2 2 0
> revoke_record 256 256 32 128 1 : tunables 0 0
> 0 : slabdata 2 2 0
> ext4_inode_cache 0 0 736 5 1 : tunables 0 0
> 0 : slabdata 0 0 0
> ext3_inode_cache 5322 5940 688 11 2 : tunables 0 0
> 0 : slabdata 540 540 0
> kioctx 50 60 384 10 1 : tunables 0 0
> 0 : slabdata 6 6 0
> fasync_cache 340 340 24 170 1 : tunables 0 0
> 0 : slabdata 2 2 0
> shmem_inode_cache 831 835 720 5 1 : tunables 0 0
> 0 : slabdata 167 167 0
> nsproxy 146 146 56 73 1 : tunables 0 0
> 0 : slabdata 2 2 0
> ip_dst_cache 132 132 320 12 1 : tunables 0 0
> 0 : slabdata 11 11 0
> TCP 26 30 1600 5 2 : tunables 0 0
> 0 : slabdata 6 6 0
> eventpoll_pwq 544 560 72 56 1 : tunables 0 0
> 0 : slabdata 10 10 0
> scsi_io_context 0 0 112 36 1 : tunables 0 0
> 0 : slabdata 0 0 0
> blkdev_queue 12 12 1648 4 2 : tunables 0 0
> 0 : slabdata 3 3 0
> blkdev_requests 36 42 288 14 1 : tunables 0 0
> 0 : slabdata 3 3 0
> sock_inode_cache 84 90 640 6 1 : tunables 0 0
> 0 : slabdata 15 15 0
> skbuff_fclone_cache 20 27 448 9 1 : tunables 0
> 0 0 : slabdata 3 3 0
> file_lock_cache 501 506 184 22 1 : tunables 0 0
> 0 : slabdata 23 23 0
> Acpi-Parse 204 204 40 102 1 : tunables 0 0
> 0 : slabdata 2 2 0
> proc_inode_cache 385 392 560 7 1 : tunables 0 0
> 0 : slabdata 56 56 0
> sigqueue 50 50 160 25 1 : tunables 0 0
> 0 : slabdata 2 2 0
> radix_tree_node 13307 16590 560 7 1 : tunables 0 0
> 0 : slabdata 2370 2370 0
> bdev_cache 12 15 768 5 1 : tunables 0 0
> 0 : slabdata 3 3 0
> sysfs_dir_cache 4641 4641 80 51 1 : tunables 0 0
> 0 : slabdata 91 91 0
> inode_cache 71 91 528 7 1 : tunables 0 0
> 0 : slabdata 13 13 0
> dentry 95161 97780 200 20 1 : tunables 0 0
> 0 : slabdata 4889 4889 0
> names_cache 8 8 4096 2 2 : tunables 0 0
> 0 : slabdata 4 4 0
> idr_layer_cache 137 140 536 7 1 : tunables 0 0
> 0 : slabdata 20 20 0
> buffer_head 58093 59748 104 39 1 : tunables 0 0
> 0 : slabdata 1532 1532 0
> mm_struct 64 70 768 5 1 : tunables 0 0
> 0 : slabdata 14 14 0
> vm_area_struct 2462 2592 168 24 1 : tunables 0 0
> 0 : slabdata 108 108 0
> files_cache 75 88 704 11 2 : tunables 0 0
> 0 : slabdata 8 8 0
> signal_cache 109 117 832 9 2 : tunables 0 0
> 0 : slabdata 13 13 0
> sighand_cache 88 90 2112 3 2 : tunables 0 0
> 0 : slabdata 30 30 0
> task_struct 102 108 1808 4 2 : tunables 0 0
> 0 : slabdata 27 27 0
> anon_vma 806 896 32 128 1 : tunables 0 0
> 0 : slabdata 7 7 0
> pid_namespace 0 0 2104 3 2 : tunables 0 0
> 0 : slabdata 0 0 0
> kmalloc-2048 592 624 2048 4 2 : tunables 0 0
> 0 : slabdata 156 156 0
> kmalloc-1024 267 276 1024 4 1 : tunables 0 0
> 0 : slabdata 69 69 0
> kmalloc-512 213 232 512 8 1 : tunables 0 0
> 0 : slabdata 29 29 0
> kmalloc-256 524 656 256 16 1 : tunables 0 0
> 0 : slabdata 41 41 0
> kmalloc-128 932 1152 128 32 1 : tunables 0 0
> 0 : slabdata 36 36 0
> kmalloc-64 27166 31104 64 64 1 : tunables 0 0
> 0 : slabdata 486 486 0
> kmalloc-32 7180 10880 32 128 1 : tunables 0 0
> 0 : slabdata 85 85 0
> kmalloc-16 1886 2816 16 256 1 : tunables 0 0
> 0 : slabdata 11 11 0
> kmalloc-8 2538 2560 8 512 1 : tunables 0 0
> 0 : slabdata 5 5 0
> kmalloc-192 2205 2667 192 21 1 : tunables 0 0
> 0 : slabdata 127 127 0
> kmalloc-96 3024 3192 96 42 1 : tunables 0 0
> 0 : slabdata 76 76 0
>
> --- /proc/meminfo
> MemTotal: 1024796 kB
> MemFree: 55160 kB
> Buffers: 12856 kB
> Cached: 315856 kB
> SwapCached: 1052 kB
> Active: 461848 kB
> Inactive: 242108 kB
> SwapTotal: 248996 kB
> SwapFree: 247724 kB
> Dirty: 28 kB
> Writeback: 0 kB
> AnonPages: 375348 kB
> Mapped: 18200 kB
> Slab: 248164 kB
> SReclaimable: 230716 kB
> SUnreclaim: 17448 kB
> PageTables: 4032 kB
> NFS_Unstable: 0 kB
> Bounce: 0 kB
> CommitLimit: 761392 kB
> Committed_AS: 541628 kB
> VmallocTotal: 34359738367 kB
> VmallocUsed: 264820 kB
> VmallocChunk: 34359473499 kB
>
> --- /proc/zoneinfo
> Node 0, zone DMA
> pages free 1259
> min 10
> low 12
> high 15
> scanned 0 (a: 2 i: 16)
> spanned 4096
> present 2559
> nr_free_pages 1259
> nr_inactive 66
> nr_active 7
> nr_anon_pages 0
> nr_mapped 0
> nr_file_pages 73
> nr_dirty 0
> nr_writeback 0
> nr_slab_reclaimable 1296
> nr_slab_unreclaimable 65
> nr_page_table_pages 0
> nr_unstable 0
> nr_bounce 0
> nr_vmscan_write 787
> protection: (0, 992, 992, 992)
> pagesets
> cpu: 0 pcp: 0
> count: 0
> high: 0
> batch: 1
> cpu: 0 pcp: 1
> count: 0
> high: 0
> batch: 1
> vm stats threshold: 4
> cpu: 1 pcp: 0
> count: 0
> high: 0
> batch: 1
> cpu: 1 pcp: 1
> count: 0
> high: 0
> batch: 1
> vm stats threshold: 4
> all_unreclaimable: 0
> prev_priority: 12
> start_pfn: 0
> Node 0, zone DMA32
> pages free 12514
> min 1002
> low 1252
> high 1503
> scanned 0 (a: 0 i: 0)
> spanned 257504
> present 253984
> nr_free_pages 12514
> nr_inactive 60499
> nr_active 115446
> nr_anon_pages 93837
> nr_mapped 4550
> nr_file_pages 82403
> nr_dirty 7
> nr_writeback 0
> nr_slab_reclaimable 56383
> nr_slab_unreclaimable 4290
> nr_page_table_pages 1008
> nr_unstable 0
> nr_bounce 0
> nr_vmscan_write 7348
> protection: (0, 0, 0, 0)
> pagesets
> cpu: 0 pcp: 0
> count: 102
> high: 186
> batch: 31
> cpu: 0 pcp: 1
> count: 64
> high: 62
> batch: 15
> vm stats threshold: 16
> cpu: 1 pcp: 0
> count: 102
> high: 186
> batch: 31
> cpu: 1 pcp: 1
> count: 65
> high: 62
> batch: 15
> vm stats threshold: 16
> all_unreclaimable: 0
> prev_priority: 12
> start_pfn: 4096
> --
> Daniel J Blueman
>
--
Daniel J Blueman
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [2.6.25-rc2, 2.6.24-rc8] page allocation failure...
2008-02-17 13:20 ` [2.6.25-rc2, 2.6.24-rc8] " Daniel J Blueman
@ 2008-02-18 12:58 ` Andrew Morton
2008-02-19 17:37 ` Kok, Auke
0 siblings, 1 reply; 5+ messages in thread
From: Andrew Morton @ 2008-02-18 12:58 UTC (permalink / raw)
To: Daniel J Blueman; +Cc: Linux Kernel, netdev, Auke Kok
On Sun, 17 Feb 2008 13:20:59 +0000 "Daniel J Blueman" <daniel.blueman@gmail.com> wrote:
> I'm still hitting this with e1000e on 2.6.25-rc2, 10 times again.
>
> It's clearly non-fatal, but then do we expect it to occur?
>
> Daniel
>
> --- [dmesg]
>
> [ 1250.822786] swapper: page allocation failure. order:3, mode:0x4020
> [ 1250.822786] Pid: 0, comm: swapper Not tainted 2.6.25-rc2-119 #2
> [ 1250.822786]
> [ 1250.822786] Call Trace:
> [ 1250.822786] <IRQ> [<ffffffff8025fe9e>] __alloc_pages+0x34e/0x3a0
> [ 1250.822786] [<ffffffff8048c6df>] ? __netdev_alloc_skb+0x1f/0x40
> [ 1250.822786] [<ffffffff8027acc2>] __slab_alloc+0x102/0x3d0
> [ 1250.822786] [<ffffffff8048c6df>] ? __netdev_alloc_skb+0x1f/0x40
> [ 1250.822786] [<ffffffff8027b8cb>] __kmalloc_track_caller+0x7b/0xc0
> [ 1250.822786] [<ffffffff8048b74f>] __alloc_skb+0x6f/0x160
> [ 1250.822786] [<ffffffff8048c6df>] __netdev_alloc_skb+0x1f/0x40
> [ 1250.822786] [<ffffffff8042652d>] e1000_alloc_rx_buffers+0x1ed/0x260
> [ 1250.822786] [<ffffffff80426b5a>] e1000_clean_rx_irq+0x22a/0x330
> [ 1250.822786] [<ffffffff80422981>] e1000_clean+0x1e1/0x540
> [ 1250.822786] [<ffffffff8024b7a5>] ? tick_program_event+0x45/0x70
> [ 1250.822786] [<ffffffff804930ba>] net_rx_action+0x9a/0x150
> [ 1250.822786] [<ffffffff802336b4>] __do_softirq+0x74/0xf0
> [ 1250.822786] [<ffffffff8020c5fc>] call_softirq+0x1c/0x30
> [ 1250.822786] [<ffffffff8020eaad>] do_softirq+0x3d/0x80
> [ 1250.822786] [<ffffffff80233635>] irq_exit+0x85/0x90
> [ 1250.822786] [<ffffffff8020eba5>] do_IRQ+0x85/0x100
> [ 1250.822786] [<ffffffff8020a5b0>] ? mwait_idle+0x0/0x50
> [ 1250.822786] [<ffffffff8020b981>] ret_from_intr+0x0/0xa
> [ 1250.822786] <EOI> [<ffffffff8020a5f5>] ? mwait_idle+0x45/0x50
> [ 1250.822786] [<ffffffff80209a92>] ? enter_idle+0x22/0x30
> [ 1250.822786] [<ffffffff8020a534>] ? cpu_idle+0x74/0xa0
> [ 1250.822786] [<ffffffff80527825>] ? rest_init+0x55/0x60
They're regularly reported with e1000 too - I don't think aything really
changed.
e1000 has this crazy problem where because of a cascade of follies (mainly
borked hardware) it has to do a 32kb allocation for a 9kb(?) packet. It
would be sad if that was carried over into e1000e?
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [2.6.25-rc2, 2.6.24-rc8] page allocation failure...
2008-02-18 12:58 ` Andrew Morton
@ 2008-02-19 17:37 ` Kok, Auke
2008-02-24 12:37 ` Daniel J Blueman
0 siblings, 1 reply; 5+ messages in thread
From: Kok, Auke @ 2008-02-19 17:37 UTC (permalink / raw)
To: Andrew Morton; +Cc: Daniel J Blueman, Linux Kernel, netdev
Andrew Morton wrote:
> On Sun, 17 Feb 2008 13:20:59 +0000 "Daniel J Blueman" <daniel.blueman@gmail.com> wrote:
>
>> I'm still hitting this with e1000e on 2.6.25-rc2, 10 times again.
are you sure? I don't think that's the case and you're seeing e1000 dumps here...
>> It's clearly non-fatal, but then do we expect it to occur?
>>
>> Daniel
>>
>> --- [dmesg]
>>
>> [ 1250.822786] swapper: page allocation failure. order:3, mode:0x4020
>> [ 1250.822786] Pid: 0, comm: swapper Not tainted 2.6.25-rc2-119 #2
>> [ 1250.822786]
>> [ 1250.822786] Call Trace:
>> [ 1250.822786] <IRQ> [<ffffffff8025fe9e>] __alloc_pages+0x34e/0x3a0
>> [ 1250.822786] [<ffffffff8048c6df>] ? __netdev_alloc_skb+0x1f/0x40
>> [ 1250.822786] [<ffffffff8027acc2>] __slab_alloc+0x102/0x3d0
>> [ 1250.822786] [<ffffffff8048c6df>] ? __netdev_alloc_skb+0x1f/0x40
>> [ 1250.822786] [<ffffffff8027b8cb>] __kmalloc_track_caller+0x7b/0xc0
>> [ 1250.822786] [<ffffffff8048b74f>] __alloc_skb+0x6f/0x160
>> [ 1250.822786] [<ffffffff8048c6df>] __netdev_alloc_skb+0x1f/0x40
>> [ 1250.822786] [<ffffffff8042652d>] e1000_alloc_rx_buffers+0x1ed/0x260
>> [ 1250.822786] [<ffffffff80426b5a>] e1000_clean_rx_irq+0x22a/0x330
>> [ 1250.822786] [<ffffffff80422981>] e1000_clean+0x1e1/0x540
>> [ 1250.822786] [<ffffffff8024b7a5>] ? tick_program_event+0x45/0x70
>> [ 1250.822786] [<ffffffff804930ba>] net_rx_action+0x9a/0x150
>> [ 1250.822786] [<ffffffff802336b4>] __do_softirq+0x74/0xf0
>> [ 1250.822786] [<ffffffff8020c5fc>] call_softirq+0x1c/0x30
>> [ 1250.822786] [<ffffffff8020eaad>] do_softirq+0x3d/0x80
>> [ 1250.822786] [<ffffffff80233635>] irq_exit+0x85/0x90
>> [ 1250.822786] [<ffffffff8020eba5>] do_IRQ+0x85/0x100
>> [ 1250.822786] [<ffffffff8020a5b0>] ? mwait_idle+0x0/0x50
>> [ 1250.822786] [<ffffffff8020b981>] ret_from_intr+0x0/0xa
>> [ 1250.822786] <EOI> [<ffffffff8020a5f5>] ? mwait_idle+0x45/0x50
>> [ 1250.822786] [<ffffffff80209a92>] ? enter_idle+0x22/0x30
>> [ 1250.822786] [<ffffffff8020a534>] ? cpu_idle+0x74/0xa0
>> [ 1250.822786] [<ffffffff80527825>] ? rest_init+0x55/0x60
>
> They're regularly reported with e1000 too - I don't think aything really
> changed.
>
> e1000 has this crazy problem where because of a cascade of follies (mainly
> borked hardware) it has to do a 32kb allocation for a 9kb(?) packet. It
> would be sad if that was carried over into e1000e?
can't be, I personally removed that code.
for MTU > 1500 e1000e uses a plain normal sized SKB. for anything bigger e1000e
uses pages.
so I don't see how this bug could still be showing up for e1000e at all. The large
skb receive code is all gone (literally, removed).
*please* rmmod e1000; modprobe e1000e and show the dumps again so we know for sure
that we're not looking at e1000 dumps.
short fix: increase ring size for e1000 with `modprobe e1000 RxDescriptors=4096`
(or use ethtool) and `echo -n 8192 > /proc/sys/vm/min_free_kbytes` or something
like that.
what nic hardware is this on? lspci?
Auke
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [2.6.25-rc2, 2.6.24-rc8] page allocation failure...
2008-02-19 17:37 ` Kok, Auke
@ 2008-02-24 12:37 ` Daniel J Blueman
0 siblings, 0 replies; 5+ messages in thread
From: Daniel J Blueman @ 2008-02-24 12:37 UTC (permalink / raw)
To: Kok, Auke; +Cc: Andrew Morton, Linux Kernel, netdev
On Tue, Feb 19, 2008 at 5:37 PM, Kok, Auke <auke-jan.h.kok@intel.com> wrote:
> Andrew Morton wrote:
> > On Sun, 17 Feb 2008 13:20:59 +0000 "Daniel J Blueman" <daniel.blueman@gmail.com> wrote:
> >> I'm still hitting this with e1000e on 2.6.25-rc2, 10 times again.
> are you sure? I don't think that's the case and you're seeing e1000 dumps here...
Indeed so! I thought I moved to e1000e a time ago, but forgot that I
had moved back due to lack of support for 82566DC, added since.
I'm not seeing any related messages with e1000e after a few days'
uptime, so all looks well...
Thanks again,
Daniel
> >> It's clearly non-fatal, but then do we expect it to occur?
> >>
> >> Daniel
> >>
> >> --- [dmesg]
> >>
> >> [ 1250.822786] swapper: page allocation failure. order:3, mode:0x4020
> >> [ 1250.822786] Pid: 0, comm: swapper Not tainted 2.6.25-rc2-119 #2
> >> [ 1250.822786]
> >> [ 1250.822786] Call Trace:
> >> [ 1250.822786] <IRQ> [<ffffffff8025fe9e>] __alloc_pages+0x34e/0x3a0
> >> [ 1250.822786] [<ffffffff8048c6df>] ? __netdev_alloc_skb+0x1f/0x40
> >> [ 1250.822786] [<ffffffff8027acc2>] __slab_alloc+0x102/0x3d0
> >> [ 1250.822786] [<ffffffff8048c6df>] ? __netdev_alloc_skb+0x1f/0x40
> >> [ 1250.822786] [<ffffffff8027b8cb>] __kmalloc_track_caller+0x7b/0xc0
> >> [ 1250.822786] [<ffffffff8048b74f>] __alloc_skb+0x6f/0x160
> >> [ 1250.822786] [<ffffffff8048c6df>] __netdev_alloc_skb+0x1f/0x40
> >> [ 1250.822786] [<ffffffff8042652d>] e1000_alloc_rx_buffers+0x1ed/0x260
> >> [ 1250.822786] [<ffffffff80426b5a>] e1000_clean_rx_irq+0x22a/0x330
> >> [ 1250.822786] [<ffffffff80422981>] e1000_clean+0x1e1/0x540
> >> [ 1250.822786] [<ffffffff8024b7a5>] ? tick_program_event+0x45/0x70
> >> [ 1250.822786] [<ffffffff804930ba>] net_rx_action+0x9a/0x150
> >> [ 1250.822786] [<ffffffff802336b4>] __do_softirq+0x74/0xf0
> >> [ 1250.822786] [<ffffffff8020c5fc>] call_softirq+0x1c/0x30
> >> [ 1250.822786] [<ffffffff8020eaad>] do_softirq+0x3d/0x80
> >> [ 1250.822786] [<ffffffff80233635>] irq_exit+0x85/0x90
> >> [ 1250.822786] [<ffffffff8020eba5>] do_IRQ+0x85/0x100
> >> [ 1250.822786] [<ffffffff8020a5b0>] ? mwait_idle+0x0/0x50
> >> [ 1250.822786] [<ffffffff8020b981>] ret_from_intr+0x0/0xa
> >> [ 1250.822786] <EOI> [<ffffffff8020a5f5>] ? mwait_idle+0x45/0x50
> >> [ 1250.822786] [<ffffffff80209a92>] ? enter_idle+0x22/0x30
> >> [ 1250.822786] [<ffffffff8020a534>] ? cpu_idle+0x74/0xa0
> >> [ 1250.822786] [<ffffffff80527825>] ? rest_init+0x55/0x60
> >
> > They're regularly reported with e1000 too - I don't think aything really
> > changed.
> >
> > e1000 has this crazy problem where because of a cascade of follies (mainly
> > borked hardware) it has to do a 32kb allocation for a 9kb(?) packet. It
> > would be sad if that was carried over into e1000e?
>
> can't be, I personally removed that code.
>
> for MTU > 1500 e1000e uses a plain normal sized SKB. for anything bigger e1000e
> uses pages.
>
> so I don't see how this bug could still be showing up for e1000e at all. The large
> skb receive code is all gone (literally, removed).
>
> *please* rmmod e1000; modprobe e1000e and show the dumps again so we know for sure
> that we're not looking at e1000 dumps.
>
> short fix: increase ring size for e1000 with `modprobe e1000 RxDescriptors=4096`
> (or use ethtool) and `echo -n 8192 > /proc/sys/vm/min_free_kbytes` or something
> like that.
>
> what nic hardware is this on? lspci?
>
> Auke
>
--
Daniel J Blueman
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2008-02-24 12:37 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2008-02-14 20:40 [2.6.24-rc8] page allocation failure Daniel J Blueman
2008-02-17 13:20 ` [2.6.25-rc2, 2.6.24-rc8] " Daniel J Blueman
2008-02-18 12:58 ` Andrew Morton
2008-02-19 17:37 ` Kok, Auke
2008-02-24 12:37 ` Daniel J Blueman
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).