linux-kernel.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
* RE: Out of Memory: Killed process 16498 (java).
@ 2006-01-23 10:03 Andy Chittenden
  2006-01-23 10:11 ` Jens Axboe
  0 siblings, 1 reply; 29+ messages in thread
From: Andy Chittenden @ 2006-01-23 10:03 UTC (permalink / raw)
  To: Jens Axboe; +Cc: Andrew Morton, davej, linux-kernel, lwoodman

> There's your problem, apparently both of these queues is 
> being set to a
> limit lower than blk_max_low_pfn which means the block layer 
> will revert
> to the isa dma bounce zone for that queue... There's room for
> improvement in the logic that chooses what zone to allocate 
> from.

So I presume that requires a source fix. Or can it be configured out?

-- 
Andy, BlueArc Engineering
 

> -----Original Message-----
> From: Jens Axboe [mailto:axboe@suse.de] 
> Sent: 23 January 2006 09:28
> To: Andy Chittenden
> Cc: Andrew Morton; davej@redhat.com; 
> linux-kernel@vger.kernel.org; lwoodman@redhat.com
> Subject: Re: Out of Memory: Killed process 16498 (java).
> 
> On Sat, Jan 21 2006, Andy Chittenden wrote:
> > ok - here is the complete dmesg as of today:
> > 
> > ksize
> > usbmon: debugfs is not available
> > mice: PS/2 mouse device common for all mice
> > NET: Registered protocol family 2
> > IP route cache hash table entries: 131072 (order: 8, 1048576 bytes)
> > TCP established hash table entries: 524288 (order: 10, 
> 4194304 bytes)
> > TCP bind hash table entries: 65536 (order: 7, 524288 bytes)
> > TCP: Hash tables configured (established 524288 bind 65536)
> > TCP reno registered
> > TCP bic registered
> > NET: Registered protocol family 1
> > NET: Registered protocol family 17
> > NET: Registered protocol family 8
> > NET: Registered protocol family 20
> > ACPI wakeup devices: 
> > PCI0 PS2K PS2M UAR2 UAR1 AC97 USB1 USB2 USB3 USB4 EHCI PWRB SLPB 
> > ACPI: (supports S0 S1 S3 S4 S5)
> > Freeing unused kernel memory: 148k freed
> > Uniform Multi-Platform E-IDE driver Revision: 7.00alpha2
> > ide: Assuming 33MHz system bus speed for PIO modes; 
> override with idebus=xx
> > input: AT Translated Set 2 keyboard as /class/input/input0
> > VP_IDE: IDE controller at PCI slot 0000:00:0f.1
> > GSI 17 sharing vector 0xB1 and IRQ 17
> > ACPI: PCI Interrupt 0000:00:0f.1[A] -> GSI 20 (level, low) -> IRQ 17
> > PCI: Via IRQ fixup for 0000:00:0f.1, from 255 to 1
> > VP_IDE: chipset revision 6
> > VP_IDE: not 100% native mode: will probe irqs later
> > VP_IDE: VIA vt8237 (rev 00) IDE UDMA133 controller on 
> pci0000:00:0f.1
> >     ide0: BM-DMA at 0xfc00-0xfc07, BIOS settings: hda:DMA, hdb:DMA
> >     ide1: BM-DMA at 0xfc08-0xfc0f, BIOS settings: hdc:DMA, hdd:pio
> > Probing IDE interface ide0...
> > hda: HDS722525VLAT80, ATA DISK drive
> > hdb: Maxtor 6Y200P0, ATA DISK drive
> > bounce: queue ffff81013fa312d8, setting pfn 1310720, max_low 1310720
> > bounce: queue ffff81013fa312d8, setting pfn 1048575, max_low 1310720
> > isa bounce pool size: 16 pages
> > bounce: queue ffff81013fa31048, setting pfn 1310720, max_low 1310720
> > bounce: queue ffff81013fa31048, setting pfn 1048575, max_low 1310720
> 
> There's your problem, apparently both of these queues is 
> being set to a
> limit lower than blk_max_low_pfn which means the block layer 
> will revert
> to the isa dma bounce zone for that queue... There's room for
> improvement in the logic that chooses what zone to allocate 
> from. It's a
> little problematic since there are two reasons why you would have to
> bounce: the hardware cannot support a "high" address, _or_ the driver
> cannot support a highmem page. The latter can be fixed, the former is
> given of course.
> 
> So choosing the gfp mask is a composite of whether the machine has
> highmem and the hardware/driver capabilities.
> 
> -- 
> Jens Axboe
> 
> 

^ permalink raw reply	[flat|nested] 29+ messages in thread

* Re: Out of Memory: Killed process 16498 (java).
  2006-01-23 10:03 Out of Memory: Killed process 16498 (java) Andy Chittenden
@ 2006-01-23 10:11 ` Jens Axboe
  0 siblings, 0 replies; 29+ messages in thread
From: Jens Axboe @ 2006-01-23 10:11 UTC (permalink / raw)
  To: Andy Chittenden; +Cc: Andrew Morton, davej, linux-kernel, lwoodman

On Mon, Jan 23 2006, Andy Chittenden wrote:
> > There's your problem, apparently both of these queues is 
> > being set to a
> > limit lower than blk_max_low_pfn which means the block layer 
> > will revert
> > to the isa dma bounce zone for that queue... There's room for
> > improvement in the logic that chooses what zone to allocate 
> > from.
> 
> So I presume that requires a source fix. Or can it be configured out?

It does, I'll see if I can get some time to generate one.

-- 
Jens Axboe


^ permalink raw reply	[flat|nested] 29+ messages in thread

* Re: Out of Memory: Killed process 16498 (java).
  2006-01-27 14:21 ` Jens Axboe
@ 2006-01-27 14:39   ` Anton Altaparmakov
  0 siblings, 0 replies; 29+ messages in thread
From: Anton Altaparmakov @ 2006-01-27 14:39 UTC (permalink / raw)
  To: Jens Axboe; +Cc: Andy Chittenden, Andrew Morton, davej, linux-kernel, lwoodman

Hi,

On Fri, 2006-01-27 at 15:21 +0100, Jens Axboe wrote:
> On Fri, Jan 27 2006, Andy Chittenden wrote:
> > Any progress on a patch?
> 
> (please don't top post)
> 
> Not yet, sorry, been busy with more critical bugs.

A colleague has a server (which does backups) that is incapable of doing
a backup due to the backup process being killed due to OOM after
anywhere between 30s and a few minutes of running...  And the backup
process is just a simple program that does the equivalent of "dd with
one source but two destinations" where the source is an lvm/dm snapshot
and the two destinations are two different tape drives attached via
scsi.  That is pretty critical, admittedly only to us and that system...

This problem means the system has to sit with an older kernel until such
time as the problem is fixed.

ps. I have no idea if it is the same problem or not but it may well be.
It is a SUSE kernel (SLES9) rather than stock kernel and the problem was
experienced with the 2.6.5-7.202.7 kernel update...  Back to .151 the
system is doing backups just fine, no kernels between .151 and .202.7
have been tried yet on that system and neither has the latest .244
kernel.  We (for a rather all encompassing value of we) pretty much
stopped at .151 due to the e1000 problems in all later kernels although
now we know the magic workaround of disabling TSO people around are
happier to try using more recent kernels (and leaving TSO disabled as
even .244 is badly borked there as soon as you get some traffic load on
any of the e1000s)...

Best regards,

        Anton
-- 
Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @)
Unix Support, Computing Service, University of Cambridge, CB2 3QH, UK
Linux NTFS maintainer / IRC: #ntfs on irc.freenode.net
WWW: http://linux-ntfs.sf.net/ & http://www-stu.christs.cam.ac.uk/~aia21/


^ permalink raw reply	[flat|nested] 29+ messages in thread

* Re: Out of Memory: Killed process 16498 (java).
  2006-01-27 11:53 Andy Chittenden
@ 2006-01-27 14:21 ` Jens Axboe
  2006-01-27 14:39   ` Anton Altaparmakov
  0 siblings, 1 reply; 29+ messages in thread
From: Jens Axboe @ 2006-01-27 14:21 UTC (permalink / raw)
  To: Andy Chittenden; +Cc: Andrew Morton, davej, linux-kernel, lwoodman

On Fri, Jan 27 2006, Andy Chittenden wrote:
> Any progress on a patch?

(please don't top post)

Not yet, sorry, been busy with more critical bugs.

-- 
Jens Axboe


^ permalink raw reply	[flat|nested] 29+ messages in thread

* RE: Out of Memory: Killed process 16498 (java).
@ 2006-01-27 11:53 Andy Chittenden
  2006-01-27 14:21 ` Jens Axboe
  0 siblings, 1 reply; 29+ messages in thread
From: Andy Chittenden @ 2006-01-27 11:53 UTC (permalink / raw)
  To: Jens Axboe; +Cc: Andrew Morton, davej, linux-kernel, lwoodman

Any progress on a patch?

-- 
Andy, BlueArc Engineering
 

> -----Original Message-----
> From: Jens Axboe [mailto:axboe@suse.de] 
> Sent: 23 January 2006 10:12
> To: Andy Chittenden
> Cc: Andrew Morton; davej@redhat.com; 
> linux-kernel@vger.kernel.org; lwoodman@redhat.com
> Subject: Re: Out of Memory: Killed process 16498 (java).
> 
> On Mon, Jan 23 2006, Andy Chittenden wrote:
> > > There's your problem, apparently both of these queues is 
> > > being set to a
> > > limit lower than blk_max_low_pfn which means the block layer 
> > > will revert
> > > to the isa dma bounce zone for that queue... There's room for
> > > improvement in the logic that chooses what zone to allocate 
> > > from.
> > 
> > So I presume that requires a source fix. Or can it be 
> configured out?
> 
> It does, I'll see if I can get some time to generate one.
> 
> -- 
> Jens Axboe
> 
> 

^ permalink raw reply	[flat|nested] 29+ messages in thread

* Re: Out of Memory: Killed process 16498 (java).
  2006-01-21 21:47 Andy Chittenden
@ 2006-01-23  9:27 ` Jens Axboe
  0 siblings, 0 replies; 29+ messages in thread
From: Jens Axboe @ 2006-01-23  9:27 UTC (permalink / raw)
  To: Andy Chittenden; +Cc: Andrew Morton, davej, linux-kernel, lwoodman

On Sat, Jan 21 2006, Andy Chittenden wrote:
> ok - here is the complete dmesg as of today:
> 
> ksize
> usbmon: debugfs is not available
> mice: PS/2 mouse device common for all mice
> NET: Registered protocol family 2
> IP route cache hash table entries: 131072 (order: 8, 1048576 bytes)
> TCP established hash table entries: 524288 (order: 10, 4194304 bytes)
> TCP bind hash table entries: 65536 (order: 7, 524288 bytes)
> TCP: Hash tables configured (established 524288 bind 65536)
> TCP reno registered
> TCP bic registered
> NET: Registered protocol family 1
> NET: Registered protocol family 17
> NET: Registered protocol family 8
> NET: Registered protocol family 20
> ACPI wakeup devices: 
> PCI0 PS2K PS2M UAR2 UAR1 AC97 USB1 USB2 USB3 USB4 EHCI PWRB SLPB 
> ACPI: (supports S0 S1 S3 S4 S5)
> Freeing unused kernel memory: 148k freed
> Uniform Multi-Platform E-IDE driver Revision: 7.00alpha2
> ide: Assuming 33MHz system bus speed for PIO modes; override with idebus=xx
> input: AT Translated Set 2 keyboard as /class/input/input0
> VP_IDE: IDE controller at PCI slot 0000:00:0f.1
> GSI 17 sharing vector 0xB1 and IRQ 17
> ACPI: PCI Interrupt 0000:00:0f.1[A] -> GSI 20 (level, low) -> IRQ 17
> PCI: Via IRQ fixup for 0000:00:0f.1, from 255 to 1
> VP_IDE: chipset revision 6
> VP_IDE: not 100% native mode: will probe irqs later
> VP_IDE: VIA vt8237 (rev 00) IDE UDMA133 controller on pci0000:00:0f.1
>     ide0: BM-DMA at 0xfc00-0xfc07, BIOS settings: hda:DMA, hdb:DMA
>     ide1: BM-DMA at 0xfc08-0xfc0f, BIOS settings: hdc:DMA, hdd:pio
> Probing IDE interface ide0...
> hda: HDS722525VLAT80, ATA DISK drive
> hdb: Maxtor 6Y200P0, ATA DISK drive
> bounce: queue ffff81013fa312d8, setting pfn 1310720, max_low 1310720
> bounce: queue ffff81013fa312d8, setting pfn 1048575, max_low 1310720
> isa bounce pool size: 16 pages
> bounce: queue ffff81013fa31048, setting pfn 1310720, max_low 1310720
> bounce: queue ffff81013fa31048, setting pfn 1048575, max_low 1310720

There's your problem, apparently both of these queues is being set to a
limit lower than blk_max_low_pfn which means the block layer will revert
to the isa dma bounce zone for that queue... There's room for
improvement in the logic that chooses what zone to allocate from. It's a
little problematic since there are two reasons why you would have to
bounce: the hardware cannot support a "high" address, _or_ the driver
cannot support a highmem page. The latter can be fixed, the former is
given of course.

So choosing the gfp mask is a composite of whether the machine has
highmem and the hardware/driver capabilities.

-- 
Jens Axboe


^ permalink raw reply	[flat|nested] 29+ messages in thread

* RE: Out of Memory: Killed process 16498 (java).
@ 2006-01-21 21:47 Andy Chittenden
  2006-01-23  9:27 ` Jens Axboe
  0 siblings, 1 reply; 29+ messages in thread
From: Andy Chittenden @ 2006-01-21 21:47 UTC (permalink / raw)
  To: Jens Axboe; +Cc: Andrew Morton, davej, linux-kernel, lwoodman

ok - here is the complete dmesg as of today:

ksize
usbmon: debugfs is not available
mice: PS/2 mouse device common for all mice
NET: Registered protocol family 2
IP route cache hash table entries: 131072 (order: 8, 1048576 bytes)
TCP established hash table entries: 524288 (order: 10, 4194304 bytes)
TCP bind hash table entries: 65536 (order: 7, 524288 bytes)
TCP: Hash tables configured (established 524288 bind 65536)
TCP reno registered
TCP bic registered
NET: Registered protocol family 1
NET: Registered protocol family 17
NET: Registered protocol family 8
NET: Registered protocol family 20
ACPI wakeup devices: 
PCI0 PS2K PS2M UAR2 UAR1 AC97 USB1 USB2 USB3 USB4 EHCI PWRB SLPB 
ACPI: (supports S0 S1 S3 S4 S5)
Freeing unused kernel memory: 148k freed
Uniform Multi-Platform E-IDE driver Revision: 7.00alpha2
ide: Assuming 33MHz system bus speed for PIO modes; override with idebus=xx
input: AT Translated Set 2 keyboard as /class/input/input0
VP_IDE: IDE controller at PCI slot 0000:00:0f.1
GSI 17 sharing vector 0xB1 and IRQ 17
ACPI: PCI Interrupt 0000:00:0f.1[A] -> GSI 20 (level, low) -> IRQ 17
PCI: Via IRQ fixup for 0000:00:0f.1, from 255 to 1
VP_IDE: chipset revision 6
VP_IDE: not 100% native mode: will probe irqs later
VP_IDE: VIA vt8237 (rev 00) IDE UDMA133 controller on pci0000:00:0f.1
    ide0: BM-DMA at 0xfc00-0xfc07, BIOS settings: hda:DMA, hdb:DMA
    ide1: BM-DMA at 0xfc08-0xfc0f, BIOS settings: hdc:DMA, hdd:pio
Probing IDE interface ide0...
hda: HDS722525VLAT80, ATA DISK drive
hdb: Maxtor 6Y200P0, ATA DISK drive
bounce: queue ffff81013fa312d8, setting pfn 1310720, max_low 1310720
bounce: queue ffff81013fa312d8, setting pfn 1048575, max_low 1310720
isa bounce pool size: 16 pages
bounce: queue ffff81013fa31048, setting pfn 1310720, max_low 1310720
bounce: queue ffff81013fa31048, setting pfn 1048575, max_low 1310720
ide0 at 0x1f0-0x1f7,0x3f6 on irq 14
Probing IDE interface ide1...
hdc: _NEC DVD_RW ND-3500AG, ATAPI CD/DVD-ROM drive
bounce: queue ffff81013f51fd18, setting pfn 1310720, max_low 1310720
bounce: queue ffff81013f51fd18, setting pfn 1310720, max_low 1310720
ide1 at 0x170-0x177,0x376 on irq 15
hda: max request size: 1024KiB
hda: 390721968 sectors (200049 MB) w/7938KiB Cache, CHS=24321/255/63, UDMA(100)
hda: cache flushes supported
 hda: hda1 hda2 hda3 < hda5 hda6 hda7 hda8 >
hdb: max request size: 1024KiB
hdb: 398297088 sectors (203928 MB) w/7936KiB Cache, CHS=24792/255/63, UDMA(133)
hdb: cache flushes supported
 hdb: hdb1 hdb2 hdb3 hdb4 < hdb5 hdb6 >
kjournald starting.  Commit interval 5 seconds
EXT3-fs: mounted filesystem with ordered data mode.
GSI 18 sharing vector 0xB9 and IRQ 18
ACPI: PCI Interrupt 0000:00:0a.0[A] -> GSI 17 (level, low) -> IRQ 18
skge 1.2 addr 0xfa900000 irq 18 chip Yukon-Lite rev 7
skge eth0: addr 00:11:2f:a7:fb:ef
USB Universal Host Controller Interface driver v2.3
GSI 19 sharing vector 0xC1 and IRQ 19
ACPI: PCI Interrupt 0000:00:10.0[A] -> GSI 21 (level, low) -> IRQ 19
PCI: Via IRQ fixup for 0000:00:10.0, from 11 to 3
uhci_hcd 0000:00:10.0: UHCI Host Controller
uhci_hcd 0000:00:10.0: new USB bus registered, assigned bus number 1
uhci_hcd 0000:00:10.0: irq 19, io base 0x0000d400
hub 1-0:1.0: USB hub found
hub 1-0:1.0: 2 ports detected
SCSI subsystem initialized
pci_hotplug: PCI Hot Plug PCI Core version: 0.5
libata version 1.20 loaded.
sata_promise 0000:00:08.0: version 1.03
GSI 20 sharing vector 0xC9 and IRQ 20
ACPI: PCI Interrupt 0000:00:08.0[A] -> GSI 18 (level, low) -> IRQ 20
ACPI: PCI Interrupt 0000:00:10.1[A] -> GSI 21 (level, low) -> IRQ 19
PCI: Via IRQ fixup for 0000:00:10.1, from 11 to 3
uhci_hcd 0000:00:10.1: UHCI Host Controller
uhci_hcd 0000:00:10.1: new USB bus registered, assigned bus number 2
uhci_hcd 0000:00:10.1: irq 19, io base 0x0000d800
hub 2-0:1.0: USB hub found
hub 2-0:1.0: 2 ports detected
ata1: SATA max UDMA/133 cmd 0xFFFFC20000004200 ctl 0xFFFFC20000004238 bmdma 0x0 irq 20
ata2: SATA max UDMA/133 cmd 0xFFFFC20000004280 ctl 0xFFFFC200000042B8 bmdma 0x0 irq 20
input: PC Speaker as /class/input/input1
ACPI: PCI Interrupt 0000:00:10.2[B] -> GSI 21 (level, low) -> IRQ 19
PCI: Via IRQ fixup for 0000:00:10.2, from 10 to 3
uhci_hcd 0000:00:10.2: UHCI Host Controller
uhci_hcd 0000:00:10.2: new USB bus registered, assigned bus number 3
uhci_hcd 0000:00:10.2: irq 19, io base 0x0000e000
hub 3-0:1.0: USB hub found
hub 3-0:1.0: 2 ports detected
hdc: ATAPI 48X DVD-ROM DVD-R CD-R/RW drive, 2048kB Cache, UDMA(33)
Uniform CD-ROM driver Revision: 3.20
ACPI: PCI Interrupt 0000:00:10.3[B] -> GSI 21 (level, low) -> IRQ 19
PCI: Via IRQ fixup for 0000:00:10.3, from 10 to 3
uhci_hcd 0000:00:10.3: UHCI Host Controller
uhci_hcd 0000:00:10.3: new USB bus registered, assigned bus number 4
uhci_hcd 0000:00:10.3: irq 19, io base 0x0000e400
hub 4-0:1.0: USB hub found
hub 4-0:1.0: 2 ports detected
ata1: no device found (phy stat 00000000)
scsi0 : sata_promise
e100: Intel(R) PRO/100 Network Driver, 3.4.14-k4-NAPI
e100: Copyright(c) 1999-2005 Intel Corporation
shpchp: Standard Hot Plug PCI Controller Driver version: 0.4
ata2: no device found (phy stat 00000000)
scsi1 : sata_promise
ACPI: PCI Interrupt 0000:00:10.4[C] -> GSI 21 (level, low) -> IRQ 19
PCI: Via IRQ fixup for 0000:00:10.4, from 5 to 3
ehci_hcd 0000:00:10.4: EHCI Host Controller
ehci_hcd 0000:00:10.4: new USB bus registered, assigned bus number 5
ehci_hcd 0000:00:10.4: irq 19, io mem 0xfae00000
ehci_hcd 0000:00:10.4: USB 2.0 started, EHCI 1.00, driver 10 Dec 2004
hub 5-0:1.0: USB hub found
hub 5-0:1.0: 8 ports detected
sata_via 0000:00:0f.0: version 1.1
ACPI: PCI Interrupt 0000:00:0f.0[B] -> GSI 20 (level, low) -> IRQ 17
PCI: Via IRQ fixup for 0000:00:0f.0, from 10 to 1
sata_via 0000:00:0f.0: routed to hard irq line 1
ACPI: PCI Interrupt 0000:00:0d.0[A] -> GSI 18 (level, low) -> IRQ 20
e100: eth1: e100_probe: addr 0xfac00000, irq 20, MAC addr 00:D0:B7:BF:92:C8
ata3: SATA max UDMA/133 cmd 0xD000 ctl 0xC802 bmdma 0xB800 irq 17
ata4: SATA max UDMA/133 cmd 0xC400 ctl 0xC002 bmdma 0xB808 irq 17
input: ImExPS/2 Generic Explorer Mouse as /class/input/input2
ata3: dev 0 cfg 49:2f00 82:3069 83:7c01 84:4003 85:3069 86:3c01 87:4003 88:203f
ata3: dev 0 ATA-6, max UDMA/100, 390721968 sectors: LBA48
ata3: dev 0 configured for UDMA/100
scsi2 : sata_via
ata4: no device found (phy stat 00000000)
scsi3 : sata_via
bounce: queue ffff81013f51fa88, setting pfn 1310720, max_low 1310720
bounce: queue ffff81013f51fa88, setting pfn 1048575, max_low 1310720
  Vendor: ATA       Model: ST3200822AS       Rev: 3.01
  Type:   Direct-Access                      ANSI SCSI revision: 05
bounce: queue ffff81013f51f7f8, setting pfn 1310720, max_low 1310720
Floppy drive(s): fd0 is 1.44M
FDC 0 is a post-1991 82077
GSI 21 sharing vector 0xD1 and IRQ 21
ACPI: PCI Interrupt 0000:00:11.5[C] -> GSI 22 (level, low) -> IRQ 21
PCI: Setting latency timer of device 0000:00:11.5 to 64
SCSI device sda: 390721968 512-byte hdwr sectors (200050 MB)
SCSI device sda: drive cache: write back
SCSI device sda: 390721968 512-byte hdwr sectors (200050 MB)
SCSI device sda: drive cache: write back
 sda: sda1 sda2 < sda5 > sda3
sd 2:0:0:0: Attached scsi disk sda
Adding 1951856k swap on /dev/hda5.  Priority:-1 extents:1 across:1951856k
Adding 1951856k swap on /dev/hda6.  Priority:-2 extents:1 across:1951856k
Adding 1951856k swap on /dev/hda7.  Priority:-3 extents:1 across:1951856k
Adding 996020k swap on /dev/hdb2.  Priority:-4 extents:1 across:996020k
Adding 996020k swap on /dev/hdb3.  Priority:-5 extents:1 across:996020k
Adding 995988k swap on /dev/hdb5.  Priority:-6 extents:1 across:995988k
Adding 843372k swap on /dev/hdb6.  Priority:-7 extents:1 across:843372k
EXT3 FS on hda1, internal journal
ieee1394: Initialized config rom entry `ip1394'
sbp2: $Rev: 1306 $ Ben Collins <bcollins@debian.org>
ieee1394: sbp2: Driver forced to serialize I/O (serialize_io=1)
ieee1394: sbp2: Try serialize_io=0 for better performance
kjournald starting.  Commit interval 5 seconds
EXT3 FS on hda2, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
kjournald starting.  Commit interval 5 seconds
EXT3 FS on hda8, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
SGI XFS with ACLs, security attributes, realtime, large block/inode numbers, no debug enabled
SGI XFS Quota Management subsystem
XFS mounting filesystem hdb1
Ending clean XFS mount for filesystem: hdb1
kjournald starting.  Commit interval 5 seconds
EXT3 FS on sda1, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
kjournald starting.  Commit interval 5 seconds
EXT3 FS on sda3, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
XFS mounting filesystem sda5
Ending clean XFS mount for filesystem: sda5
e100: intel: e100_watchdog: link up, 100Mbps, half-duplex
NET: Registered protocol family 10
lo: Disabled Privacy Extensions
IPv6 over IPv4 tunneling driver
ACPI: Power Button (FF) [PWRF]
ACPI: Power Button (CM) [PWRB]
ACPI: Sleep Button (CM) [SLPB]
pnp: the driver 'parport_pc' has been registered
lp: driver loaded but no devices found
intel: no IPv6 routers present
Installing knfsd (copyright (C) 1996 okir@monad.swb.de).
NFSD: Using /var/lib/nfs/v4recovery as the NFSv4 state recovery directory
NFSD: recovery directory /var/lib/nfs/v4recovery doesn't exist
NFSD: starting 90-second grace period
mtrr: type mismatch for f0000000,8000000 old: write-back new: write-combining
mtrr: type mismatch for f0000000,8000000 old: write-back new: write-combining
oom-killer: gfp_mask=0xd1, order=0

Call Trace:<ffffffff8014d7bc>{out_of_memory+48} <ffffffff8014f4b0>{__alloc_pages+536}
       <ffffffff80169788>{bio_alloc_bioset+232} <ffffffff80169d03>{bio_copy_user+218}
       <ffffffff801bd674>{blk_rq_map_user+136} <ffffffff801c0024>{sg_io+328}
       <ffffffff801c0498>{scsi_cmd_ioctl+491} <ffffffff88005e22>{:ide_core:generic_ide_ioctl+631}
       <ffffffff88206d0c>{:sd_mod:sd_ioctl+371} <ffffffff802a6dd6>{schedule_timeout+158}
       <ffffffff801bf181>{blkdev_ioctl+1365} <ffffffff80243cd2>{sys_sendto+251}
       <ffffffff801751e5>{__pollwait+0} <ffffffff8016b16a>{block_ioctl+25}
       <ffffffff801749f4>{do_ioctl+24} <ffffffff80174c46>{vfs_ioctl+541}
       <ffffffff80174cb4>{sys_ioctl+89} <ffffffff8010e4ba>{system_call+126}
       
Mem-info:
DMA per-cpu:
cpu 0 hot: low 0, high 0, batch 1 used:0
cpu 0 cold: low 0, high 0, batch 1 used:0
DMA32 per-cpu:
cpu 0 hot: low 0, high 186, batch 31 used:35
cpu 0 cold: low 0, high 62, batch 15 used:57
Normal per-cpu:
cpu 0 hot: low 0, high 186, batch 31 used:31
cpu 0 cold: low 0, high 62, batch 15 used:60
HighMem per-cpu: empty
Free pages:       20120kB (0kB HighMem)
Active:49696 inactive:905346 dirty:401104 writeback:10576 unstable:0 free:5030 slab:40994 mapped:49180 pagetables:1655
DMA free:20kB min:24kB low:28kB high:36kB active:0kB inactive:0kB present:12740kB pages_scanned:12 all_unreclaimable? yes
lowmem_reserve[]: 0 2999 4009 4009
DMA32 free:17032kB min:6052kB low:7564kB high:9076kB active:176kB inactive:2854812kB present:3071904kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 1010 1010
Normal free:3068kB min:2036kB low:2544kB high:3052kB active:198608kB inactive:766572kB present:1034240kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
DMA: 1*4kB 0*8kB 1*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 20kB
DMA32: 48*4kB 77*8kB 68*16kB 55*32kB 29*64kB 14*128kB 0*256kB 1*512kB 1*1024kB 0*2048kB 2*4096kB = 17032kB
Normal: 17*4kB 19*8kB 8*16kB 9*32kB 2*64kB 6*128kB 0*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 3068kB
HighMem: empty
Swap cache: add 38, delete 0, find 0/0, race 0+0
Free swap  = 9686816kB
Total swap = 9686968kB
Free swap:       9686816kB
1310720 pages of RAM
299467 reserved pages
955800 pages shared
38 pages swap cached
Out of Memory: Killed process 7054 (java).
oom-killer: gfp_mask=0xd1, order=0

Call Trace:<ffffffff8014d7bc>{out_of_memory+48} <ffffffff8014f4b0>{__alloc_pages+536}
       <ffffffff80169788>{bio_alloc_bioset+232} <ffffffff80169d03>{bio_copy_user+218}
       <ffffffff801bd674>{blk_rq_map_user+136} <ffffffff801c0024>{sg_io+328}
       <ffffffff801c0498>{scsi_cmd_ioctl+491} <ffffffff88005e22>{:ide_core:generic_ide_ioctl+631}
       <ffffffff88206d0c>{:sd_mod:sd_ioctl+371} <ffffffff802a6dd6>{schedule_timeout+158}
       <ffffffff801bf181>{blkdev_ioctl+1365} <ffffffff80243cd2>{sys_sendto+251}
       <ffffffff801751e5>{__pollwait+0} <ffffffff8016b16a>{block_ioctl+25}
       <ffffffff801749f4>{do_ioctl+24} <ffffffff80174c46>{vfs_ioctl+541}
       <ffffffff80174cb4>{sys_ioctl+89} <ffffffff8010e4ba>{system_call+126}
       
Mem-info:
DMA per-cpu:
cpu 0 hot: low 0, high 0, batch 1 used:0
cpu 0 cold: low 0, high 0, batch 1 used:0
DMA32 per-cpu:
cpu 0 hot: low 0, high 186, batch 31 used:49
cpu 0 cold: low 0, high 62, batch 15 used:57
Normal per-cpu:
cpu 0 hot: low 0, high 186, batch 31 used:171
cpu 0 cold: low 0, high 62, batch 15 used:60
HighMem per-cpu: empty
Free pages:       48660kB (0kB HighMem)
Active:42535 inactive:905346 dirty:401112 writeback:10064 unstable:0 free:12165 slab:40987 mapped:38361 pagetables:1578
DMA free:20kB min:24kB low:28kB high:36kB active:0kB inactive:0kB present:12740kB pages_scanned:14 all_unreclaimable? yes
lowmem_reserve[]: 0 2999 4009 4009
DMA32 free:17032kB min:6052kB low:7564kB high:9076kB active:176kB inactive:2854812kB present:3071904kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 1010 1010
Normal free:31608kB min:2036kB low:2544kB high:3052kB active:169964kB inactive:766572kB present:1034240kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
DMA: 1*4kB 0*8kB 1*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 20kB
DMA32: 48*4kB 77*8kB 68*16kB 55*32kB 29*64kB 14*128kB 0*256kB 1*512kB 1*1024kB 0*2048kB 2*4096kB = 17032kB
Normal: 454*4kB 328*8kB 222*16kB 148*32kB 99*64kB 40*128kB 13*256kB 4*512kB 2*1024kB 0*2048kB 0*4096kB = 31608kB
HighMem: empty
Swap cache: add 38, delete 0, find 0/0, race 0+0
Free swap  = 9686816kB
Total swap = 9686968kB
Free swap:       9686816kB
1310720 pages of RAM
299467 reserved pages
951709 pages shared
38 pages swap cached
oom-killer: gfp_mask=0xd1, order=0

Call Trace:<ffffffff8014d7bc>{out_of_memory+48} <ffffffff8014f4b0>{__alloc_pages+536}
       <ffffffff80169788>{bio_alloc_bioset+232} <ffffffff80169d03>{bio_copy_user+218}
       <ffffffff801bd674>{blk_rq_map_user+136} <ffffffff801c0024>{sg_io+328}
       <ffffffff801c0498>{scsi_cmd_ioctl+491} <ffffffff88005e22>{:ide_core:generic_ide_ioctl+631}
       <ffffffff88206d0c>{:sd_mod:sd_ioctl+371} <ffffffff802a6dd6>{schedule_timeout+158}
       <ffffffff801bf181>{blkdev_ioctl+1365} <ffffffff80243cd2>{sys_sendto+251}
       <ffffffff801751e5>{__pollwait+0} <ffffffff8016b16a>{block_ioctl+25}
       <ffffffff801749f4>{do_ioctl+24} <ffffffff80174c46>{vfs_ioctl+541}
       <ffffffff80174cb4>{sys_ioctl+89} <ffffffff8010e4ba>{system_call+126}
       
Mem-info:
DMA per-cpu:
cpu 0 hot: low 0, high 0, batch 1 used:0
cpu 0 cold: low 0, high 0, batch 1 used:0
DMA32 per-cpu:
cpu 0 hot: low 0, high 186, batch 31 used:56
cpu 0 cold: low 0, high 62, batch 15 used:57
Normal per-cpu:
cpu 0 hot: low 0, high 186, batch 31 used:171
cpu 0 cold: low 0, high 62, batch 15 used:60
HighMem per-cpu: empty
Free pages:       48660kB (0kB HighMem)
Active:42535 inactive:905346 dirty:401112 writeback:9808 unstable:0 free:12165 slab:40980 mapped:38361 pagetables:1578
DMA free:20kB min:24kB low:28kB high:36kB active:0kB inactive:0kB present:12740kB pages_scanned:14 all_unreclaimable? yes
lowmem_reserve[]: 0 2999 4009 4009
DMA32 free:17032kB min:6052kB low:7564kB high:9076kB active:176kB inactive:2854812kB present:3071904kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 1010 1010
Normal free:31608kB min:2036kB low:2544kB high:3052kB active:169964kB inactive:766572kB present:1034240kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
DMA: 1*4kB 0*8kB 1*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 20kB
DMA32: 48*4kB 77*8kB 68*16kB 55*32kB 29*64kB 14*128kB 0*256kB 1*512kB 1*1024kB 0*2048kB 2*4096kB = 17032kB
Normal: 454*4kB 328*8kB 222*16kB 148*32kB 99*64kB 40*128kB 13*256kB 4*512kB 2*1024kB 0*2048kB 0*4096kB = 31608kB
HighMem: empty
Swap cache: add 38, delete 0, find 0/0, race 0+0
Free swap  = 9686816kB
Total swap = 9686968kB
Free swap:       9686816kB
1310720 pages of RAM
299467 reserved pages
951709 pages shared
38 pages swap cached
oom-killer: gfp_mask=0xd1, order=0

Call Trace:<ffffffff8014d7bc>{out_of_memory+48} <ffffffff8014f4b0>{__alloc_pages+536}
       <ffffffff80169788>{bio_alloc_bioset+232} <ffffffff80169d03>{bio_copy_user+218}
       <ffffffff801bd674>{blk_rq_map_user+136} <ffffffff801c0024>{sg_io+328}
       <ffffffff801c0498>{scsi_cmd_ioctl+491} <ffffffff88005e22>{:ide_core:generic_ide_ioctl+631}
       <ffffffff88206d0c>{:sd_mod:sd_ioctl+371} <ffffffff802a6dd6>{schedule_timeout+158}
       <ffffffff801bf181>{blkdev_ioctl+1365} <ffffffff80243cd2>{sys_sendto+251}
       <ffffffff801751e5>{__pollwait+0} <ffffffff8016b16a>{block_ioctl+25}
       <ffffffff801749f4>{do_ioctl+24} <ffffffff80174c46>{vfs_ioctl+541}
       <ffffffff80174cb4>{sys_ioctl+89} <ffffffff8010e4ba>{system_call+126}
       
Mem-info:
DMA per-cpu:
cpu 0 hot: low 0, high 0, batch 1 used:0
cpu 0 cold: low 0, high 0, batch 1 used:0
DMA32 per-cpu:
cpu 0 hot: low 0, high 186, batch 31 used:64
cpu 0 cold: low 0, high 62, batch 15 used:57
Normal per-cpu:
cpu 0 hot: low 0, high 186, batch 31 used:171
cpu 0 cold: low 0, high 62, batch 15 used:60
HighMem per-cpu: empty
Free pages:       48660kB (0kB HighMem)
Active:42535 inactive:905346 dirty:401112 writeback:9552 unstable:0 free:12165 slab:40972 mapped:38361 pagetables:1578
DMA free:20kB min:24kB low:28kB high:36kB active:0kB inactive:0kB present:12740kB pages_scanned:14 all_unreclaimable? yes
lowmem_reserve[]: 0 2999 4009 4009
DMA32 free:17032kB min:6052kB low:7564kB high:9076kB active:176kB inactive:2854812kB present:3071904kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 1010 1010
Normal free:31608kB min:2036kB low:2544kB high:3052kB active:169964kB inactive:766572kB present:1034240kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
DMA: 1*4kB 0*8kB 1*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 20kB
DMA32: 48*4kB 77*8kB 68*16kB 55*32kB 29*64kB 14*128kB 0*256kB 1*512kB 1*1024kB 0*2048kB 2*4096kB = 17032kB
Normal: 454*4kB 328*8kB 222*16kB 148*32kB 99*64kB 40*128kB 13*256kB 4*512kB 2*1024kB 0*2048kB 0*4096kB = 31608kB
HighMem: empty
Swap cache: add 38, delete 0, find 0/0, race 0+0
Free swap  = 9686816kB
Total swap = 9686968kB
Free swap:       9686816kB
1310720 pages of RAM
299467 reserved pages
951709 pages shared
38 pages swap cached
oom-killer: gfp_mask=0xd1, order=0

Call Trace:<ffffffff8014d7bc>{out_of_memory+48} <ffffffff8014f4b0>{__alloc_pages+536}
       <ffffffff80169788>{bio_alloc_bioset+232} <ffffffff80169d03>{bio_copy_user+218}
       <ffffffff801bd674>{blk_rq_map_user+136} <ffffffff801c0024>{sg_io+328}
       <ffffffff801c0498>{scsi_cmd_ioctl+491} <ffffffff88005e22>{:ide_core:generic_ide_ioctl+631}
       <ffffffff88206d0c>{:sd_mod:sd_ioctl+371} <ffffffff802a6dd6>{schedule_timeout+158}
       <ffffffff801bf181>{blkdev_ioctl+1365} <ffffffff80243cd2>{sys_sendto+251}
       <ffffffff801751e5>{__pollwait+0} <ffffffff8016b16a>{block_ioctl+25}
       <ffffffff801749f4>{do_ioctl+24} <ffffffff80174c46>{vfs_ioctl+541}
       <ffffffff80174cb4>{sys_ioctl+89} <ffffffff8010e4ba>{system_call+126}
       
Mem-info:
DMA per-cpu:
cpu 0 hot: low 0, high 0, batch 1 used:0
cpu 0 cold: low 0, high 0, batch 1 used:0
DMA32 per-cpu:
cpu 0 hot: low 0, high 186, batch 31 used:74
cpu 0 cold: low 0, high 62, batch 15 used:57
Normal per-cpu:
cpu 0 hot: low 0, high 186, batch 31 used:180
cpu 0 cold: low 0, high 62, batch 15 used:60
HighMem per-cpu: empty
Free pages:       48660kB (0kB HighMem)
Active:42535 inactive:905346 dirty:401112 writeback:9040 unstable:0 free:12165 slab:40953 mapped:38361 pagetables:1578
DMA free:20kB min:24kB low:28kB high:36kB active:0kB inactive:0kB present:12740kB pages_scanned:14 all_unreclaimable? yes
lowmem_reserve[]: 0 2999 4009 4009
DMA32 free:17032kB min:6052kB low:7564kB high:9076kB active:176kB inactive:2854812kB present:3071904kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 1010 1010
Normal free:31608kB min:2036kB low:2544kB high:3052kB active:169964kB inactive:766572kB present:1034240kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
DMA: 1*4kB 0*8kB 1*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 20kB
DMA32: 48*4kB 77*8kB 68*16kB 55*32kB 29*64kB 14*128kB 0*256kB 1*512kB 1*1024kB 0*2048kB 2*4096kB = 17032kB
Normal: 454*4kB 328*8kB 222*16kB 148*32kB 99*64kB 40*128kB 13*256kB 4*512kB 2*1024kB 0*2048kB 0*4096kB = 31608kB
HighMem: empty
Swap cache: add 38, delete 0, find 0/0, race 0+0
Free swap  = 9686816kB
Total swap = 9686968kB
Free swap:       9686816kB
1310720 pages of RAM
299467 reserved pages
951709 pages shared
38 pages swap cached
oom-killer: gfp_mask=0xd1, order=0

Call Trace:<ffffffff8014d7bc>{out_of_memory+48} <ffffffff8014f4b0>{__alloc_pages+536}
       <ffffffff80169788>{bio_alloc_bioset+232} <ffffffff80169d03>{bio_copy_user+218}
       <ffffffff801bd674>{blk_rq_map_user+136} <ffffffff801c0024>{sg_io+328}
       <ffffffff801c0498>{scsi_cmd_ioctl+491} <ffffffff88005e22>{:ide_core:generic_ide_ioctl+631}
       <ffffffff88206d0c>{:sd_mod:sd_ioctl+371} <ffffffff802a6dd6>{schedule_timeout+158}
       <ffffffff801bf181>{blkdev_ioctl+1365} <ffffffff80243cd2>{sys_sendto+251}
       <ffffffff801751e5>{__pollwait+0} <ffffffff8016b16a>{block_ioctl+25}
       <ffffffff801749f4>{do_ioctl+24} <ffffffff80174c46>{vfs_ioctl+541}
       <ffffffff80174cb4>{sys_ioctl+89} <ffffffff8010e4ba>{system_call+126}
       
Mem-info:
DMA per-cpu:
cpu 0 hot: low 0, high 0, batch 1 used:0
cpu 0 cold: low 0, high 0, batch 1 used:0
DMA32 per-cpu:
cpu 0 hot: low 0, high 186, batch 31 used:74
cpu 0 cold: low 0, high 62, batch 15 used:57
Normal per-cpu:
cpu 0 hot: low 0, high 186, batch 31 used:159
cpu 0 cold: low 0, high 62, batch 15 used:60
HighMem per-cpu: empty
Free pages:       48784kB (0kB HighMem)
Active:42535 inactive:905346 dirty:401112 writeback:8784 unstable:0 free:12196 slab:40943 mapped:38361 pagetables:1578
DMA free:20kB min:24kB low:28kB high:36kB active:0kB inactive:0kB present:12740kB pages_scanned:14 all_unreclaimable? yes
lowmem_reserve[]: 0 2999 4009 4009
DMA32 free:17032kB min:6052kB low:7564kB high:9076kB active:176kB inactive:2854812kB present:3071904kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 1010 1010
Normal free:31732kB min:2036kB low:2544kB high:3052kB active:169964kB inactive:766572kB present:1034240kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
DMA: 1*4kB 0*8kB 1*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 20kB
DMA32: 48*4kB 77*8kB 68*16kB 55*32kB 29*64kB 14*128kB 0*256kB 1*512kB 1*1024kB 0*2048kB 2*4096kB = 17032kB
Normal: 453*4kB 328*8kB 222*16kB 148*32kB 99*64kB 41*128kB 13*256kB 4*512kB 2*1024kB 0*2048kB 0*4096kB = 31732kB
HighMem: empty
Swap cache: add 38, delete 0, find 0/0, race 0+0
Free swap  = 9686816kB
Total swap = 9686968kB
Free swap:       9686816kB
1310720 pages of RAM
299467 reserved pages
951709 pages shared
38 pages swap cached
oom-killer: gfp_mask=0xd1, order=0

Call Trace:<ffffffff8014d7bc>{out_of_memory+48} <ffffffff8014f4b0>{__alloc_pages+536}
       <ffffffff80169788>{bio_alloc_bioset+232} <ffffffff80169d03>{bio_copy_user+218}
       <ffffffff801bd674>{blk_rq_map_user+136} <ffffffff801c0024>{sg_io+328}
       <ffffffff801c0498>{scsi_cmd_ioctl+491} <ffffffff88005e22>{:ide_core:generic_ide_ioctl+631}
       <ffffffff88206d0c>{:sd_mod:sd_ioctl+371} <ffffffff802a6dd6>{schedule_timeout+158}
       <ffffffff801bf181>{blkdev_ioctl+1365} <ffffffff80243cd2>{sys_sendto+251}
       <ffffffff801751e5>{__pollwait+0} <ffffffff8016b16a>{block_ioctl+25}
       <ffffffff801749f4>{do_ioctl+24} <ffffffff80174c46>{vfs_ioctl+541}
       <ffffffff80174cb4>{sys_ioctl+89} <ffffffff8010e4ba>{system_call+126}
       
Mem-info:
DMA per-cpu:
cpu 0 hot: low 0, high 0, batch 1 used:0
cpu 0 cold: low 0, high 0, batch 1 used:0
DMA32 per-cpu:
cpu 0 hot: low 0, high 186, batch 31 used:74
cpu 0 cold: low 0, high 62, batch 15 used:57
Normal per-cpu:
cpu 0 hot: low 0, high 186, batch 31 used:170
cpu 0 cold: low 0, high 62, batch 15 used:60
HighMem per-cpu: empty
Free pages:       48784kB (0kB HighMem)
Active:42535 inactive:905346 dirty:401112 writeback:8528 unstable:0 free:12196 slab:40932 mapped:38361 pagetables:1578
DMA free:20kB min:24kB low:28kB high:36kB active:0kB inactive:0kB present:12740kB pages_scanned:14 all_unreclaimable? yes
lowmem_reserve[]: 0 2999 4009 4009
DMA32 free:17032kB min:6052kB low:7564kB high:9076kB active:176kB inactive:2854812kB present:3071904kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 1010 1010
Normal free:31732kB min:2036kB low:2544kB high:3052kB active:169964kB inactive:766572kB present:1034240kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
DMA: 1*4kB 0*8kB 1*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 20kB
DMA32: 48*4kB 77*8kB 68*16kB 55*32kB 29*64kB 14*128kB 0*256kB 1*512kB 1*1024kB 0*2048kB 2*4096kB = 17032kB
Normal: 453*4kB 328*8kB 222*16kB 148*32kB 99*64kB 41*128kB 13*256kB 4*512kB 2*1024kB 0*2048kB 0*4096kB = 31732kB
HighMem: empty
Swap cache: add 38, delete 0, find 0/0, race 0+0
Free swap  = 9686816kB
Total swap = 9686968kB
Free swap:       9686816kB
1310720 pages of RAM
299467 reserved pages
951709 pages shared
38 pages swap cached
oom-killer: gfp_mask=0xd1, order=0

Call Trace:<ffffffff8014d7bc>{out_of_memory+48} <ffffffff8014f4b0>{__alloc_pages+536}
       <ffffffff80169788>{bio_alloc_bioset+232} <ffffffff80169d03>{bio_copy_user+218}
       <ffffffff801bd674>{blk_rq_map_user+136} <ffffffff801c0024>{sg_io+328}
       <ffffffff801c0498>{scsi_cmd_ioctl+491} <ffffffff88005e22>{:ide_core:generic_ide_ioctl+631}
       <ffffffff88206d0c>{:sd_mod:sd_ioctl+371} <ffffffff802a6dd6>{schedule_timeout+158}
       <ffffffff801bf181>{blkdev_ioctl+1365} <ffffffff80243cd2>{sys_sendto+251}
       <ffffffff801751e5>{__pollwait+0} <ffffffff8016b16a>{block_ioctl+25}
       <ffffffff801749f4>{do_ioctl+24} <ffffffff80174c46>{vfs_ioctl+541}
       <ffffffff80174cb4>{sys_ioctl+89} <ffffffff8010e4ba>{system_call+126}
       
Mem-info:
DMA per-cpu:
cpu 0 hot: low 0, high 0, batch 1 used:0
cpu 0 cold: low 0, high 0, batch 1 used:0
DMA32 per-cpu:
cpu 0 hot: low 0, high 186, batch 31 used:74
cpu 0 cold: low 0, high 62, batch 15 used:57
Normal per-cpu:
cpu 0 hot: low 0, high 186, batch 31 used:179
cpu 0 cold: low 0, high 62, batch 15 used:60
HighMem per-cpu: empty
Free pages:       48784kB (0kB HighMem)
Active:42536 inactive:905349 dirty:401112 writeback:8272 unstable:0 free:12196 slab:40923 mapped:38361 pagetables:1578
DMA free:20kB min:24kB low:28kB high:36kB active:0kB inactive:0kB present:12740kB pages_scanned:16 all_unreclaimable? yes
lowmem_reserve[]: 0 2999 4009 4009
DMA32 free:17032kB min:6052kB low:7564kB high:9076kB active:176kB inactive:2854812kB present:3071904kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 1010 1010
Normal free:31732kB min:2036kB low:2544kB high:3052kB active:169968kB inactive:766584kB present:1034240kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
DMA: 1*4kB 0*8kB 1*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 20kB
DMA32: 48*4kB 77*8kB 68*16kB 55*32kB 29*64kB 14*128kB 0*256kB 1*512kB 1*1024kB 0*2048kB 2*4096kB = 17032kB
Normal: 453*4kB 328*8kB 222*16kB 148*32kB 99*64kB 41*128kB 13*256kB 4*512kB 2*1024kB 0*2048kB 0*4096kB = 31732kB
HighMem: empty
Swap cache: add 38, delete 0, find 0/0, race 0+0
Free swap  = 9686816kB
Total swap = 9686968kB
Free swap:       9686816kB
1310720 pages of RAM
299467 reserved pages
951705 pages shared
38 pages swap cached
oom-killer: gfp_mask=0xd1, order=0

Call Trace:<ffffffff8014d7bc>{out_of_memory+48} <ffffffff8014f4b0>{__alloc_pages+536}
       <ffffffff80169788>{bio_alloc_bioset+232} <ffffffff80169d03>{bio_copy_user+218}
       <ffffffff801bd674>{blk_rq_map_user+136} <ffffffff801c0024>{sg_io+328}
       <ffffffff801c0498>{scsi_cmd_ioctl+491} <ffffffff88005e22>{:ide_core:generic_ide_ioctl+631}
       <ffffffff88206d0c>{:sd_mod:sd_ioctl+371} <ffffffff802a6dd6>{schedule_timeout+158}
       <ffffffff801bf181>{blkdev_ioctl+1365} <ffffffff80243cd2>{sys_sendto+251}
       <ffffffff801751e5>{__pollwait+0} <ffffffff8016b16a>{block_ioctl+25}
       <ffffffff801749f4>{do_ioctl+24} <ffffffff80174c46>{vfs_ioctl+541}
       <ffffffff80174cb4>{sys_ioctl+89} <ffffffff8010e4ba>{system_call+126}
       
Mem-info:
DMA per-cpu:
cpu 0 hot: low 0, high 0, batch 1 used:0
cpu 0 cold: low 0, high 0, batch 1 used:0
DMA32 per-cpu:
cpu 0 hot: low 0, high 186, batch 31 used:74
cpu 0 cold: low 0, high 62, batch 15 used:57
Normal per-cpu:
cpu 0 hot: low 0, high 186, batch 31 used:157
cpu 0 cold: low 0, high 62, batch 15 used:60
HighMem per-cpu: empty
Free pages:       48908kB (0kB HighMem)
Active:42536 inactive:905349 dirty:401112 writeback:8016 unstable:0 free:12227 slab:40914 mapped:38361 pagetables:1578
DMA free:20kB min:24kB low:28kB high:36kB active:0kB inactive:0kB present:12740kB pages_scanned:16 all_unreclaimable? yes
lowmem_reserve[]: 0 2999 4009 4009
DMA32 free:17032kB min:6052kB low:7564kB high:9076kB active:176kB inactive:2854812kB present:3071904kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 1010 1010
Normal free:31856kB min:2036kB low:2544kB high:3052kB active:169968kB inactive:766584kB present:1034240kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
DMA: 1*4kB 0*8kB 1*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 20kB
DMA32: 48*4kB 77*8kB 68*16kB 55*32kB 29*64kB 14*128kB 0*256kB 1*512kB 1*1024kB 0*2048kB 2*4096kB = 17032kB
Normal: 452*4kB 324*8kB 218*16kB 143*32kB 97*64kB 39*128kB 14*256kB 5*512kB 2*1024kB 0*2048kB 0*4096kB = 31856kB
HighMem: empty
Swap cache: add 38, delete 0, find 0/0, race 0+0
Free swap  = 9686816kB
Total swap = 9686968kB
Free swap:       9686816kB
1310720 pages of RAM
299467 reserved pages
951705 pages shared
38 pages swap cached
oom-killer: gfp_mask=0xd1, order=0

Call Trace:<ffffffff8014d7bc>{out_of_memory+48} <ffffffff8014f4b0>{__alloc_pages+536}
       <ffffffff80169788>{bio_alloc_bioset+232} <ffffffff80169d03>{bio_copy_user+218}
       <ffffffff801bd674>{blk_rq_map_user+136} <ffffffff801c0024>{sg_io+328}
       <ffffffff801c0498>{scsi_cmd_ioctl+491} <ffffffff88005e22>{:ide_core:generic_ide_ioctl+631}
       <ffffffff88206d0c>{:sd_mod:sd_ioctl+371} <ffffffff802a6dd6>{schedule_timeout+158}
       <ffffffff801bf181>{blkdev_ioctl+1365} <ffffffff80243cd2>{sys_sendto+251}
       <ffffffff801751e5>{__pollwait+0} <ffffffff8016b16a>{block_ioctl+25}
       <ffffffff801749f4>{do_ioctl+24} <ffffffff80174c46>{vfs_ioctl+541}
       <ffffffff80174cb4>{sys_ioctl+89} <ffffffff8010e4ba>{system_call+126}
       
Mem-info:
DMA per-cpu:
cpu 0 hot: low 0, high 0, batch 1 used:0
cpu 0 cold: low 0, high 0, batch 1 used:0
DMA32 per-cpu:
cpu 0 hot: low 0, high 186, batch 31 used:74
cpu 0 cold: low 0, high 62, batch 15 used:57
Normal per-cpu:
cpu 0 hot: low 0, high 186, batch 31 used:178
cpu 0 cold: low 0, high 62, batch 15 used:60
HighMem per-cpu: empty
Free pages:       48908kB (0kB HighMem)
Active:42536 inactive:905349 dirty:401112 writeback:7504 unstable:0 free:12227 slab:40893 mapped:38361 pagetables:1578
DMA free:20kB min:24kB low:28kB high:36kB active:0kB inactive:0kB present:12740kB pages_scanned:16 all_unreclaimable? yes
lowmem_reserve[]: 0 2999 4009 4009
DMA32 free:17032kB min:6052kB low:7564kB high:9076kB active:176kB inactive:2854812kB present:3071904kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 1010 1010
Normal free:31856kB min:2036kB low:2544kB high:3052kB active:169968kB inactive:766584kB present:1034240kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
DMA: 1*4kB 0*8kB 1*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 20kB
DMA32: 48*4kB 77*8kB 68*16kB 55*32kB 29*64kB 14*128kB 0*256kB 1*512kB 1*1024kB 0*2048kB 2*4096kB = 17032kB
Normal: 452*4kB 324*8kB 218*16kB 143*32kB 97*64kB 39*128kB 14*256kB 5*512kB 2*1024kB 0*2048kB 0*4096kB = 31856kB
HighMem: empty
Swap cache: add 38, delete 0, find 0/0, race 0+0
Free swap  = 9686816kB
Total swap = 9686968kB
Free swap:       9686816kB
1310720 pages of RAM
299467 reserved pages
951705 pages shared
38 pages swap cached
printk: 116 messages suppressed.
oom-killer: gfp_mask=0xd1, order=0

Call Trace:<ffffffff8014d7bc>{out_of_memory+48} <ffffffff8014f4b0>{__alloc_pages+536}
       <ffffffff80169788>{bio_alloc_bioset+232} <ffffffff80169d03>{bio_copy_user+218}
       <ffffffff801bd674>{blk_rq_map_user+136} <ffffffff801c0024>{sg_io+328}
       <ffffffff801c0498>{scsi_cmd_ioctl+491} <ffffffff88005e22>{:ide_core:generic_ide_ioctl+631}
       <ffffffff88206d0c>{:sd_mod:sd_ioctl+371} <ffffffff802a6dd6>{schedule_timeout+158}
       <ffffffff801bf181>{blkdev_ioctl+1365} <ffffffff80243cd2>{sys_sendto+251}
       <ffffffff801751e5>{__pollwait+0} <ffffffff8016b16a>{block_ioctl+25}
       <ffffffff801749f4>{do_ioctl+24} <ffffffff80174c46>{vfs_ioctl+541}
       <ffffffff80174cb4>{sys_ioctl+89} <ffffffff8010e4ba>{system_call+126}
       
Mem-info:
DMA per-cpu:
cpu 0 hot: low 0, high 0, batch 1 used:0
cpu 0 cold: low 0, high 0, batch 1 used:0
DMA32 per-cpu:
cpu 0 hot: low 0, high 186, batch 31 used:176
cpu 0 cold: low 0, high 62, batch 15 used:9
Normal per-cpu:
cpu 0 hot: low 0, high 186, batch 31 used:169
cpu 0 cold: low 0, high 62, batch 15 used:10
HighMem per-cpu: empty
Free pages:     2937644kB (0kB HighMem)
Active:234053 inactive:21105 dirty:85603 writeback:3267 unstable:0 free:734411 slab:11180 mapped:40545 pagetables:1672
DMA free:20kB min:24kB low:28kB high:36kB active:0kB inactive:0kB present:12740kB pages_scanned:0 all_unreclaimable? yes
lowmem_reserve[]: 0 2999 4009 4009
DMA32 free:2474820kB min:6052kB low:7564kB high:9076kB active:491432kB inactive:920kB present:3071904kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 1010 1010
Normal free:462804kB min:2036kB low:2544kB high:3052kB active:444780kB inactive:83500kB present:1034240kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
DMA: 1*4kB 0*8kB 1*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 20kB
DMA32: 5617*4kB 5710*8kB 5539*16kB 5003*32kB 3444*64kB 943*128kB 1009*256kB 594*512kB 409*1024kB 242*2048kB 83*4096kB = 2474820kB
Normal: 4939*4kB 111*8kB 1539*16kB 5268*32kB 2514*64kB 556*128kB 58*256kB 2*512kB 1*1024kB 0*2048kB 0*4096kB = 462804kB
HighMem: empty
Swap cache: add 38, delete 38, find 0/0, race 0+0
Free swap  = 9686816kB
Total swap = 9686968kB
Free swap:       9686816kB
1310720 pages of RAM
299467 reserved pages
238615 pages shared
0 pages swap cached
Out of Memory: Killed process 6451 (nautilus).
oom-killer: gfp_mask=0xd1, order=0

Call Trace:<ffffffff8014d7bc>{out_of_memory+48} <ffffffff8014f4b0>{__alloc_pages+536}
       <ffffffff80169788>{bio_alloc_bioset+232} <ffffffff80169d03>{bio_copy_user+218}
       <ffffffff801bd674>{blk_rq_map_user+136} <ffffffff801c0024>{sg_io+328}
       <ffffffff801c0498>{scsi_cmd_ioctl+491} <ffffffff88005e22>{:ide_core:generic_ide_ioctl+631}
       <ffffffff88206d0c>{:sd_mod:sd_ioctl+371} <ffffffff802a6dd6>{schedule_timeout+158}
       <ffffffff801bf181>{blkdev_ioctl+1365} <ffffffff80243cd2>{sys_sendto+251}
       <ffffffff801751e5>{__pollwait+0} <ffffffff8016b16a>{block_ioctl+25}
       <ffffffff801749f4>{do_ioctl+24} <ffffffff80174c46>{vfs_ioctl+541}
       <ffffffff80174cb4>{sys_ioctl+89} <ffffffff8010e4ba>{system_call+126}
       
Mem-info:
DMA per-cpu:
cpu 0 hot: low 0, high 0, batch 1 used:0
cpu 0 cold: low 0, high 0, batch 1 used:0
DMA32 per-cpu:
cpu 0 hot: low 0, high 186, batch 31 used:176
cpu 0 cold: low 0, high 62, batch 15 used:9
Normal per-cpu:
cpu 0 hot: low 0, high 186, batch 31 used:155
cpu 0 cold: low 0, high 62, batch 15 used:10
HighMem per-cpu: empty
Free pages:     2944280kB (0kB HighMem)
Active:232520 inactive:21107 dirty:84963 writeback:3267 unstable:0 free:736070 slab:11173 mapped:38188 pagetables:1599
DMA free:20kB min:24kB low:28kB high:36kB active:0kB inactive:0kB present:12740kB pages_scanned:0 all_unreclaimable? yes
lowmem_reserve[]: 0 2999 4009 4009
DMA32 free:2474820kB min:6052kB low:7564kB high:9076kB active:491432kB inactive:920kB present:3071904kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 1010 1010
Normal free:469440kB min:2036kB low:2544kB high:3052kB active:438648kB inactive:83508kB present:1034240kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
DMA: 1*4kB 0*8kB 1*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 20kB
DMA32: 5617*4kB 5710*8kB 5539*16kB 5003*32kB 3444*64kB 943*128kB 1009*256kB 594*512kB 409*1024kB 242*2048kB 83*4096kB = 2474820kB
Normal: 5176*4kB 246*8kB 1593*16kB 5305*32kB 2528*64kB 561*128kB 60*256kB 3*512kB 1*1024kB 0*2048kB 0*4096kB = 469440kB
HighMem: empty
Swap cache: add 38, delete 38, find 0/0, race 0+0
Free swap  = 9686816kB
Total swap = 9686968kB
Free swap:       9686816kB
1310720 pages of RAM
299467 reserved pages
235419 pages shared
0 pages swap cached
Out of Memory: Killed process 6475 (clock-applet).
oom-killer: gfp_mask=0xd1, order=0

Call Trace:<ffffffff8014d7bc>{out_of_memory+48} <ffffffff8014f4b0>{__alloc_pages+536}
       <ffffffff80169788>{bio_alloc_bioset+232} <ffffffff80169d03>{bio_copy_user+218}
       <ffffffff801bd674>{blk_rq_map_user+136} <ffffffff801c0024>{sg_io+328}
       <ffffffff801c0498>{scsi_cmd_ioctl+491} <ffffffff88005e22>{:ide_core:generic_ide_ioctl+631}
       <ffffffff88206d0c>{:sd_mod:sd_ioctl+371} <ffffffff802a6dd6>{schedule_timeout+158}
       <ffffffff801bf181>{blkdev_ioctl+1365} <ffffffff80243cd2>{sys_sendto+251}
       <ffffffff801751e5>{__pollwait+0} <ffffffff8016b16a>{block_ioctl+25}
       <ffffffff801749f4>{do_ioctl+24} <ffffffff80174c46>{vfs_ioctl+541}
       <ffffffff80174cb4>{sys_ioctl+89} <ffffffff8010e4ba>{system_call+126}
       
Mem-info:
DMA per-cpu:
cpu 0 hot: low 0, high 0, batch 1 used:0
cpu 0 cold: low 0, high 0, batch 1 used:0
DMA32 per-cpu:
cpu 0 hot: low 0, high 186, batch 31 used:185
cpu 0 cold: low 0, high 62, batch 15 used:9
Normal per-cpu:
cpu 0 hot: low 0, high 186, batch 31 used:58
cpu 0 cold: low 0, high 62, batch 15 used:10
HighMem per-cpu: empty
Free pages:     2947628kB (0kB HighMem)
Active:231721 inactive:21107 dirty:84451 writeback:3267 unstable:0 free:736907 slab:11167 mapped:37179 pagetables:1633
DMA free:20kB min:24kB low:28kB high:36kB active:0kB inactive:0kB present:12740kB pages_scanned:0 all_unreclaimable? yes
lowmem_reserve[]: 0 2999 4009 4009
DMA32 free:2474820kB min:6052kB low:7564kB high:9076kB active:491432kB inactive:920kB present:3071904kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 1010 1010
Normal free:472788kB min:2036kB low:2544kB high:3052kB active:435452kB inactive:83508kB present:1034240kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
DMA: 1*4kB 0*8kB 1*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 20kB
DMA32: 5617*4kB 5710*8kB 5539*16kB 5003*32kB 3444*64kB 943*128kB 1009*256kB 594*512kB 409*1024kB 242*2048kB 83*4096kB = 2474820kB
Normal: 5275*4kB 297*8kB 1616*16kB 5317*32kB 2536*64kB 567*128kB 62*256kB 3*512kB 1*1024kB 0*2048kB 0*4096kB = 472788kB
HighMem: empty
Swap cache: add 38, delete 38, find 0/0, race 0+0
Free swap  = 9686816kB
Total swap = 9686968kB
Free swap:       9686816kB
1310720 pages of RAM
299467 reserved pages
235131 pages shared
0 pages swap cached
Out of Memory: Killed process 6413 (ssh-agent).
oom-killer: gfp_mask=0xd1, order=0

Call Trace:<ffffffff8014d7bc>{out_of_memory+48} <ffffffff8014f4b0>{__alloc_pages+536}
       <ffffffff80169788>{bio_alloc_bioset+232} <ffffffff80169d03>{bio_copy_user+218}
       <ffffffff801bd674>{blk_rq_map_user+136} <ffffffff801c0024>{sg_io+328}
       <ffffffff801c0498>{scsi_cmd_ioctl+491} <ffffffff88005e22>{:ide_core:generic_ide_ioctl+631}
       <ffffffff88206d0c>{:sd_mod:sd_ioctl+371} <ffffffff802a6dd6>{schedule_timeout+158}
       <ffffffff801bf181>{blkdev_ioctl+1365} <ffffffff80243cd2>{sys_sendto+251}
       <ffffffff801751e5>{__pollwait+0} <ffffffff8016b16a>{block_ioctl+25}
       <ffffffff801749f4>{do_ioctl+24} <ffffffff80174c46>{vfs_ioctl+541}
       <ffffffff80174cb4>{sys_ioctl+89} <ffffffff8010e4ba>{system_call+126}
       
Mem-info:
DMA per-cpu:
cpu 0 hot: low 0, high 0, batch 1 used:0
cpu 0 cold: low 0, high 0, batch 1 used:0
DMA32 per-cpu:
cpu 0 hot: low 0, high 186, batch 31 used:155
cpu 0 cold: low 0, high 62, batch 15 used:9
Normal per-cpu:
cpu 0 hot: low 0, high 186, batch 31 used:157
cpu 0 cold: low 0, high 62, batch 15 used:10
HighMem per-cpu: empty
Free pages:     2999552kB (0kB HighMem)
Active:218989 inactive:21102 dirty:81156 writeback:3259 unstable:0 free:749888 slab:10888 mapped:37088 pagetables:1621
DMA free:20kB min:24kB low:28kB high:36kB active:0kB inactive:0kB present:12740kB pages_scanned:0 all_unreclaimable? yes
lowmem_reserve[]: 0 2999 4009 4009
DMA32 free:2474944kB min:6052kB low:7564kB high:9076kB active:491432kB inactive:920kB present:3071904kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 1010 1010
Normal free:524588kB min:2036kB low:2544kB high:3052kB active:384524kB inactive:83488kB present:1034240kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
DMA: 1*4kB 0*8kB 1*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 20kB
DMA32: 5600*4kB 5702*8kB 5533*16kB 5000*32kB 3441*64kB 944*128kB 1007*256kB 596*512kB 409*1024kB 242*2048kB 83*4096kB = 2474944kB
Normal: 5403*4kB 3298*8kB 3305*16kB 5317*32kB 2536*64kB 567*128kB 63*256kB 3*512kB 1*1024kB 0*2048kB 0*4096kB = 524588kB
HighMem: empty
Swap cache: add 38, delete 38, find 0/0, race 0+0
Free swap  = 9686816kB
Total swap = 9686968kB
Free swap:       9686816kB
1310720 pages of RAM
299467 reserved pages
222355 pages shared
0 pages swap cached
Out of Memory: Killed process 24590 (gnome-session).
oom-killer: gfp_mask=0xd1, order=0

Call Trace:<ffffffff8014d7bc>{out_of_memory+48} <ffffffff8014f4b0>{__alloc_pages+536}
       <ffffffff80169788>{bio_alloc_bioset+232} <ffffffff80169d03>{bio_copy_user+218}
       <ffffffff801bd674>{blk_rq_map_user+136} <ffffffff801c0024>{sg_io+328}
       <ffffffff801c0498>{scsi_cmd_ioctl+491} <ffffffff88005e22>{:ide_core:generic_ide_ioctl+631}
       <ffffffff88206d0c>{:sd_mod:sd_ioctl+371} <ffffffff802a6dd6>{schedule_timeout+158}
       <ffffffff801bf181>{blkdev_ioctl+1365} <ffffffff80243cd2>{sys_sendto+251}
       <ffffffff801751e5>{__pollwait+0} <ffffffff8016b16a>{block_ioctl+25}
       <ffffffff801749f4>{do_ioctl+24} <ffffffff80174c46>{vfs_ioctl+541}
       <ffffffff80174cb4>{sys_ioctl+89} <ffffffff8010e4ba>{system_call+126}
       
Mem-info:
DMA per-cpu:
cpu 0 hot: low 0, high 0, batch 1 used:0
cpu 0 cold: low 0, high 0, batch 1 used:0
DMA32 per-cpu:
cpu 0 hot: low 0, high 186, batch 31 used:155
cpu 0 cold: low 0, high 62, batch 15 used:9
Normal per-cpu:
cpu 0 hot: low 0, high 186, batch 31 used:131
cpu 0 cold: low 0, high 62, batch 15 used:10
HighMem per-cpu: empty
Free pages:     2999808kB (0kB HighMem)
Active:219006 inactive:21102 dirty:81148 writeback:3267 unstable:0 free:749952 slab:10888 mapped:37105 pagetables:1575
DMA free:20kB min:24kB low:28kB high:36kB active:0kB inactive:0kB present:12740kB pages_scanned:0 all_unreclaimable? yes
lowmem_reserve[]: 0 2999 4009 4009
DMA32 free:2474944kB min:6052kB low:7564kB high:9076kB active:491432kB inactive:920kB present:3071904kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 1010 1010
Normal free:524844kB min:2036kB low:2544kB high:3052kB active:384592kB inactive:83488kB present:1034240kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
DMA: 1*4kB 0*8kB 1*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 20kB
DMA32: 5600*4kB 5702*8kB 5533*16kB 5000*32kB 3441*64kB 944*128kB 1007*256kB 596*512kB 409*1024kB 242*2048kB 83*4096kB = 2474944kB
Normal: 5405*4kB 3297*8kB 3319*16kB 5316*32kB 2535*64kB 568*128kB 63*256kB 3*512kB 1*1024kB 0*2048kB 0*4096kB = 524844kB
HighMem: empty
Swap cache: add 38, delete 38, find 0/0, race 0+0
Free swap  = 9686816kB
Total swap = 9686968kB
Free swap:       9686816kB
1310720 pages of RAM
299467 reserved pages
221469 pages shared
0 pages swap cached
Out of Memory: Killed process 6430 (gnome-settings-).
oom-killer: gfp_mask=0xd1, order=0

Call Trace:<ffffffff8014d7bc>{out_of_memory+48} <ffffffff8014f4b0>{__alloc_pages+536}
       <ffffffff80169788>{bio_alloc_bioset+232} <ffffffff80169d03>{bio_copy_user+218}
       <ffffffff801bd674>{blk_rq_map_user+136} <ffffffff801c0024>{sg_io+328}
       <ffffffff801c0498>{scsi_cmd_ioctl+491} <ffffffff88005e22>{:ide_core:generic_ide_ioctl+631}
       <ffffffff88206d0c>{:sd_mod:sd_ioctl+371} <ffffffff802a6dd6>{schedule_timeout+158}
       <ffffffff801bf181>{blkdev_ioctl+1365} <ffffffff80243cd2>{sys_sendto+251}
       <ffffffff801751e5>{__pollwait+0} <ffffffff8016b16a>{block_ioctl+25}
       <ffffffff801749f4>{do_ioctl+24} <ffffffff80174c46>{vfs_ioctl+541}
       <ffffffff80174cb4>{sys_ioctl+89} <ffffffff8010e4ba>{system_call+126}
       
Mem-info:
DMA per-cpu:
cpu 0 hot: low 0, high 0, batch 1 used:0
cpu 0 cold: low 0, high 0, batch 1 used:0
DMA32 per-cpu:
cpu 0 hot: low 0, high 186, batch 31 used:155
cpu 0 cold: low 0, high 62, batch 15 used:9
Normal per-cpu:
cpu 0 hot: low 0, high 186, batch 31 used:132
cpu 0 cold: low 0, high 62, batch 15 used:10
HighMem per-cpu: empty
Free pages:     2999808kB (0kB HighMem)
Active:219006 inactive:21102 dirty:80636 writeback:3267 unstable:0 free:749952 slab:10888 mapped:37105 pagetables:1575
DMA free:20kB min:24kB low:28kB high:36kB active:0kB inactive:0kB present:12740kB pages_scanned:0 all_unreclaimable? yes
lowmem_reserve[]: 0 2999 4009 4009
DMA32 free:2474944kB min:6052kB low:7564kB high:9076kB active:491432kB inactive:920kB present:3071904kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 1010 1010
Normal free:524844kB min:2036kB low:2544kB high:3052kB active:384592kB inactive:83488kB present:1034240kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
DMA: 1*4kB 0*8kB 1*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 20kB
DMA32: 5600*4kB 5702*8kB 5533*16kB 5000*32kB 3441*64kB 944*128kB 1007*256kB 596*512kB 409*1024kB 242*2048kB 83*4096kB = 2474944kB
Normal: 5405*4kB 3297*8kB 3319*16kB 5316*32kB 2535*64kB 568*128kB 63*256kB 3*512kB 1*1024kB 0*2048kB 0*4096kB = 524844kB
HighMem: empty
Swap cache: add 38, delete 38, find 0/0, race 0+0
Free swap  = 9686816kB
Total swap = 9686968kB
Free swap:       9686816kB
1310720 pages of RAM
299467 reserved pages
221469 pages shared
0 pages swap cached
Out of Memory: Killed process 6438 (gnome-settings-).
oom-killer: gfp_mask=0xd1, order=0

Call Trace:<ffffffff8014d7bc>{out_of_memory+48} <ffffffff8014f4b0>{__alloc_pages+536}
       <ffffffff80169788>{bio_alloc_bioset+232} <ffffffff80169d03>{bio_copy_user+218}
       <ffffffff801bd674>{blk_rq_map_user+136} <ffffffff801c0024>{sg_io+328}
       <ffffffff801c0498>{scsi_cmd_ioctl+491} <ffffffff88005e22>{:ide_core:generic_ide_ioctl+631}
       <ffffffff88206d0c>{:sd_mod:sd_ioctl+371} <ffffffff802a6dd6>{schedule_timeout+158}
       <ffffffff801bf181>{blkdev_ioctl+1365} <ffffffff80243cd2>{sys_sendto+251}
       <ffffffff801751e5>{__pollwait+0} <ffffffff8016b16a>{block_ioctl+25}
       <ffffffff801749f4>{do_ioctl+24} <ffffffff80174c46>{vfs_ioctl+541}
       <ffffffff80174cb4>{sys_ioctl+89} <ffffffff8010e4ba>{system_call+126}
       
Mem-info:
DMA per-cpu:
cpu 0 hot: low 0, high 0, batch 1 used:0
cpu 0 cold: low 0, high 0, batch 1 used:0
DMA32 per-cpu:
cpu 0 hot: low 0, high 186, batch 31 used:155
cpu 0 cold: low 0, high 62, batch 15 used:9
Normal per-cpu:
cpu 0 hot: low 0, high 186, batch 31 used:132
cpu 0 cold: low 0, high 62, batch 15 used:10
HighMem per-cpu: empty
Free pages:     2999808kB (0kB HighMem)
Active:219006 inactive:21102 dirty:80252 writeback:3267 unstable:0 free:749952 slab:10888 mapped:37105 pagetables:1575
DMA free:20kB min:24kB low:28kB high:36kB active:0kB inactive:0kB present:12740kB pages_scanned:2 all_unreclaimable? yes
lowmem_reserve[]: 0 2999 4009 4009
DMA32 free:2474944kB min:6052kB low:7564kB high:9076kB active:491432kB inactive:920kB present:3071904kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 1010 1010
Normal free:524844kB min:2036kB low:2544kB high:3052kB active:384592kB inactive:83488kB present:1034240kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
DMA: 1*4kB 0*8kB 1*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 20kB
DMA32: 5600*4kB 5702*8kB 5533*16kB 5000*32kB 3441*64kB 944*128kB 1007*256kB 596*512kB 409*1024kB 242*2048kB 83*4096kB = 2474944kB
Normal: 5405*4kB 3297*8kB 3319*16kB 5316*32kB 2535*64kB 568*128kB 63*256kB 3*512kB 1*1024kB 0*2048kB 0*4096kB = 524844kB
HighMem: empty
Swap cache: add 38, delete 38, find 0/0, race 0+0
Free swap  = 9686816kB
Total swap = 9686968kB
Free swap:       9686816kB
1310720 pages of RAM
299467 reserved pages
221469 pages shared
0 pages swap cached
oom-killer: gfp_mask=0xd1, order=0

Call Trace:<ffffffff8014d7bc>{out_of_memory+48} <ffffffff8014f4b0>{__alloc_pages+536}
       <ffffffff80169788>{bio_alloc_bioset+232} <ffffffff80169d03>{bio_copy_user+218}
       <ffffffff801bd674>{blk_rq_map_user+136} <ffffffff801c0024>{sg_io+328}
       <ffffffff801c0498>{scsi_cmd_ioctl+491} <ffffffff88005e22>{:ide_core:generic_ide_ioctl+631}
       <ffffffff88206d0c>{:sd_mod:sd_ioctl+371} <ffffffff802a6dd6>{schedule_timeout+158}
       <ffffffff801bf181>{blkdev_ioctl+1365} <ffffffff80243cd2>{sys_sendto+251}
       <ffffffff801751e5>{__pollwait+0} <ffffffff8016b16a>{block_ioctl+25}
       <ffffffff801749f4>{do_ioctl+24} <ffffffff80174c46>{vfs_ioctl+541}
       <ffffffff80174cb4>{sys_ioctl+89} <ffffffff8010e4ba>{system_call+126}
       
Mem-info:
DMA per-cpu:
cpu 0 hot: low 0, high 0, batch 1 used:0
cpu 0 cold: low 0, high 0, batch 1 used:0
DMA32 per-cpu:
cpu 0 hot: low 0, high 186, batch 31 used:155
cpu 0 cold: low 0, high 62, batch 15 used:9
Normal per-cpu:
cpu 0 hot: low 0, high 186, batch 31 used:132
cpu 0 cold: low 0, high 62, batch 15 used:10
HighMem per-cpu: empty
Free pages:     2999808kB (0kB HighMem)
Active:219006 inactive:21102 dirty:79868 writeback:3267 unstable:0 free:749952 slab:10888 mapped:37105 pagetables:1575
DMA free:20kB min:24kB low:28kB high:36kB active:0kB inactive:0kB present:12740kB pages_scanned:0 all_unreclaimable? yes
lowmem_reserve[]: 0 2999 4009 4009
DMA32 free:2474944kB min:6052kB low:7564kB high:9076kB active:491432kB inactive:920kB present:3071904kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 1010 1010
Normal free:524844kB min:2036kB low:2544kB high:3052kB active:384592kB inactive:83488kB present:1034240kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
DMA: 1*4kB 0*8kB 1*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 20kB
DMA32: 5600*4kB 5702*8kB 5533*16kB 5000*32kB 3441*64kB 944*128kB 1007*256kB 596*512kB 409*1024kB 242*2048kB 83*4096kB = 2474944kB
Normal: 5405*4kB 3297*8kB 3319*16kB 5316*32kB 2535*64kB 568*128kB 63*256kB 3*512kB 1*1024kB 0*2048kB 0*4096kB = 524844kB
HighMem: empty
Swap cache: add 38, delete 38, find 0/0, race 0+0
Free swap  = 9686816kB
Total swap = 9686968kB
Free swap:       9686816kB
1310720 pages of RAM
299467 reserved pages
221469 pages shared
0 pages swap cached
oom-killer: gfp_mask=0xd1, order=0

Call Trace:<ffffffff8014d7bc>{out_of_memory+48} <ffffffff8014f4b0>{__alloc_pages+536}
       <ffffffff80169788>{bio_alloc_bioset+232} <ffffffff80169d03>{bio_copy_user+218}
       <ffffffff801bd674>{blk_rq_map_user+136} <ffffffff801c0024>{sg_io+328}
       <ffffffff801c0498>{scsi_cmd_ioctl+491} <ffffffff88005e22>{:ide_core:generic_ide_ioctl+631}
       <ffffffff88206d0c>{:sd_mod:sd_ioctl+371} <ffffffff802a6dd6>{schedule_timeout+158}
       <ffffffff801bf181>{blkdev_ioctl+1365} <ffffffff80243cd2>{sys_sendto+251}
       <ffffffff801751e5>{__pollwait+0} <ffffffff8016b16a>{block_ioctl+25}
       <ffffffff801749f4>{do_ioctl+24} <ffffffff80174c46>{vfs_ioctl+541}
       <ffffffff80174cb4>{sys_ioctl+89} <ffffffff8010e4ba>{system_call+126}
       
Mem-info:
DMA per-cpu:
cpu 0 hot: low 0, high 0, batch 1 used:0
cpu 0 cold: low 0, high 0, batch 1 used:0
DMA32 per-cpu:
cpu 0 hot: low 0, high 186, batch 31 used:155
cpu 0 cold: low 0, high 62, batch 15 used:9
Normal per-cpu:
cpu 0 hot: low 0, high 186, batch 31 used:132
cpu 0 cold: low 0, high 62, batch 15 used:10
HighMem per-cpu: empty
Free pages:     2999808kB (0kB HighMem)
Active:219006 inactive:21102 dirty:79484 writeback:3267 unstable:0 free:749952 slab:10888 mapped:37105 pagetables:1575
DMA free:20kB min:24kB low:28kB high:36kB active:0kB inactive:0kB present:12740kB pages_scanned:0 all_unreclaimable? yes
lowmem_reserve[]: 0 2999 4009 4009
DMA32 free:2474944kB min:6052kB low:7564kB high:9076kB active:491432kB inactive:920kB present:3071904kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 1010 1010
Normal free:524844kB min:2036kB low:2544kB high:3052kB active:384592kB inactive:83488kB present:1034240kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
DMA: 1*4kB 0*8kB 1*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 20kB
DMA32: 5600*4kB 5702*8kB 5533*16kB 5000*32kB 3441*64kB 944*128kB 1007*256kB 596*512kB 409*1024kB 242*2048kB 83*4096kB = 2474944kB
Normal: 5405*4kB 3297*8kB 3319*16kB 5316*32kB 2535*64kB 568*128kB 63*256kB 3*512kB 1*1024kB 0*2048kB 0*4096kB = 524844kB
HighMem: empty
Swap cache: add 38, delete 38, find 0/0, race 0+0
Free swap  = 9686816kB
Total swap = 9686968kB
Free swap:       9686816kB
1310720 pages of RAM
299467 reserved pages
221469 pages shared
0 pages swap cached
oom-killer: gfp_mask=0xd1, order=0

Call Trace:<ffffffff8014d7bc>{out_of_memory+48} <ffffffff8014f4b0>{__alloc_pages+536}
       <ffffffff80169788>{bio_alloc_bioset+232} <ffffffff80169d03>{bio_copy_user+218}
       <ffffffff801bd674>{blk_rq_map_user+136} <ffffffff801c0024>{sg_io+328}
       <ffffffff801c0498>{scsi_cmd_ioctl+491} <ffffffff88005e22>{:ide_core:generic_ide_ioctl+631}
       <ffffffff88206d0c>{:sd_mod:sd_ioctl+371} <ffffffff802a6dd6>{schedule_timeout+158}
       <ffffffff801bf181>{blkdev_ioctl+1365} <ffffffff80243cd2>{sys_sendto+251}
       <ffffffff801751e5>{__pollwait+0} <ffffffff8016b16a>{block_ioctl+25}
       <ffffffff801749f4>{do_ioctl+24} <ffffffff80174c46>{vfs_ioctl+541}
       <ffffffff80174cb4>{sys_ioctl+89} <ffffffff8010e4ba>{system_call+126}
       
Mem-info:
DMA per-cpu:
cpu 0 hot: low 0, high 0, batch 1 used:0
cpu 0 cold: low 0, high 0, batch 1 used:0
DMA32 per-cpu:
cpu 0 hot: low 0, high 186, batch 31 used:155
cpu 0 cold: low 0, high 62, batch 15 used:9
Normal per-cpu:
cpu 0 hot: low 0, high 186, batch 31 used:132
cpu 0 cold: low 0, high 62, batch 15 used:10
HighMem per-cpu: empty
Free pages:     2999808kB (0kB HighMem)
Active:219006 inactive:21102 dirty:79100 writeback:3267 unstable:0 free:749952 slab:10888 mapped:37105 pagetables:1575
DMA free:20kB min:24kB low:28kB high:36kB active:0kB inactive:0kB present:12740kB pages_scanned:0 all_unreclaimable? yes
lowmem_reserve[]: 0 2999 4009 4009
DMA32 free:2474944kB min:6052kB low:7564kB high:9076kB active:491432kB inactive:920kB present:3071904kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 1010 1010
Normal free:524844kB min:2036kB low:2544kB high:3052kB active:384592kB inactive:83488kB present:1034240kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
DMA: 1*4kB 0*8kB 1*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 20kB
DMA32: 5600*4kB 5702*8kB 5533*16kB 5000*32kB 3441*64kB 944*128kB 1007*256kB 596*512kB 409*1024kB 242*2048kB 83*4096kB = 2474944kB
Normal: 5405*4kB 3297*8kB 3319*16kB 5316*32kB 2535*64kB 568*128kB 63*256kB 3*512kB 1*1024kB 0*2048kB 0*4096kB = 524844kB
HighMem: empty
Swap cache: add 38, delete 38, find 0/0, race 0+0
Free swap  = 9686816kB
Total swap = 9686968kB
Free swap:       9686816kB
1310720 pages of RAM
299467 reserved pages
221469 pages shared
0 pages swap cached
Out of Memory: Killed process 6477 (mixer_applet2).
printk: 90 messages suppressed.
oom-killer: gfp_mask=0xd1, order=0

Call Trace:<ffffffff8014d7bc>{out_of_memory+48} <ffffffff8014f4b0>{__alloc_pages+536}
       <ffffffff80169788>{bio_alloc_bioset+232} <ffffffff80169d03>{bio_copy_user+218}
       <ffffffff801bd674>{blk_rq_map_user+136} <ffffffff801c0024>{sg_io+328}
       <ffffffff801c0498>{scsi_cmd_ioctl+491} <ffffffff88005e22>{:ide_core:generic_ide_ioctl+631}
       <ffffffff88206d0c>{:sd_mod:sd_ioctl+371} <ffffffff802a6dd6>{schedule_timeout+158}
       <ffffffff801bf181>{blkdev_ioctl+1365} <ffffffff80243cd2>{sys_sendto+251}
       <ffffffff801751e5>{__pollwait+0} <ffffffff8016b16a>{block_ioctl+25}
       <ffffffff801749f4>{do_ioctl+24} <ffffffff80174c46>{vfs_ioctl+541}
       <ffffffff80174cb4>{sys_ioctl+89} <ffffffff8010e4ba>{system_call+126}
       
Mem-info:
DMA per-cpu:
cpu 0 hot: low 0, high 0, batch 1 used:0
cpu 0 cold: low 0, high 0, batch 1 used:0
DMA32 per-cpu:
cpu 0 hot: low 0, high 186, batch 31 used:177
cpu 0 cold: low 0, high 62, batch 15 used:53
Normal per-cpu:
cpu 0 hot: low 0, high 186, batch 31 used:178
cpu 0 cold: low 0, high 62, batch 15 used:0
HighMem per-cpu: empty
Free pages:     3169120kB (0kB HighMem)
Active:169664 inactive:27316 dirty:25704 writeback:3254 unstable:0 free:792280 slab:11571 mapped:39271 pagetables:1562
DMA free:20kB min:24kB low:28kB high:36kB active:0kB inactive:0kB present:12740kB pages_scanned:2 all_unreclaimable? yes
lowmem_reserve[]: 0 2999 4009 4009
DMA32 free:2816320kB min:6052kB low:7564kB high:9076kB active:139680kB inactive:11112kB present:3071904kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 1010 1010
Normal free:352780kB min:2036kB low:2544kB high:3052kB active:538976kB inactive:98152kB present:1034240kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
DMA: 1*4kB 0*8kB 1*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 20kB
DMA32: 11666*4kB 10505*8kB 10673*16kB 9309*32kB 7290*64kB 5039*128kB 2420*256kB 711*512kB 107*1024kB 2*2048kB 2*4096kB = 2816320kB
Normal: 10207*4kB 6816*8kB 5839*16kB 939*32kB 867*64kB 513*128kB 44*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 352780kB
HighMem: empty
Swap cache: add 38, delete 38, find 0/0, race 0+0
Free swap  = 9686816kB
Total swap = 9686968kB
Free swap:       9686816kB
1310720 pages of RAM
299467 reserved pages
106408 pages shared
0 pages swap cached
Out of Memory: Killed process 24591 (nautilus).




-----Original Message-----
From: Jens Axboe [mailto:axboe@suse.de]
Sent: Fri 2006-01-20 17:39
To: Andy Chittenden
Cc: Andrew Morton; davej@redhat.com; linux-kernel@vger.kernel.org; lwoodman@redhat.com
Subject: Re: Out of Memory: Killed process 16498 (java).
 
On Fri, Jan 20 2006, Andy Chittenden wrote:
> > Andy, can you try and boot with this applied?
> 
> At boot,
> 
> # dmesg | grep bounce:
> bounce: queue ffff81013fa5cd18, setting pfn 1310720, max_low 1310720
> bounce: queue ffff81013fa5ca88, setting pfn 1310720, max_low 1310720
> bounce: queue ffff81013fa5c7f8, setting pfn 1310720, max_low 1310720
> bounce: queue ffff81013fa5c568, setting pfn 1310720, max_low 1310720
> bounce: queue ffff81013fa5c2d8, setting pfn 1310720, max_low 1310720
> bounce: queue ffff81013fa5c048, setting pfn 1310720, max_low 1310720
> bounce: queue ffff81013f93bd18, setting pfn 1310720, max_low 1310720
> bounce: queue ffff81013f93ba88, setting pfn 1310720, max_low 1310720
> bounce: queue ffff81013f93b7f8, setting pfn 1310720, max_low 1310720
> bounce: queue ffff81013f93b568, setting pfn 1310720, max_low 1310720
> bounce: queue ffff81013f93b2d8, setting pfn 1310720, max_low 1310720
> bounce: queue ffff81013f93b048, setting pfn 1310720, max_low 1310720
> bounce: queue ffff81013fa31d18, setting pfn 1310720, max_low 1310720
> bounce: queue ffff81013fa31a88, setting pfn 1310720, max_low 1310720
> bounce: queue ffff81013fa317f8, setting pfn 1310720, max_low 1310720
> bounce: queue ffff81013fa31568, setting pfn 1310720, max_low 1310720
> bounce: queue ffff81013fa312d8, setting pfn 1310720, max_low 1310720
> bounce: queue ffff81013fa312d8, setting pfn 1048575, max_low 1310720
> bounce: queue ffff81013fa31048, setting pfn 1310720, max_low 1310720
> bounce: queue ffff81013fa31048, setting pfn 1048575, max_low 1310720
> bounce: queue ffff81013f51fd18, setting pfn 1310720, max_low 1310720
> bounce: queue ffff81013f51fd18, setting pfn 1310720, max_low 1310720
> bounce: queue ffff81013f51fa88, setting pfn 1310720, max_low 1310720
> bounce: queue ffff81013f51fa88, setting pfn 1048575, max_low 1310720
> bounce: queue ffff81013f51f7f8, setting pfn 1310720, max_low 1310720
> 
> No change when I ran the test.

Please include the full dmesg, so I can see which queue is what. Most of
them look ok (1310720 is BLK_BOUNCE_HIGH), 1048575 is a little weird
though. So full dmesg please!

Perhaps this is a wrapping problem of some sort.

-- 
Jens Axboe



^ permalink raw reply	[flat|nested] 29+ messages in thread

* Re: Out of Memory: Killed process 16498 (java).
  2006-01-20  1:49       ` Andrew Morton
@ 2006-01-21 20:00         ` Dan Aloni
  0 siblings, 0 replies; 29+ messages in thread
From: Dan Aloni @ 2006-01-21 20:00 UTC (permalink / raw)
  To: Andrew Morton; +Cc: Dave Jones, AChittenden, linux-kernel, lwoodman, Jens Axboe

On Thu, Jan 19, 2006 at 05:49:20PM -0800, Andrew Morton wrote:
> Dave Jones <davej@redhat.com> wrote:
> >
> > The Fedora user in the bug report
> > https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=175173
> > who on x86-64 with 5GB saw zone_dma exhausted saw a similar result,
> > delays the kill, but it does still happen.
> 
> Again, that person's ZONE_DMA has *zero* pages on the LRU.  Something has
> consumed all of the piddling little zone for kernel data structures.  It is
> a true oom.
> 
> We need to work out who is using all this ZONE_DMA memory and make them
> stop it.

It reminds me, we had a similar problem in the 2.4.x kernel with 
many SCSI hosts loaded - that caused scsi_malloc (which uses DMA) 
to consume a lot of DMA memory and trigger the OOM. I know that code 
was much rewritten for 2.6.x, through DMA allocations in the SCSI 
subsystem (regardless of drivers) still exist here and there.

-- 
Dan Aloni
da-x@monatomic.org, da-x@colinux.org, da-x@gmx.net, dan@xiv.co.il

^ permalink raw reply	[flat|nested] 29+ messages in thread

* Re: Out of Memory: Killed process 16498 (java).
       [not found] <89E85E0168AD994693B574C80EDB9C27035561DE@uk-email.terastack.bluearc.com>
@ 2006-01-20 17:39 ` Jens Axboe
  0 siblings, 0 replies; 29+ messages in thread
From: Jens Axboe @ 2006-01-20 17:39 UTC (permalink / raw)
  To: Andy Chittenden; +Cc: Andrew Morton, davej, linux-kernel, lwoodman

On Fri, Jan 20 2006, Andy Chittenden wrote:
> > Andy, can you try and boot with this applied?
> 
> At boot,
> 
> # dmesg | grep bounce:
> bounce: queue ffff81013fa5cd18, setting pfn 1310720, max_low 1310720
> bounce: queue ffff81013fa5ca88, setting pfn 1310720, max_low 1310720
> bounce: queue ffff81013fa5c7f8, setting pfn 1310720, max_low 1310720
> bounce: queue ffff81013fa5c568, setting pfn 1310720, max_low 1310720
> bounce: queue ffff81013fa5c2d8, setting pfn 1310720, max_low 1310720
> bounce: queue ffff81013fa5c048, setting pfn 1310720, max_low 1310720
> bounce: queue ffff81013f93bd18, setting pfn 1310720, max_low 1310720
> bounce: queue ffff81013f93ba88, setting pfn 1310720, max_low 1310720
> bounce: queue ffff81013f93b7f8, setting pfn 1310720, max_low 1310720
> bounce: queue ffff81013f93b568, setting pfn 1310720, max_low 1310720
> bounce: queue ffff81013f93b2d8, setting pfn 1310720, max_low 1310720
> bounce: queue ffff81013f93b048, setting pfn 1310720, max_low 1310720
> bounce: queue ffff81013fa31d18, setting pfn 1310720, max_low 1310720
> bounce: queue ffff81013fa31a88, setting pfn 1310720, max_low 1310720
> bounce: queue ffff81013fa317f8, setting pfn 1310720, max_low 1310720
> bounce: queue ffff81013fa31568, setting pfn 1310720, max_low 1310720
> bounce: queue ffff81013fa312d8, setting pfn 1310720, max_low 1310720
> bounce: queue ffff81013fa312d8, setting pfn 1048575, max_low 1310720
> bounce: queue ffff81013fa31048, setting pfn 1310720, max_low 1310720
> bounce: queue ffff81013fa31048, setting pfn 1048575, max_low 1310720
> bounce: queue ffff81013f51fd18, setting pfn 1310720, max_low 1310720
> bounce: queue ffff81013f51fd18, setting pfn 1310720, max_low 1310720
> bounce: queue ffff81013f51fa88, setting pfn 1310720, max_low 1310720
> bounce: queue ffff81013f51fa88, setting pfn 1048575, max_low 1310720
> bounce: queue ffff81013f51f7f8, setting pfn 1310720, max_low 1310720
> 
> No change when I ran the test.

Please include the full dmesg, so I can see which queue is what. Most of
them look ok (1310720 is BLK_BOUNCE_HIGH), 1048575 is a little weird
though. So full dmesg please!

Perhaps this is a wrapping problem of some sort.

-- 
Jens Axboe


^ permalink raw reply	[flat|nested] 29+ messages in thread

* Re: Out of Memory: Killed process 16498 (java).
  2006-01-20 12:28             ` Jens Axboe
@ 2006-01-20 12:51               ` Andrew Morton
  0 siblings, 0 replies; 29+ messages in thread
From: Andrew Morton @ 2006-01-20 12:51 UTC (permalink / raw)
  To: Jens Axboe; +Cc: davej, AChittenden, linux-kernel, lwoodman

Jens Axboe <axboe@suse.de> wrote:
>
> > > better candidate to fix that :-)
> > 
> > It's not ZONE_DMA32.  It's the 12MB ZONE_DMA which is being exhausted on
> > this 4GB 64-bit machine.
> > 
> > Andy put a dump_stack() into the oom code and it pointed at 
> > 
> > 
> >  Call Trace:<ffffffff8014d7bc>{out_of_memory+48}
> >  <ffffffff8014f4b0>{__alloc_pages+536}
> >         <ffffffff80169788>{bio_alloc_bioset+232}
> >  <ffffffff80169d03>{bio_copy_user+218}
> >         <ffffffff801bd657>{blk_rq_map_user+136}
> >  <ffffffff801c0008>{sg_io+328}
> >         <ffffffff801c047c>{scsi_cmd_ioctl+491}
> >  <ffffffff88005e22>{:ide_core:generic_ide_ioctl+631}
> >         <ffffffff88202d0c>{:sd_mod:sd_ioctl+371}
> >  <ffffffff802a6db6>{schedule_timeout+158}
> >         <ffffffff801bf165>{blkdev_ioctl+1365}
> >  <ffffffff80243cb2>{sys_sendto+251}
> >         <ffffffff801751e5>{__pollwait+0}
> >  <ffffffff8016b16a>{block_ioctl+25}
> >         <ffffffff801749f4>{do_ioctl+24} <ffffffff80174c46>{vfs_ioctl+541}
> >         <ffffffff80174cb4>{sys_ioctl+89}
> 
> Hmm strange, what kind of device is this? I'm guessing it's not ISA.

 sata_via 0000:00:0f.0: routed to hard irq line 1
 ata1: SATA max UDMA/133 cmd 0xD000 ctl 0xC802 bmdma 0xB800 irq 17
 ata2: SATA max UDMA/133 cmd 0xC400 ctl 0xC002 bmdma 0xB808 irq 17
 sata_promise 0000:00:08.0: version 1.03
 ACPI: PCI Interrupt 0000:00:08.0[A] -> GSI 18 (level, low) -> IRQ 19
 ata3: SATA max UDMA/133 cmd 0xFFFFC20000006200 ctl 0xFFFFC20000006238
 bmdma 0x0 irq 19
 ata4: SATA max UDMA/133 cmd 0xFFFFC20000006280 ctl 0xFFFFC200000062B8
 bmdma 0x0 irq 19
 ...
 ata4: no device found (phy stat 00000000)
 scsi3 : sata_promise
 ata1: dev 0 cfg 49:2f00 82:3069 83:7c01 84:4003 85:3069 86:3c01 87:4003
 88:203f
 ata1: dev 0 ATA-6, max UDMA/100, 390721968 sectors: LBA48
 ata1: dev 0 configured for UDMA/100
 scsi0 : sata_via
 ata2: no device found (phy stat 00000000)
 scsi1 : sata_via
   Vendor: ATA       Model: ST3200822AS       Rev: 3.01
   Type:   Direct-Access                      ANSI SCSI revision: 05

> Andy, can you try and boot with this applied?

yes, that'll be interesting.

> Did the blk_max_low_pfn stuff get a different meaning with the addition
> of the DMA32 zone?

The Fedora reporter
(https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=175173) hit what
appears to be the same thing (gfp_mask=0xd1 - that's ZONE_DMA unless I've
lost it completely?) in 2.6.14, which doesn't have ZONE_DMA32.


^ permalink raw reply	[flat|nested] 29+ messages in thread

* Re: Out of Memory: Killed process 16498 (java).
       [not found] <89E85E0168AD994693B574C80EDB9C27035560E4@uk-email.terastack.bluearc.com>
@ 2006-01-20 12:37 ` Larry Woodman
  0 siblings, 0 replies; 29+ messages in thread
From: Larry Woodman @ 2006-01-20 12:37 UTC (permalink / raw)
  To: Andy Chittenden; +Cc: Jens Axboe, Andrew Morton, Dave Jones, linux-kernel

Andy Chittenden wrote:

>>Andy, what are
>>you running that uses SG_IO - cd ripping, burning, something else?
>>    
>>
>
>I'm sorry but I haven't a clue what uses SG_IO! All I did was boot up a
>debian unstable machine on my amd64 machine that uses 2.6.15. I log in
>via gdm and get a gnome session so I guess that's using nautilus (I've
>seen that killed in the past). I use the sawfish window manager and
>start up 7 rxvt windows and that java app I mentioned (terminator) (I
>suspect that's a red herring as other processes have been killed).
>
>  
>
for starters you should probably change build_zonelists so that the DMA 
zone is
not included in any of the zone lists except the DMA.  This will prevent 
__alloc_pages()
from exhausting the hignmem/normal zones then falling into the DMA zone 
and exhausting
that with non-reclamable memory like the slabcache.

--- linux-2.6.9/mm/page_alloc.c.orig
+++ linux-2.6.9/mm/page_alloc.c
@@ -1170,6 +1170,9 @@ static int __init build_zonelists_node(p
                zone = pgdat->node_zones + ZONE_NORMAL;
                if (zone->present_pages)
                        zonelist->zones[j++] = zone;
+#if defined(CONFIG_HIGHMEM64G) || defined(CONFIG_X86_64)
+               break;
+#endif
        case ZONE_DMA:
                zone = pgdat->node_zones + ZONE_DMA;
                if (zone->present_pages)
~




^ permalink raw reply	[flat|nested] 29+ messages in thread

* Re: Out of Memory: Killed process 16498 (java).
  2006-01-20 12:17           ` Andrew Morton
@ 2006-01-20 12:28             ` Jens Axboe
  2006-01-20 12:51               ` Andrew Morton
  0 siblings, 1 reply; 29+ messages in thread
From: Jens Axboe @ 2006-01-20 12:28 UTC (permalink / raw)
  To: Andrew Morton; +Cc: davej, AChittenden, linux-kernel, lwoodman

On Fri, Jan 20 2006, Andrew Morton wrote:
> Jens Axboe <axboe@suse.de> wrote:
> >
> > On Fri, Jan 20 2006, Andrew Morton wrote:
> > > Jens Axboe <axboe@suse.de> wrote:
> > > >
> > > > On Thu, Jan 19 2006, Andrew Morton wrote:
> > > > > Dave Jones <davej@redhat.com> wrote:
> > > > > >
> > > > > > On Thu, Jan 19, 2006 at 03:11:45PM -0000, Andy Chittenden wrote:
> > > > > >  > DMA free:20kB min:24kB low:28kB high:36kB active:0kB inactive:0kB
> > > > > >  > present:12740kB pages_scanned:4 all_unreclaimable? yes
> > > > > > 
> > > > > > Note we only scanned 4 pages before we gave up.
> > > > > > Larry Woodman came up with this patch below that clears all_unreclaimable
> > > > > > when in two places where we've made progress at freeing up some pages
> > > > > > which has helped oom situations for some of our users.
> > > > > 
> > > > > That won't help - there are exactly zero pages on ZONE_DMA's LRU.
> > > > > 
> > > > > The problem appears to be that all of the DMA zone has been gobbled up by
> > > > > the BIO layer.  It seems quite inappropriate that a modern 64-bit machine
> > > > > is allocating tons of disk I/O pages from the teeny ZONE_DMA.  I'm
> > > > > suspecting that someone has gone and set a queue's ->bounce_gfp to the wrong
> > > > > thing.
> > > > > 
> > > > > Jens, would you have time to investigate please?
> > > > 
> > > > Certainly, I'll get this tested and fixed this afternoon.
> > > 
> > > Wow ;)
> > >
> > > You may find it's an x86_64 glitch - setting max_[low_]pfn wrong down in
> > > the bowels of the arch mm init code, something like that.
> > > 
> > > I thought it might have been a regression which came in when we added
> > > ZONE_DMA32 but the RH reporter is based on 2.6.14-<redhat stuff>, and he
> > > didn't have ZONE_DMA32.
> > 
> > Sorry, spoke too soon, I thought this was the 'bio/scsi leaks' which
> > most likely is a scsi leak that also results in the bios not getting
> > freed.
> > 
> > This DMA32 zone shortage looks like a vm short coming, you're likely the
> > better candidate to fix that :-)
> 
> It's not ZONE_DMA32.  It's the 12MB ZONE_DMA which is being exhausted on
> this 4GB 64-bit machine.
> 
> Andy put a dump_stack() into the oom code and it pointed at 
> 
> 
>  Call Trace:<ffffffff8014d7bc>{out_of_memory+48}
>  <ffffffff8014f4b0>{__alloc_pages+536}
>         <ffffffff80169788>{bio_alloc_bioset+232}
>  <ffffffff80169d03>{bio_copy_user+218}
>         <ffffffff801bd657>{blk_rq_map_user+136}
>  <ffffffff801c0008>{sg_io+328}
>         <ffffffff801c047c>{scsi_cmd_ioctl+491}
>  <ffffffff88005e22>{:ide_core:generic_ide_ioctl+631}
>         <ffffffff88202d0c>{:sd_mod:sd_ioctl+371}
>  <ffffffff802a6db6>{schedule_timeout+158}
>         <ffffffff801bf165>{blkdev_ioctl+1365}
>  <ffffffff80243cb2>{sys_sendto+251}
>         <ffffffff801751e5>{__pollwait+0}
>  <ffffffff8016b16a>{block_ioctl+25}
>         <ffffffff801749f4>{do_ioctl+24} <ffffffff80174c46>{vfs_ioctl+541}
>         <ffffffff80174cb4>{sys_ioctl+89}

Hmm strange, what kind of device is this? I'm guessing it's not ISA.
Andy, can you try and boot with this applied?

Did the blk_max_low_pfn stuff get a different meaning with the addition
of the DMA32 zone?

diff --git a/block/ll_rw_blk.c b/block/ll_rw_blk.c
index 8e27d0a..ab897de 100644
--- a/block/ll_rw_blk.c
+++ b/block/ll_rw_blk.c
@@ -636,6 +636,8 @@ void blk_queue_bounce_limit(request_queu
 {
 	unsigned long bounce_pfn = dma_addr >> PAGE_SHIFT;
 
+	printk("bounce: queue %p, setting pfn %lu, max_low %lu\n", q, bounce_pfn, blk_max_low_pfn);
+
 	/*
 	 * set appropriate bounce gfp mask -- unfortunately we don't have a
 	 * full 4GB zone, so we have to resort to low memory for any bounces.

-- 
Jens Axboe


^ permalink raw reply related	[flat|nested] 29+ messages in thread

* Re: Out of Memory: Killed process 16498 (java).
  2006-01-20 12:08         ` Jens Axboe
@ 2006-01-20 12:17           ` Andrew Morton
  2006-01-20 12:28             ` Jens Axboe
  0 siblings, 1 reply; 29+ messages in thread
From: Andrew Morton @ 2006-01-20 12:17 UTC (permalink / raw)
  To: Jens Axboe; +Cc: davej, AChittenden, linux-kernel, lwoodman

Jens Axboe <axboe@suse.de> wrote:
>
> On Fri, Jan 20 2006, Andrew Morton wrote:
> > Jens Axboe <axboe@suse.de> wrote:
> > >
> > > On Thu, Jan 19 2006, Andrew Morton wrote:
> > > > Dave Jones <davej@redhat.com> wrote:
> > > > >
> > > > > On Thu, Jan 19, 2006 at 03:11:45PM -0000, Andy Chittenden wrote:
> > > > >  > DMA free:20kB min:24kB low:28kB high:36kB active:0kB inactive:0kB
> > > > >  > present:12740kB pages_scanned:4 all_unreclaimable? yes
> > > > > 
> > > > > Note we only scanned 4 pages before we gave up.
> > > > > Larry Woodman came up with this patch below that clears all_unreclaimable
> > > > > when in two places where we've made progress at freeing up some pages
> > > > > which has helped oom situations for some of our users.
> > > > 
> > > > That won't help - there are exactly zero pages on ZONE_DMA's LRU.
> > > > 
> > > > The problem appears to be that all of the DMA zone has been gobbled up by
> > > > the BIO layer.  It seems quite inappropriate that a modern 64-bit machine
> > > > is allocating tons of disk I/O pages from the teeny ZONE_DMA.  I'm
> > > > suspecting that someone has gone and set a queue's ->bounce_gfp to the wrong
> > > > thing.
> > > > 
> > > > Jens, would you have time to investigate please?
> > > 
> > > Certainly, I'll get this tested and fixed this afternoon.
> > 
> > Wow ;)
> >
> > You may find it's an x86_64 glitch - setting max_[low_]pfn wrong down in
> > the bowels of the arch mm init code, something like that.
> > 
> > I thought it might have been a regression which came in when we added
> > ZONE_DMA32 but the RH reporter is based on 2.6.14-<redhat stuff>, and he
> > didn't have ZONE_DMA32.
> 
> Sorry, spoke too soon, I thought this was the 'bio/scsi leaks' which
> most likely is a scsi leak that also results in the bios not getting
> freed.
> 
> This DMA32 zone shortage looks like a vm short coming, you're likely the
> better candidate to fix that :-)

It's not ZONE_DMA32.  It's the 12MB ZONE_DMA which is being exhausted on
this 4GB 64-bit machine.

Andy put a dump_stack() into the oom code and it pointed at 


 Call Trace:<ffffffff8014d7bc>{out_of_memory+48}
 <ffffffff8014f4b0>{__alloc_pages+536}
        <ffffffff80169788>{bio_alloc_bioset+232}
 <ffffffff80169d03>{bio_copy_user+218}
        <ffffffff801bd657>{blk_rq_map_user+136}
 <ffffffff801c0008>{sg_io+328}
        <ffffffff801c047c>{scsi_cmd_ioctl+491}
 <ffffffff88005e22>{:ide_core:generic_ide_ioctl+631}
        <ffffffff88202d0c>{:sd_mod:sd_ioctl+371}
 <ffffffff802a6db6>{schedule_timeout+158}
        <ffffffff801bf165>{blkdev_ioctl+1365}
 <ffffffff80243cb2>{sys_sendto+251}
        <ffffffff801751e5>{__pollwait+0}
 <ffffffff8016b16a>{block_ioctl+25}
        <ffffffff801749f4>{do_ioctl+24} <ffffffff80174c46>{vfs_ioctl+541}
        <ffffffff80174cb4>{sys_ioctl+89}

^ permalink raw reply	[flat|nested] 29+ messages in thread

* Re: Out of Memory: Killed process 16498 (java).
  2006-01-20  8:23       ` Andrew Morton
@ 2006-01-20 12:08         ` Jens Axboe
  2006-01-20 12:17           ` Andrew Morton
  0 siblings, 1 reply; 29+ messages in thread
From: Jens Axboe @ 2006-01-20 12:08 UTC (permalink / raw)
  To: Andrew Morton; +Cc: davej, AChittenden, linux-kernel, lwoodman

On Fri, Jan 20 2006, Andrew Morton wrote:
> Jens Axboe <axboe@suse.de> wrote:
> >
> > On Thu, Jan 19 2006, Andrew Morton wrote:
> > > Dave Jones <davej@redhat.com> wrote:
> > > >
> > > > On Thu, Jan 19, 2006 at 03:11:45PM -0000, Andy Chittenden wrote:
> > > >  > DMA free:20kB min:24kB low:28kB high:36kB active:0kB inactive:0kB
> > > >  > present:12740kB pages_scanned:4 all_unreclaimable? yes
> > > > 
> > > > Note we only scanned 4 pages before we gave up.
> > > > Larry Woodman came up with this patch below that clears all_unreclaimable
> > > > when in two places where we've made progress at freeing up some pages
> > > > which has helped oom situations for some of our users.
> > > 
> > > That won't help - there are exactly zero pages on ZONE_DMA's LRU.
> > > 
> > > The problem appears to be that all of the DMA zone has been gobbled up by
> > > the BIO layer.  It seems quite inappropriate that a modern 64-bit machine
> > > is allocating tons of disk I/O pages from the teeny ZONE_DMA.  I'm
> > > suspecting that someone has gone and set a queue's ->bounce_gfp to the wrong
> > > thing.
> > > 
> > > Jens, would you have time to investigate please?
> > 
> > Certainly, I'll get this tested and fixed this afternoon.
> 
> Wow ;)
>
> You may find it's an x86_64 glitch - setting max_[low_]pfn wrong down in
> the bowels of the arch mm init code, something like that.
> 
> I thought it might have been a regression which came in when we added
> ZONE_DMA32 but the RH reporter is based on 2.6.14-<redhat stuff>, and he
> didn't have ZONE_DMA32.

Sorry, spoke too soon, I thought this was the 'bio/scsi leaks' which
most likely is a scsi leak that also results in the bios not getting
freed.

This DMA32 zone shortage looks like a vm short coming, you're likely the
better candidate to fix that :-)

-- 
Jens Axboe


^ permalink raw reply	[flat|nested] 29+ messages in thread

* Re: Out of Memory: Killed process 16498 (java).
  2006-01-20  8:12     ` Jens Axboe
@ 2006-01-20  8:23       ` Andrew Morton
  2006-01-20 12:08         ` Jens Axboe
  0 siblings, 1 reply; 29+ messages in thread
From: Andrew Morton @ 2006-01-20  8:23 UTC (permalink / raw)
  To: Jens Axboe; +Cc: davej, AChittenden, linux-kernel, lwoodman

Jens Axboe <axboe@suse.de> wrote:
>
> On Thu, Jan 19 2006, Andrew Morton wrote:
> > Dave Jones <davej@redhat.com> wrote:
> > >
> > > On Thu, Jan 19, 2006 at 03:11:45PM -0000, Andy Chittenden wrote:
> > >  > DMA free:20kB min:24kB low:28kB high:36kB active:0kB inactive:0kB
> > >  > present:12740kB pages_scanned:4 all_unreclaimable? yes
> > > 
> > > Note we only scanned 4 pages before we gave up.
> > > Larry Woodman came up with this patch below that clears all_unreclaimable
> > > when in two places where we've made progress at freeing up some pages
> > > which has helped oom situations for some of our users.
> > 
> > That won't help - there are exactly zero pages on ZONE_DMA's LRU.
> > 
> > The problem appears to be that all of the DMA zone has been gobbled up by
> > the BIO layer.  It seems quite inappropriate that a modern 64-bit machine
> > is allocating tons of disk I/O pages from the teeny ZONE_DMA.  I'm
> > suspecting that someone has gone and set a queue's ->bounce_gfp to the wrong
> > thing.
> > 
> > Jens, would you have time to investigate please?
> 
> Certainly, I'll get this tested and fixed this afternoon.

Wow ;)

You may find it's an x86_64 glitch - setting max_[low_]pfn wrong down in
the bowels of the arch mm init code, something like that.

I thought it might have been a regression which came in when we added
ZONE_DMA32 but the RH reporter is based on 2.6.14-<redhat stuff>, and he
didn't have ZONE_DMA32.


^ permalink raw reply	[flat|nested] 29+ messages in thread

* Re: Out of Memory: Killed process 16498 (java).
  2006-01-19 22:15   ` Andrew Morton
@ 2006-01-20  8:12     ` Jens Axboe
  2006-01-20  8:23       ` Andrew Morton
  0 siblings, 1 reply; 29+ messages in thread
From: Jens Axboe @ 2006-01-20  8:12 UTC (permalink / raw)
  To: Andrew Morton; +Cc: Dave Jones, AChittenden, linux-kernel, lwoodman

On Thu, Jan 19 2006, Andrew Morton wrote:
> Dave Jones <davej@redhat.com> wrote:
> >
> > On Thu, Jan 19, 2006 at 03:11:45PM -0000, Andy Chittenden wrote:
> >  > DMA free:20kB min:24kB low:28kB high:36kB active:0kB inactive:0kB
> >  > present:12740kB pages_scanned:4 all_unreclaimable? yes
> > 
> > Note we only scanned 4 pages before we gave up.
> > Larry Woodman came up with this patch below that clears all_unreclaimable
> > when in two places where we've made progress at freeing up some pages
> > which has helped oom situations for some of our users.
> 
> That won't help - there are exactly zero pages on ZONE_DMA's LRU.
> 
> The problem appears to be that all of the DMA zone has been gobbled up by
> the BIO layer.  It seems quite inappropriate that a modern 64-bit machine
> is allocating tons of disk I/O pages from the teeny ZONE_DMA.  I'm
> suspecting that someone has gone and set a queue's ->bounce_gfp to the wrong
> thing.
> 
> Jens, would you have time to investigate please?

Certainly, I'll get this tested and fixed this afternoon. Andy, what are
you running that uses SG_IO - cd ripping, burning, something else?

-- 
Jens Axboe


^ permalink raw reply	[flat|nested] 29+ messages in thread

* Re: Out of Memory: Killed process 16498 (java).
  2006-01-20  1:28     ` Dave Jones
@ 2006-01-20  1:49       ` Andrew Morton
  2006-01-21 20:00         ` Dan Aloni
  0 siblings, 1 reply; 29+ messages in thread
From: Andrew Morton @ 2006-01-20  1:49 UTC (permalink / raw)
  To: Dave Jones; +Cc: AChittenden, linux-kernel, lwoodman, Jens Axboe

Dave Jones <davej@redhat.com> wrote:
>
> The Fedora user in the bug report
> https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=175173
> who on x86-64 with 5GB saw zone_dma exhausted saw a similar result,
> delays the kill, but it does still happen.

Again, that person's ZONE_DMA has *zero* pages on the LRU.  Something has
consumed all of the piddling little zone for kernel data structures.  It is
a true oom.

We need to work out who is using all this ZONE_DMA memory and make them
stop it.


^ permalink raw reply	[flat|nested] 29+ messages in thread

* Re: Out of Memory: Killed process 16498 (java).
  2006-01-20  1:03   ` Andrew Morton
@ 2006-01-20  1:28     ` Dave Jones
  2006-01-20  1:49       ` Andrew Morton
  0 siblings, 1 reply; 29+ messages in thread
From: Dave Jones @ 2006-01-20  1:28 UTC (permalink / raw)
  To: Andrew Morton; +Cc: AChittenden, linux-kernel, lwoodman

On Thu, Jan 19, 2006 at 05:03:05PM -0800, Andrew Morton wrote:
 > This is the bypass-the-batching patch.  It's a reasonable thing to do, but I'd
 > just do it unconditionally and remove the code which clears
 > ->all_unreclaimable from free_pages_bulk(), if possible.
 > 
 > Has this patch been shown to have any effect?  If so, what was it, and
 > under what conditions?

Larry originally came up with the patch for our RHEL4 2.6.9 kernel,
after customers were hitting OOM under some heavy workload he can
probably recall better than I can.

iirc it didn't solve the users OOM entirely, but it does make their workload
run longer before the kill happens.

A little while later, after Fedora users started reporting oom kills,
I forward-ported it to 2.6.14, and threw it out in an update.
The Fedora user in the bug report
https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=175173
who on x86-64 with 5GB saw zone_dma exhausted saw a similar result,
delays the kill, but it does still happen.

Hmm.

		Dave

^ permalink raw reply	[flat|nested] 29+ messages in thread

* Re: Out of Memory: Killed process 16498 (java).
  2006-01-19 19:48 ` Dave Jones
  2006-01-19 22:15   ` Andrew Morton
@ 2006-01-20  1:03   ` Andrew Morton
  2006-01-20  1:28     ` Dave Jones
  1 sibling, 1 reply; 29+ messages in thread
From: Andrew Morton @ 2006-01-20  1:03 UTC (permalink / raw)
  To: Dave Jones; +Cc: AChittenden, linux-kernel, lwoodman

Dave Jones <davej@redhat.com> wrote:
>
> On Thu, Jan 19, 2006 at 03:11:45PM -0000, Andy Chittenden wrote:
>  > DMA free:20kB min:24kB low:28kB high:36kB active:0kB inactive:0kB
>  > present:12740kB pages_scanned:4 all_unreclaimable? yes
> 
> Note we only scanned 4 pages before we gave up.
> Larry Woodman came up with this patch below that clears all_unreclaimable
> when in two places where we've made progress at freeing up some pages
> which has helped oom situations for some of our users.
> 

We already clear ->all_unreclaimable in free_pages_bulk, so I guess the
changes here are a) bypass the per-cpu-pages magazining (fair enough I
suppose) and b) clear all_unreclaimable earlier: as a page becomes
reclaimable, not as we reclaim it.

I wonder if it really makes a difference.  Given that various processes are
currently scanning their little hearts out, if a reclaimable page pops up
at the tail of the LRU, we'll reclaim it pretty much immediately and go off
and, after the per-cpu batching, will clear ->all_unreclaimable.


> --- linux-2.6/mm/filemap.c~	2005-12-10 01:47:15.000000000 -0500
> +++ linux-2.6/mm/filemap.c	2005-12-10 01:47:46.000000000 -0500
> @@ -471,11 +471,18 @@ EXPORT_SYMBOL(unlock_page);
>   */
>  void end_page_writeback(struct page *page)
>  {
> +	struct zone *zone = page_zone(page);
>  	if (!TestClearPageReclaim(page) || rotate_reclaimable_page(page)) {
>  		if (!test_clear_page_writeback(page))
>  			BUG();
>  	}
>  	smp_mb__after_clear_bit();
> +	if (zone->all_unreclaimable) {
> +		spin_lock(&zone->lock);
> +		zone->all_unreclaimable = 0;
> +		zone->pages_scanned = 0;
> +		spin_unlock(&zone->lock);
> +	}
>  	wake_up_page(page, PG_writeback);
>  }

Wouldn't it be better to only clear ->all_unreclaimable if the page was
actually reclaimable?  ie: inside rotate_reclaimable_page()?

Doing that would also fix the deadlock in the above code: zone.lock is
supposed to be irq-safe.

>  EXPORT_SYMBOL(end_page_writeback);
> --- linux-2.6.15/mm/page_alloc.c~	2006-01-09 13:40:03.000000000 -0500
> +++ linux-2.6.15/mm/page_alloc.c	2006-01-09 13:40:50.000000000 -0500
> @@ -722,6 +722,11 @@ static void fastcall free_hot_cold_page(
>  	if (pcp->count >= pcp->high) {
>  		free_pages_bulk(zone, pcp->batch, &pcp->list, 0);
>  		pcp->count -= pcp->batch;
> +	} else if (zone->all_unreclaimable) {
> +		spin_lock(&zone->lock);
> +		zone->all_unreclaimable = 0;
> +		zone->pages_scanned = 0;
> +		spin_unlock(&zone->lock);
>  	}

This is the bypass-the-batching patch.  It's a reasonable thing to do, but I'd
just do it unconditionally and remove the code which clears
->all_unreclaimable from free_pages_bulk(), if possible.

Has this patch been shown to have any effect?  If so, what was it, and
under what conditions?


^ permalink raw reply	[flat|nested] 29+ messages in thread

* Re: Out of Memory: Killed process 16498 (java).
  2006-01-19 19:48 ` Dave Jones
@ 2006-01-19 22:15   ` Andrew Morton
  2006-01-20  8:12     ` Jens Axboe
  2006-01-20  1:03   ` Andrew Morton
  1 sibling, 1 reply; 29+ messages in thread
From: Andrew Morton @ 2006-01-19 22:15 UTC (permalink / raw)
  To: Dave Jones; +Cc: AChittenden, linux-kernel, lwoodman, Jens Axboe

Dave Jones <davej@redhat.com> wrote:
>
> On Thu, Jan 19, 2006 at 03:11:45PM -0000, Andy Chittenden wrote:
>  > DMA free:20kB min:24kB low:28kB high:36kB active:0kB inactive:0kB
>  > present:12740kB pages_scanned:4 all_unreclaimable? yes
> 
> Note we only scanned 4 pages before we gave up.
> Larry Woodman came up with this patch below that clears all_unreclaimable
> when in two places where we've made progress at freeing up some pages
> which has helped oom situations for some of our users.

That won't help - there are exactly zero pages on ZONE_DMA's LRU.

The problem appears to be that all of the DMA zone has been gobbled up by
the BIO layer.  It seems quite inappropriate that a modern 64-bit machine
is allocating tons of disk I/O pages from the teeny ZONE_DMA.  I'm
suspecting that someone has gone and set a queue's ->bounce_gfp to the wrong
thing.

Jens, would you have time to investigate please?

^ permalink raw reply	[flat|nested] 29+ messages in thread

* Re: Out of Memory: Killed process 16498 (java).
  2006-01-19  8:43 Andy Chittenden
  2006-01-19  8:56 ` Andrew Morton
@ 2006-01-19 20:54 ` Martin Bligh
  1 sibling, 0 replies; 29+ messages in thread
From: Martin Bligh @ 2006-01-19 20:54 UTC (permalink / raw)
  To: Andy Chittenden; +Cc: linux-kernel

Andy Chittenden wrote:
> Why does running the following command cause processes to be killed:
> 
> 	dd if=/dev/zero of=/u/u1/andyc/tmpfile bs=1M count=8k
> 
> And I noticed one of my windows disappeared. Further investigation
> showed that was my terminator window (java based app: see
> http://software.jessies.org/terminator/). I found this in my syslog:
> 
> Jan 17 11:12:58 boco kernel: Out of Memory: Killed process 16498 (java).

Ah well. You can't fault us for having good taste ;-)

M.

^ permalink raw reply	[flat|nested] 29+ messages in thread

* Re: Out of Memory: Killed process 16498 (java).
  2006-01-19 15:11 Andy Chittenden
@ 2006-01-19 19:48 ` Dave Jones
  2006-01-19 22:15   ` Andrew Morton
  2006-01-20  1:03   ` Andrew Morton
  0 siblings, 2 replies; 29+ messages in thread
From: Dave Jones @ 2006-01-19 19:48 UTC (permalink / raw)
  To: Andy Chittenden; +Cc: Andrew Morton, linux-kernel, lwoodman

On Thu, Jan 19, 2006 at 03:11:45PM -0000, Andy Chittenden wrote:
 > DMA free:20kB min:24kB low:28kB high:36kB active:0kB inactive:0kB
 > present:12740kB pages_scanned:4 all_unreclaimable? yes

Note we only scanned 4 pages before we gave up.
Larry Woodman came up with this patch below that clears all_unreclaimable
when in two places where we've made progress at freeing up some pages
which has helped oom situations for some of our users.

From: Larry Woodman <lwoodman@redhat.com>
Signed-off-by: Dave Jones <davej@redhat.com>

--- linux-2.6/mm/filemap.c~	2005-12-10 01:47:15.000000000 -0500
+++ linux-2.6/mm/filemap.c	2005-12-10 01:47:46.000000000 -0500
@@ -471,11 +471,18 @@ EXPORT_SYMBOL(unlock_page);
  */
 void end_page_writeback(struct page *page)
 {
+	struct zone *zone = page_zone(page);
 	if (!TestClearPageReclaim(page) || rotate_reclaimable_page(page)) {
 		if (!test_clear_page_writeback(page))
 			BUG();
 	}
 	smp_mb__after_clear_bit();
+	if (zone->all_unreclaimable) {
+		spin_lock(&zone->lock);
+		zone->all_unreclaimable = 0;
+		zone->pages_scanned = 0;
+		spin_unlock(&zone->lock);
+	}
 	wake_up_page(page, PG_writeback);
 }
 EXPORT_SYMBOL(end_page_writeback);
--- linux-2.6.15/mm/page_alloc.c~	2006-01-09 13:40:03.000000000 -0500
+++ linux-2.6.15/mm/page_alloc.c	2006-01-09 13:40:50.000000000 -0500
@@ -722,6 +722,11 @@ static void fastcall free_hot_cold_page(
 	if (pcp->count >= pcp->high) {
 		free_pages_bulk(zone, pcp->batch, &pcp->list, 0);
 		pcp->count -= pcp->batch;
+	} else if (zone->all_unreclaimable) {
+		spin_lock(&zone->lock);
+		zone->all_unreclaimable = 0;
+		zone->pages_scanned = 0;
+		spin_unlock(&zone->lock);
 	}
 	local_irq_restore(flags);
 	put_cpu();

^ permalink raw reply	[flat|nested] 29+ messages in thread

* RE: Out of Memory: Killed process 16498 (java).
@ 2006-01-19 15:11 Andy Chittenden
  2006-01-19 19:48 ` Dave Jones
  0 siblings, 1 reply; 29+ messages in thread
From: Andy Chittenden @ 2006-01-19 15:11 UTC (permalink / raw)
  To: Andrew Morton; +Cc: linux-kernel

> Can you please add this patch, retest?

Dmesg now reports:

# dmesg
Bootdata ok (command line is root=/dev/hda1 ro )
Linux version 2.6.15-1-amd64-k8 (Debian 2.6.15-1) (luther@debian.org)
(gcc version 4.0.3 20060115 (prerelease) (Debian 4.0.2-7)) #2 Thu Jan 19
13:00:18 GMT 2006
BIOS-provided physical RAM map:
 BIOS-e820: 0000000000000000 - 000000000009fc00 (usable)
 BIOS-e820: 000000000009fc00 - 00000000000a0000 (reserved)
 BIOS-e820: 00000000000e4000 - 0000000000100000 (reserved)
 BIOS-e820: 0000000000100000 - 00000000bffb0000 (usable)
 BIOS-e820: 00000000bffb0000 - 00000000bffc0000 (ACPI data)
 BIOS-e820: 00000000bffc0000 - 00000000bfff0000 (ACPI NVS)
 BIOS-e820: 00000000bfff0000 - 00000000c0000000 (reserved)
 BIOS-e820: 00000000ff780000 - 0000000100000000 (reserved)
 BIOS-e820: 0000000100000000 - 0000000140000000 (usable)
ACPI: RSDP (v000 ACPIAM                                ) @
0x00000000000fa7c0
ACPI: RSDT (v001 A M I  OEMRSDT  0x10000506 MSFT 0x00000097) @
0x00000000bffb0000
ACPI: FADT (v001 A M I  OEMFACP  0x10000506 MSFT 0x00000097) @
0x00000000bffb0200
ACPI: MADT (v001 A M I  OEMAPIC  0x10000506 MSFT 0x00000097) @
0x00000000bffb0390
ACPI: OEMB (v001 A M I  OEMBIOS  0x10000506 MSFT 0x00000097) @
0x00000000bffc0040
ACPI: DSDT (v001  A0036 A0036001 0x00000001 MSFT 0x0100000d) @
0x0000000000000000
On node 0 totalpages: 1029721
  DMA zone: 3185 pages, LIFO batch:0
  DMA32 zone: 767976 pages, LIFO batch:31
  Normal zone: 258560 pages, LIFO batch:31
  HighMem zone: 0 pages, LIFO batch:0
Looks like a VIA chipset. Disabling IOMMU. Overwrite with
"iommu=allowed"
ACPI: PM-Timer IO Port: 0x808
ACPI: Local APIC address 0xfee00000
ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled)
Processor #0 15:15 APIC version 16
ACPI: LAPIC (acpi_id[0x02] lapic_id[0x81] disabled)
ACPI: IOAPIC (id[0x01] address[0xfec00000] gsi_base[0])
IOAPIC[0]: apic_id 1, version 3, address 0xfec00000, GSI 0-23
ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
ACPI: IRQ0 used by override.
ACPI: IRQ2 used by override.
ACPI: IRQ9 used by override.
Setting APIC routing to flat
Using ACPI (MADT) for SMP configuration information
Allocating PCI resources starting at c4000000 (gap: c0000000:3f780000)
Built 1 zonelists
Kernel command line: root=/dev/hda1 ro 
Initializing CPU#0
PID hash table entries: 4096 (order: 12, 131072 bytes)
time.c: Using 3.579545 MHz PM timer.
time.c: Detected 2202.924 MHz processor.
time.c: Using PIT/TSC based timekeeping.
Console: colour VGA+ 80x25
Dentry cache hash table entries: 524288 (order: 10, 4194304 bytes)
Inode-cache hash table entries: 262144 (order: 9, 2097152 bytes)
Placing software IO TLB between 0x5c39000 - 0x9c39000
Memory: 4045192k/5242880k available (1703k kernel code, 148144k
reserved, 738k data, 148k init)
Calibrating delay using timer specific routine.. 4412.87 BogoMIPS
(lpj=2206436)
Security Framework v1.0.0 initialized
SELinux:  Disabled at boot.
Capability LSM initialized
Mount-cache hash table entries: 256
CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
CPU: L2 Cache: 512K (64 bytes/line)
mtrr: v2.0 (20020519)
CPU: AMD Athlon(tm) 64 Processor 3500+ stepping 00
Using local APIC timer interrupts.
Detected 12.516 MHz APIC timer.
testing NMI watchdog ... OK.
checking if image is initramfs... it is
NET: Registered protocol family 16
ACPI: bus type pci registered
PCI: Using configuration type 1
ACPI: Subsystem revision 20050902
ACPI: Interpreter enabled
ACPI: Using IOAPIC for interrupt routing
ACPI: PCI Root Bridge [PCI0] (0000:00)
PCI: Probing PCI hardware (bus 00)
Boot video device is 0000:01:00.0
ACPI: PCI Interrupt Routing Table [\_SB_.PCI0._PRT]
ACPI: PCI Interrupt Link [LNKA] (IRQs 3 4 5 7 10 *11 14 15)
ACPI: PCI Interrupt Link [LNKB] (IRQs 3 4 5 7 *10 11 14 15)
ACPI: PCI Interrupt Link [LNKC] (IRQs 3 4 *5 7 10 11 14 15)
ACPI: PCI Interrupt Link [LNKD] (IRQs 3 4 5 7 10 11 14 15) *0, disabled.
ACPI: PCI Interrupt Link [LNKE] (IRQs 3 4 5 7 10 11 14 15) *0, disabled.
ACPI: PCI Interrupt Link [LNKF] (IRQs 3 4 5 7 10 11 14 15) *0, disabled.
ACPI: PCI Interrupt Link [LNKG] (IRQs 3 4 5 7 10 11 14 15) *0, disabled.
ACPI: PCI Interrupt Link [LNKH] (IRQs 3 4 5 7 10 11 14 15) *0, disabled.
Linux Plug and Play Support v0.97 (c) Adam Belay
pnp: PnP ACPI init
pnp: ACPI device : hid PNP0200
pnp: ACPI device : hid PNP0B00
pnp: ACPI device : hid PNP0303
pnp: ACPI device : hid PNP0F03
pnp: ACPI device : hid PNP0800
pnp: ACPI device : hid PNP0C04
pnp: ACPI device : hid PNP0700
pnp: ACPI device : hid PNP0C02
pnp: ACPI device : hid PNP0C02
pnp: ACPI device : hid PNP0C02
pnp: ACPI device : hid PNP0501
pnp: ACPI device : hid PNP0501
pnp: ACPI device : hid PNP0C01
pnp: PnP ACPI: found 13 devices
usbcore: registered new driver usbfs
usbcore: registered new driver hub
PCI: Using ACPI for IRQ routing
PCI: If a device doesn't work, try "pci=routeirq".  If it helps, post a
report
TC classifier action (bugs to netdev@vger.kernel.org cc hadi@cyberus.ca)
agpgart: Detected AGP bridge 0
agpgart: AGP aperture is 64M @ 0xec000000
PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
pnp: the driver 'system' has been registered
pnp: match found with the PnP device '00:07' and the driver 'system'
pnp: 00:07: ioport range 0x680-0x6ff has been reserved
pnp: 00:07: ioport range 0x290-0x297 has been reserved
pnp: match found with the PnP device '00:08' and the driver 'system'
pnp: match found with the PnP device '00:09' and the driver 'system'
pnp: match found with the PnP device '00:0c' and the driver 'system'
PCI: Bridge: 0000:00:01.0
  IO window: disabled.
  MEM window: faf00000-fbffffff
  PREFETCH window: f0000000-f9ffffff
PCI: Setting latency timer of device 0000:00:01.0 to 64
IA32 emulation $Id: sys_ia32.c,v 1.32 2002/03/24 13:02:28 ak Exp $
audit: initializing netlink socket (disabled)
audit(1137682942.372:1): initialized
VFS: Disk quotas dquot_6.5.1
Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
Initializing Cryptographic API
io scheduler noop registered
io scheduler anticipatory registered
io scheduler deadline registered
io scheduler cfq registered
Real Time Clock Driver v1.12
Linux agpgart interface v0.101 (c) Dave Jones
pnp: the driver 'i8042 kbd' has been registered
pnp: match found with the PnP device '00:02' and the driver 'i8042 kbd'
pnp: the driver 'i8042 aux' has been registered
pnp: match found with the PnP device '00:03' and the driver 'i8042 aux'
PNP: PS/2 Controller [PNP0303:PS2K,PNP0f03:PS2M] at 0x60,0x64 irq 1,12
serio: i8042 AUX port at 0x60,0x64 irq 12
serio: i8042 KBD port at 0x60,0x64 irq 1
Serial: 8250/16550 driver $Revision: 1.90 $ 4 ports, IRQ sharing enabled
serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
serial8250: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
serial8250: ttyS1 at I/O 0x2f8 (irq = 0) is a 16550A
serial8250: ttyS0 at I/O 0x3f8 (irq = 0) is a 16550A
pnp: the driver 'serial' has been registered
pnp: match found with the PnP device '00:0a' and the driver 'serial'
00:0a: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
pnp: match found with the PnP device '00:0b' and the driver 'serial'
00:0b: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
GSI 16 sharing vector 0xA9 and IRQ 16
ACPI: PCI Interrupt 0000:00:09.0[A] -> GSI 16 (level, low) -> IRQ 16
0000:00:09.0: ttyS2 at I/O 0xa400 (irq = 16) is a 16550A
0000:00:09.0: ttyS3 at I/O 0xa000 (irq = 16) is a 16550A
RAMDISK driver initialized: 16 RAM disks of 65536K size 1024 blocksize
usbmon: debugfs is not available
mice: PS/2 mouse device common for all mice
NET: Registered protocol family 2
IP route cache hash table entries: 131072 (order: 8, 1048576 bytes)
TCP established hash table entries: 524288 (order: 10, 4194304 bytes)
TCP bind hash table entries: 65536 (order: 7, 524288 bytes)
TCP: Hash tables configured (established 524288 bind 65536)
TCP reno registered
TCP bic registered
NET: Registered protocol family 1
NET: Registered protocol family 17
NET: Registered protocol family 8
NET: Registered protocol family 20
ACPI wakeup devices: 
PCI0 PS2K PS2M UAR2 UAR1 AC97 USB1 USB2 USB3 USB4 EHCI PWRB SLPB 
ACPI: (supports S0 S1 S3 S4 S5)
Freeing unused kernel memory: 148k freed
Uniform Multi-Platform E-IDE driver Revision: 7.00alpha2
ide: Assuming 33MHz system bus speed for PIO modes; override with
idebus=xx
input: AT Translated Set 2 keyboard as /class/input/input0
VP_IDE: IDE controller at PCI slot 0000:00:0f.1
GSI 17 sharing vector 0xB1 and IRQ 17
ACPI: PCI Interrupt 0000:00:0f.1[A] -> GSI 20 (level, low) -> IRQ 17
PCI: Via IRQ fixup for 0000:00:0f.1, from 255 to 1
VP_IDE: chipset revision 6
VP_IDE: not 100% native mode: will probe irqs later
VP_IDE: VIA vt8237 (rev 00) IDE UDMA133 controller on pci0000:00:0f.1
    ide0: BM-DMA at 0xfc00-0xfc07, BIOS settings: hda:DMA, hdb:DMA
    ide1: BM-DMA at 0xfc08-0xfc0f, BIOS settings: hdc:DMA, hdd:pio
Probing IDE interface ide0...
hda: HDS722525VLAT80, ATA DISK drive
hdb: Maxtor 6Y200P0, ATA DISK drive
isa bounce pool size: 16 pages
ide0 at 0x1f0-0x1f7,0x3f6 on irq 14
Probing IDE interface ide1...
hdc: _NEC DVD_RW ND-3500AG, ATAPI CD/DVD-ROM drive
ide1 at 0x170-0x177,0x376 on irq 15
hda: max request size: 1024KiB
hda: 390721968 sectors (200049 MB) w/7938KiB Cache, CHS=24321/255/63,
UDMA(100)
hda: cache flushes supported
 hda: hda1 hda2 hda3 < hda5 hda6 hda7 hda8 >
hdb: max request size: 1024KiB
hdb: 398297088 sectors (203928 MB) w/7936KiB Cache, CHS=24792/255/63,
UDMA(133)
hdb: cache flushes supported
 hdb: hdb1 hdb2 hdb3 hdb4 < hdb5 hdb6 >
kjournald starting.  Commit interval 5 seconds
EXT3-fs: mounted filesystem with ordered data mode.
hdc: ATAPI 48X DVD-ROM DVD-R CD-R/RW drive, 2048kB Cache, UDMA(33)
Uniform CD-ROM driver Revision: 3.20
SCSI subsystem initialized
pci_hotplug: PCI Hot Plug PCI Core version: 0.5
shpchp: Standard Hot Plug PCI Controller Driver version: 0.4
USB Universal Host Controller Interface driver v2.3
GSI 18 sharing vector 0xB9 and IRQ 18
ACPI: PCI Interrupt 0000:00:10.0[A] -> GSI 21 (level, low) -> IRQ 18
PCI: Via IRQ fixup for 0000:00:10.0, from 11 to 2
uhci_hcd 0000:00:10.0: UHCI Host Controller
uhci_hcd 0000:00:10.0: new USB bus registered, assigned bus number 1
uhci_hcd 0000:00:10.0: irq 18, io base 0x0000d400
hub 1-0:1.0: USB hub found
hub 1-0:1.0: 2 ports detected
libata version 1.20 loaded.
GSI 19 sharing vector 0xC1 and IRQ 19
ACPI: PCI Interrupt 0000:00:0a.0[A] -> GSI 17 (level, low) -> IRQ 19
skge 1.2 addr 0xfa900000 irq 19 chip Yukon-Lite rev 7
skge eth0: addr 00:11:2f:a7:fb:ef
ACPI: PCI Interrupt 0000:00:10.1[A] -> GSI 21 (level, low) -> IRQ 18
PCI: Via IRQ fixup for 0000:00:10.1, from 11 to 2
uhci_hcd 0000:00:10.1: UHCI Host Controller
uhci_hcd 0000:00:10.1: new USB bus registered, assigned bus number 2
uhci_hcd 0000:00:10.1: irq 18, io base 0x0000d800
hub 2-0:1.0: USB hub found
hub 2-0:1.0: 2 ports detected
input: PC Speaker as /class/input/input1
e100: Intel(R) PRO/100 Network Driver, 3.4.14-k4-NAPI
e100: Copyright(c) 1999-2005 Intel Corporation
GSI 20 sharing vector 0xC9 and IRQ 20
ACPI: PCI Interrupt 0000:00:0d.0[A] -> GSI 18 (level, low) -> IRQ 20
e100: eth1: e100_probe: addr 0xfac00000, irq 20, MAC addr
00:D0:B7:BF:92:C8
sata_promise 0000:00:08.0: version 1.03
ACPI: PCI Interrupt 0000:00:08.0[A] -> GSI 18 (level, low) -> IRQ 20
ata1: SATA max UDMA/133 cmd 0xFFFFC20000006200 ctl 0xFFFFC20000006238
bmdma 0x0 irq 20
ata2: SATA max UDMA/133 cmd 0xFFFFC20000006280 ctl 0xFFFFC200000062B8
bmdma 0x0 irq 20
ACPI: PCI Interrupt 0000:00:10.2[B] -> GSI 21 (level, low) -> IRQ 18
PCI: Via IRQ fixup for 0000:00:10.2, from 10 to 2
uhci_hcd 0000:00:10.2: UHCI Host Controller
uhci_hcd 0000:00:10.2: new USB bus registered, assigned bus number 3
uhci_hcd 0000:00:10.2: irq 18, io base 0x0000e000
hub 3-0:1.0: USB hub found
hub 3-0:1.0: 2 ports detected
ACPI: PCI Interrupt 0000:00:10.3[B] -> GSI 21 (level, low) -> IRQ 18
PCI: Via IRQ fixup for 0000:00:10.3, from 10 to 2
uhci_hcd 0000:00:10.3: UHCI Host Controller
uhci_hcd 0000:00:10.3: new USB bus registered, assigned bus number 4
uhci_hcd 0000:00:10.3: irq 18, io base 0x0000e400
hub 4-0:1.0: USB hub found
hub 4-0:1.0: 2 ports detected
ata1: no device found (phy stat 00000000)
scsi0 : sata_promise
ACPI: PCI Interrupt 0000:00:10.4[C] -> GSI 21 (level, low) -> IRQ 18
PCI: Via IRQ fixup for 0000:00:10.4, from 5 to 2
ehci_hcd 0000:00:10.4: EHCI Host Controller
ehci_hcd 0000:00:10.4: new USB bus registered, assigned bus number 5
ehci_hcd 0000:00:10.4: irq 18, io mem 0xfae00000
ehci_hcd 0000:00:10.4: USB 2.0 started, EHCI 1.00, driver 10 Dec 2004
hub 5-0:1.0: USB hub found
hub 5-0:1.0: 8 ports detected
ata2: no device found (phy stat 00000000)
scsi1 : sata_promise
sata_via 0000:00:0f.0: version 1.1
ACPI: PCI Interrupt 0000:00:0f.0[B] -> GSI 20 (level, low) -> IRQ 17
PCI: Via IRQ fixup for 0000:00:0f.0, from 10 to 1
sata_via 0000:00:0f.0: routed to hard irq line 1
ata3: SATA max UDMA/133 cmd 0xD000 ctl 0xC802 bmdma 0xB800 irq 17
ata4: SATA max UDMA/133 cmd 0xC400 ctl 0xC002 bmdma 0xB808 irq 17
ata3: dev 0 cfg 49:2f00 82:3069 83:7c01 84:4003 85:3069 86:3c01 87:4003
88:203f
ata3: dev 0 ATA-6, max UDMA/100, 390721968 sectors: LBA48
ata3: dev 0 configured for UDMA/100
scsi2 : sata_via
ata4: no device found (phy stat 00000000)
scsi3 : sata_via
input: ImExPS/2 Generic Explorer Mouse as /class/input/input2
  Vendor: ATA       Model: ST3200822AS       Rev: 3.01
  Type:   Direct-Access                      ANSI SCSI revision: 05
Floppy drive(s): fd0 is 1.44M
FDC 0 is a post-1991 82077
GSI 21 sharing vector 0xD1 and IRQ 21
ACPI: PCI Interrupt 0000:00:11.5[C] -> GSI 22 (level, low) -> IRQ 21
PCI: Setting latency timer of device 0000:00:11.5 to 64
SCSI device sda: 390721968 512-byte hdwr sectors (200050 MB)
SCSI device sda: drive cache: write back
SCSI device sda: 390721968 512-byte hdwr sectors (200050 MB)
SCSI device sda: drive cache: write back
 sda: sda1 sda2 < sda5 > sda3
sd 2:0:0:0: Attached scsi disk sda
Adding 1951856k swap on /dev/hda5.  Priority:-1 extents:1
across:1951856k
Adding 1951856k swap on /dev/hda6.  Priority:-2 extents:1
across:1951856k
Adding 1951856k swap on /dev/hda7.  Priority:-3 extents:1
across:1951856k
Adding 996020k swap on /dev/hdb2.  Priority:-4 extents:1 across:996020k
Adding 996020k swap on /dev/hdb3.  Priority:-5 extents:1 across:996020k
Adding 995988k swap on /dev/hdb5.  Priority:-6 extents:1 across:995988k
Adding 843372k swap on /dev/hdb6.  Priority:-7 extents:1 across:843372k
EXT3 FS on hda1, internal journal
ieee1394: Initialized config rom entry `ip1394'
sbp2: $Rev: 1306 $ Ben Collins <bcollins@debian.org>
ieee1394: sbp2: Driver forced to serialize I/O (serialize_io=1)
ieee1394: sbp2: Try serialize_io=0 for better performance
kjournald starting.  Commit interval 5 seconds
EXT3 FS on hda2, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
kjournald starting.  Commit interval 5 seconds
EXT3 FS on hda8, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
SGI XFS with ACLs, security attributes, realtime, large block/inode
numbers, no debug enabled
SGI XFS Quota Management subsystem
XFS mounting filesystem hdb1
Ending clean XFS mount for filesystem: hdb1
kjournald starting.  Commit interval 5 seconds
EXT3 FS on sda1, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
kjournald starting.  Commit interval 5 seconds
EXT3 FS on sda3, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
XFS mounting filesystem sda5
Ending clean XFS mount for filesystem: sda5
e100: intel: e100_watchdog: link up, 100Mbps, half-duplex
NET: Registered protocol family 10
lo: Disabled Privacy Extensions
IPv6 over IPv4 tunneling driver
ACPI: Power Button (FF) [PWRF]
ACPI: Power Button (CM) [PWRB]
ACPI: Sleep Button (CM) [SLPB]
pnp: the driver 'parport_pc' has been registered
lp: driver loaded but no devices found
intel: no IPv6 routers present
Installing knfsd (copyright (C) 1996 okir@monad.swb.de).
NFSD: Using /var/lib/nfs/v4recovery as the NFSv4 state recovery
directory
NFSD: recovery directory /var/lib/nfs/v4recovery doesn't exist
NFSD: starting 90-second grace period
mtrr: type mismatch for f0000000,8000000 old: write-back new:
write-combining
oom-killer: gfp_mask=0xd1, order=0

Call Trace:<ffffffff8014d7bc>{out_of_memory+48}
<ffffffff8014f4b0>{__alloc_pages+536}
       <ffffffff80169788>{bio_alloc_bioset+232}
<ffffffff80169d03>{bio_copy_user+218}
       <ffffffff801bd657>{blk_rq_map_user+136}
<ffffffff801c0008>{sg_io+328}
       <ffffffff801c047c>{scsi_cmd_ioctl+491}
<ffffffff88005e22>{:ide_core:generic_ide_ioctl+631}
       <ffffffff88202d0c>{:sd_mod:sd_ioctl+371}
<ffffffff802a6db6>{schedule_timeout+158}
       <ffffffff801bf165>{blkdev_ioctl+1365}
<ffffffff80243cb2>{sys_sendto+251}
       <ffffffff801751e5>{__pollwait+0}
<ffffffff8016b16a>{block_ioctl+25}
       <ffffffff801749f4>{do_ioctl+24} <ffffffff80174c46>{vfs_ioctl+541}
       <ffffffff80174cb4>{sys_ioctl+89}
<ffffffff8010e4ba>{system_call+126}
       
Mem-info:
DMA per-cpu:
cpu 0 hot: low 0, high 0, batch 1 used:0
cpu 0 cold: low 0, high 0, batch 1 used:0
DMA32 per-cpu:
cpu 0 hot: low 0, high 186, batch 31 used:155
cpu 0 cold: low 0, high 62, batch 15 used:56
Normal per-cpu:
cpu 0 hot: low 0, high 186, batch 31 used:150
cpu 0 cold: low 0, high 62, batch 15 used:53
HighMem per-cpu: empty
Free pages:       43224kB (0kB HighMem)
Active:48792 inactive:903480 dirty:404079 writeback:90 unstable:0
free:10806 slab:37887 mapped:48269 pagetables:1588
DMA free:20kB min:24kB low:28kB high:36kB active:0kB inactive:0kB
present:12740kB pages_scanned:2 all_unreclaimable? yes
lowmem_reserve[]: 0 2999 4009 4009
DMA32 free:34928kB min:6052kB low:7564kB high:9076kB active:700kB
inactive:2845172kB present:3071904kB pages_scanned:0 all_unreclaimable?
no
lowmem_reserve[]: 0 0 1010 1010
Normal free:8276kB min:2036kB low:2544kB high:3052kB active:194468kB
inactive:768748kB present:1034240kB pages_scanned:0 all_unreclaimable?
no
lowmem_reserve[]: 0 0 0 0
HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB
present:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
DMA: 1*4kB 0*8kB 1*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB
0*2048kB 0*4096kB = 20kB
DMA32: 4390*4kB 61*8kB 43*16kB 36*32kB 37*64kB 11*128kB 0*256kB 0*512kB
1*1024kB 1*2048kB 2*4096kB = 34928kB
Normal: 863*4kB 85*8kB 45*16kB 17*32kB 9*64kB 6*128kB 0*256kB 1*512kB
1*1024kB 0*2048kB 0*4096kB = 8276kB
HighMem: empty
Swap cache: add 38, delete 0, find 0/0, race 0+0
Free swap  = 9686816kB
Total swap = 9686968kB
Free swap:       9686816kB
1310720 pages of RAM
299467 reserved pages
940324 pages shared
38 pages swap cached
Out of Memory: Killed process 6780 (java).
oom-killer: gfp_mask=0xd1, order=0

Call Trace:<ffffffff8014d7bc>{out_of_memory+48}
<ffffffff8014f4b0>{__alloc_pages+536}
       <ffffffff80169788>{bio_alloc_bioset+232}
<ffffffff80169d03>{bio_copy_user+218}
       <ffffffff801bd657>{blk_rq_map_user+136}
<ffffffff801c0008>{sg_io+328}
       <ffffffff801c047c>{scsi_cmd_ioctl+491}
<ffffffff88005e22>{:ide_core:generic_ide_ioctl+631}
       <ffffffff88202d0c>{:sd_mod:sd_ioctl+371}
<ffffffff802a6db6>{schedule_timeout+158}
       <ffffffff801bf165>{blkdev_ioctl+1365}
<ffffffff80243cb2>{sys_sendto+251}
       <ffffffff801751e5>{__pollwait+0}
<ffffffff8016b16a>{block_ioctl+25}
       <ffffffff801749f4>{do_ioctl+24} <ffffffff80174c46>{vfs_ioctl+541}
       <ffffffff80174cb4>{sys_ioctl+89}
<ffffffff8010e4ba>{system_call+126}
       
Mem-info:
DMA per-cpu:
cpu 0 hot: low 0, high 0, batch 1 used:0
cpu 0 cold: low 0, high 0, batch 1 used:0
DMA32 per-cpu:
cpu 0 hot: low 0, high 186, batch 31 used:178
cpu 0 cold: low 0, high 62, batch 15 used:56
Normal per-cpu:
cpu 0 hot: low 0, high 186, batch 31 used:160
cpu 0 cold: low 0, high 62, batch 15 used:53
HighMem per-cpu: empty
Free pages:       87768kB (0kB HighMem)
Active:37781 inactive:903473 dirty:404087 writeback:90 unstable:0
free:21942 slab:37866 mapped:37226 pagetables:1509
DMA free:20kB min:24kB low:28kB high:36kB active:0kB inactive:0kB
present:12740kB pages_scanned:0 all_unreclaimable? yes
lowmem_reserve[]: 0 2999 4009 4009
DMA32 free:35300kB min:6052kB low:7564kB high:9076kB active:260kB
inactive:2845148kB present:3071904kB pages_scanned:0 all_unreclaimable?
no
lowmem_reserve[]: 0 0 1010 1010
Normal free:52448kB min:2036kB low:2544kB high:3052kB active:150864kB
inactive:768744kB present:1034240kB pages_scanned:0 all_unreclaimable?
no
lowmem_reserve[]: 0 0 0 0
HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB
present:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
DMA: 1*4kB 0*8kB 1*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB
0*2048kB 0*4096kB = 20kB
DMA32: 4483*4kB 61*8kB 43*16kB 36*32kB 37*64kB 11*128kB 0*256kB 0*512kB
1*1024kB 1*2048kB 2*4096kB = 35300kB
Normal: 1618*4kB 479*8kB 324*16kB 231*32kB 124*64kB 59*128kB 21*256kB
9*512kB 2*1024kB 1*2048kB 0*4096kB = 52448kB
HighMem: empty
Swap cache: add 38, delete 0, find 0/0, race 0+0
Free swap  = 9686816kB
Total swap = 9686968kB
Free swap:       9686816kB
1310720 pages of RAM
299467 reserved pages
936211 pages shared
38 pages swap cached
oom-killer: gfp_mask=0xd1, order=0

Call Trace:<ffffffff8014d7bc>{out_of_memory+48}
<ffffffff8014f4b0>{__alloc_pages+536}
       <ffffffff80169788>{bio_alloc_bioset+232}
<ffffffff80169d03>{bio_copy_user+218}
       <ffffffff801bd657>{blk_rq_map_user+136}
<ffffffff801c0008>{sg_io+328}
       <ffffffff801c047c>{scsi_cmd_ioctl+491}
<ffffffff88005e22>{:ide_core:generic_ide_ioctl+631}
       <ffffffff88202d0c>{:sd_mod:sd_ioctl+371}
<ffffffff802a6db6>{schedule_timeout+158}
       <ffffffff801bf165>{blkdev_ioctl+1365}
<ffffffff80243cb2>{sys_sendto+251}
       <ffffffff801751e5>{__pollwait+0}
<ffffffff8016b16a>{block_ioctl+25}
       <ffffffff801749f4>{do_ioctl+24} <ffffffff80174c46>{vfs_ioctl+541}
       <ffffffff80174cb4>{sys_ioctl+89}
<ffffffff8010e4ba>{system_call+126}
       
Mem-info:
DMA per-cpu:
cpu 0 hot: low 0, high 0, batch 1 used:0
cpu 0 cold: low 0, high 0, batch 1 used:0
DMA32 per-cpu:
cpu 0 hot: low 0, high 186, batch 31 used:178
cpu 0 cold: low 0, high 62, batch 15 used:56
Normal per-cpu:
cpu 0 hot: low 0, high 186, batch 31 used:161
cpu 0 cold: low 0, high 62, batch 15 used:53
HighMem per-cpu: empty
Free pages:       87768kB (0kB HighMem)
Active:37781 inactive:903473 dirty:404087 writeback:90 unstable:0
free:21942 slab:37865 mapped:37226 pagetables:1509
DMA free:20kB min:24kB low:28kB high:36kB active:0kB inactive:0kB
present:12740kB pages_scanned:0 all_unreclaimable? yes
lowmem_reserve[]: 0 2999 4009 4009
DMA32 free:35300kB min:6052kB low:7564kB high:9076kB active:260kB
inactive:2845148kB present:3071904kB pages_scanned:0 all_unreclaimable?
no
lowmem_reserve[]: 0 0 1010 1010
Normal free:52448kB min:2036kB low:2544kB high:3052kB active:150864kB
inactive:768744kB present:1034240kB pages_scanned:0 all_unreclaimable?
no
lowmem_reserve[]: 0 0 0 0
HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB
present:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
DMA: 1*4kB 0*8kB 1*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB
0*2048kB 0*4096kB = 20kB
DMA32: 4483*4kB 61*8kB 43*16kB 36*32kB 37*64kB 11*128kB 0*256kB 0*512kB
1*1024kB 1*2048kB 2*4096kB = 35300kB
Normal: 1618*4kB 479*8kB 324*16kB 231*32kB 124*64kB 59*128kB 21*256kB
9*512kB 2*1024kB 1*2048kB 0*4096kB = 52448kB
HighMem: empty
Swap cache: add 38, delete 0, find 0/0, race 0+0
Free swap  = 9686816kB
Total swap = 9686968kB
Free swap:       9686816kB
1310720 pages of RAM
299467 reserved pages
936211 pages shared
38 pages swap cached
oom-killer: gfp_mask=0xd1, order=0

Call Trace:<ffffffff8014d7bc>{out_of_memory+48}
<ffffffff8014f4b0>{__alloc_pages+536}
       <ffffffff80169788>{bio_alloc_bioset+232}
<ffffffff80169d03>{bio_copy_user+218}
       <ffffffff801bd657>{blk_rq_map_user+136}
<ffffffff801c0008>{sg_io+328}
       <ffffffff801c047c>{scsi_cmd_ioctl+491}
<ffffffff88005e22>{:ide_core:generic_ide_ioctl+631}
       <ffffffff88202d0c>{:sd_mod:sd_ioctl+371}
<ffffffff802a6db6>{schedule_timeout+158}
       <ffffffff801bf165>{blkdev_ioctl+1365}
<ffffffff80243cb2>{sys_sendto+251}
       <ffffffff801751e5>{__pollwait+0}
<ffffffff8016b16a>{block_ioctl+25}
       <ffffffff801749f4>{do_ioctl+24} <ffffffff80174c46>{vfs_ioctl+541}
       <ffffffff80174cb4>{sys_ioctl+89}
<ffffffff8010e4ba>{system_call+126}
       
Mem-info:
DMA per-cpu:
cpu 0 hot: low 0, high 0, batch 1 used:0
cpu 0 cold: low 0, high 0, batch 1 used:0
DMA32 per-cpu:
cpu 0 hot: low 0, high 186, batch 31 used:179
cpu 0 cold: low 0, high 62, batch 15 used:56
Normal per-cpu:
cpu 0 hot: low 0, high 186, batch 31 used:170
cpu 0 cold: low 0, high 62, batch 15 used:53
HighMem per-cpu: empty
Free pages:       87768kB (0kB HighMem)
Active:37781 inactive:903473 dirty:404087 writeback:90 unstable:0
free:21942 slab:37855 mapped:37226 pagetables:1509
DMA free:20kB min:24kB low:28kB high:36kB active:0kB inactive:0kB
present:12740kB pages_scanned:2 all_unreclaimable? yes
lowmem_reserve[]: 0 2999 4009 4009
DMA32 free:35300kB min:6052kB low:7564kB high:9076kB active:260kB
inactive:2845148kB present:3071904kB pages_scanned:0 all_unreclaimable?
no
lowmem_reserve[]: 0 0 1010 1010
Normal free:52448kB min:2036kB low:2544kB high:3052kB active:150864kB
inactive:768744kB present:1034240kB pages_scanned:0 all_unreclaimable?
no
lowmem_reserve[]: 0 0 0 0
HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB
present:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
DMA: 1*4kB 0*8kB 1*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB
0*2048kB 0*4096kB = 20kB
DMA32: 4483*4kB 61*8kB 43*16kB 36*32kB 37*64kB 11*128kB 0*256kB 0*512kB
1*1024kB 1*2048kB 2*4096kB = 35300kB
Normal: 1618*4kB 479*8kB 324*16kB 231*32kB 124*64kB 59*128kB 21*256kB
9*512kB 2*1024kB 1*2048kB 0*4096kB = 52448kB
HighMem: empty
Swap cache: add 38, delete 0, find 0/0, race 0+0
Free swap  = 9686816kB
Total swap = 9686968kB
Free swap:       9686816kB
1310720 pages of RAM
299467 reserved pages
936211 pages shared
38 pages swap cached
oom-killer: gfp_mask=0xd1, order=0

Call Trace:<ffffffff8014d7bc>{out_of_memory+48}
<ffffffff8014f4b0>{__alloc_pages+536}
       <ffffffff80169788>{bio_alloc_bioset+232}
<ffffffff80169d03>{bio_copy_user+218}
       <ffffffff801bd657>{blk_rq_map_user+136}
<ffffffff801c0008>{sg_io+328}
       <ffffffff801c047c>{scsi_cmd_ioctl+491}
<ffffffff88005e22>{:ide_core:generic_ide_ioctl+631}
       <ffffffff88202d0c>{:sd_mod:sd_ioctl+371}
<ffffffff802a6db6>{schedule_timeout+158}
       <ffffffff801bf165>{blkdev_ioctl+1365}
<ffffffff80243cb2>{sys_sendto+251}
       <ffffffff801751e5>{__pollwait+0}
<ffffffff8016b16a>{block_ioctl+25}
       <ffffffff801749f4>{do_ioctl+24} <ffffffff80174c46>{vfs_ioctl+541}
       <ffffffff80174cb4>{sys_ioctl+89}
<ffffffff8010e4ba>{system_call+126}
       
Mem-info:
DMA per-cpu:
cpu 0 hot: low 0, high 0, batch 1 used:0
cpu 0 cold: low 0, high 0, batch 1 used:0
DMA32 per-cpu:
cpu 0 hot: low 0, high 186, batch 31 used:179
cpu 0 cold: low 0, high 62, batch 15 used:56
Normal per-cpu:
cpu 0 hot: low 0, high 186, batch 31 used:177
cpu 0 cold: low 0, high 62, batch 15 used:53
HighMem per-cpu: empty
Free pages:       87768kB (0kB HighMem)
Active:37781 inactive:903473 dirty:404070 writeback:45 unstable:0
free:21942 slab:37848 mapped:37226 pagetables:1509
DMA free:20kB min:24kB low:28kB high:36kB active:0kB inactive:0kB
present:12740kB pages_scanned:0 all_unreclaimable? yes
lowmem_reserve[]: 0 2999 4009 4009
DMA32 free:35300kB min:6052kB low:7564kB high:9076kB active:260kB
inactive:2845148kB present:3071904kB pages_scanned:0 all_unreclaimable?
no
lowmem_reserve[]: 0 0 1010 1010
Normal free:52448kB min:2036kB low:2544kB high:3052kB active:150864kB
inactive:768744kB present:1034240kB pages_scanned:0 all_unreclaimable?
no
lowmem_reserve[]: 0 0 0 0
HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB
present:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
DMA: 1*4kB 0*8kB 1*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB
0*2048kB 0*4096kB = 20kB
DMA32: 4483*4kB 61*8kB 43*16kB 36*32kB 37*64kB 11*128kB 0*256kB 0*512kB
1*1024kB 1*2048kB 2*4096kB = 35300kB
Normal: 1618*4kB 479*8kB 324*16kB 231*32kB 124*64kB 59*128kB 21*256kB
9*512kB 2*1024kB 1*2048kB 0*4096kB = 52448kB
HighMem: empty
Swap cache: add 38, delete 0, find 0/0, race 0+0
Free swap  = 9686816kB
Total swap = 9686968kB
Free swap:       9686816kB
1310720 pages of RAM
299467 reserved pages
936211 pages shared
38 pages swap cached
oom-killer: gfp_mask=0xd1, order=0

Call Trace:<ffffffff8014d7bc>{out_of_memory+48}
<ffffffff8014f4b0>{__alloc_pages+536}
       <ffffffff80169788>{bio_alloc_bioset+232}
<ffffffff80169d03>{bio_copy_user+218}
       <ffffffff801bd657>{blk_rq_map_user+136}
<ffffffff801c0008>{sg_io+328}
       <ffffffff801c047c>{scsi_cmd_ioctl+491}
<ffffffff88005e22>{:ide_core:generic_ide_ioctl+631}
       <ffffffff88202d0c>{:sd_mod:sd_ioctl+371}
<ffffffff802a6db6>{schedule_timeout+158}
       <ffffffff801bf165>{blkdev_ioctl+1365}
<ffffffff80243cb2>{sys_sendto+251}
       <ffffffff801751e5>{__pollwait+0}
<ffffffff8016b16a>{block_ioctl+25}
       <ffffffff801749f4>{do_ioctl+24} <ffffffff80174c46>{vfs_ioctl+541}
       <ffffffff80174cb4>{sys_ioctl+89}
<ffffffff8010e4ba>{system_call+126}
       
Mem-info:
DMA per-cpu:
cpu 0 hot: low 0, high 0, batch 1 used:0
cpu 0 cold: low 0, high 0, batch 1 used:0
DMA32 per-cpu:
cpu 0 hot: low 0, high 186, batch 31 used:179
cpu 0 cold: low 0, high 62, batch 15 used:56
Normal per-cpu:
cpu 0 hot: low 0, high 186, batch 31 used:177
cpu 0 cold: low 0, high 62, batch 15 used:53
HighMem per-cpu: empty
Free pages:       87768kB (0kB HighMem)
Active:37781 inactive:903473 dirty:404070 writeback:45 unstable:0
free:21942 slab:37848 mapped:37226 pagetables:1509
DMA free:20kB min:24kB low:28kB high:36kB active:0kB inactive:0kB
present:12740kB pages_scanned:0 all_unreclaimable? yes
lowmem_reserve[]: 0 2999 4009 4009
DMA32 free:35300kB min:6052kB low:7564kB high:9076kB active:260kB
inactive:2845148kB present:3071904kB pages_scanned:0 all_unreclaimable?
no
lowmem_reserve[]: 0 0 1010 1010
Normal free:52448kB min:2036kB low:2544kB high:3052kB active:150864kB
inactive:768744kB present:1034240kB pages_scanned:0 all_unreclaimable?
no
lowmem_reserve[]: 0 0 0 0
HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB
present:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
DMA: 1*4kB 0*8kB 1*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB
0*2048kB 0*4096kB = 20kB
DMA32: 4483*4kB 61*8kB 43*16kB 36*32kB 37*64kB 11*128kB 0*256kB 0*512kB
1*1024kB 1*2048kB 2*4096kB = 35300kB
Normal: 1618*4kB 479*8kB 324*16kB 231*32kB 124*64kB 59*128kB 21*256kB
9*512kB 2*1024kB 1*2048kB 0*4096kB = 52448kB
HighMem: empty
Swap cache: add 38, delete 0, find 0/0, race 0+0
Free swap  = 9686816kB
Total swap = 9686968kB
Free swap:       9686816kB
1310720 pages of RAM
299467 reserved pages
936211 pages shared
38 pages swap cached
oom-killer: gfp_mask=0xd1, order=0

Call Trace:<ffffffff8014d7bc>{out_of_memory+48}
<ffffffff8014f4b0>{__alloc_pages+536}
       <ffffffff80169788>{bio_alloc_bioset+232}
<ffffffff80169d03>{bio_copy_user+218}
       <ffffffff801bd657>{blk_rq_map_user+136}
<ffffffff801c0008>{sg_io+328}
       <ffffffff801c047c>{scsi_cmd_ioctl+491}
<ffffffff88005e22>{:ide_core:generic_ide_ioctl+631}
       <ffffffff88202d0c>{:sd_mod:sd_ioctl+371}
<ffffffff802a6db6>{schedule_timeout+158}
       <ffffffff801bf165>{blkdev_ioctl+1365}
<ffffffff80243cb2>{sys_sendto+251}
       <ffffffff801751e5>{__pollwait+0}
<ffffffff8016b16a>{block_ioctl+25}
       <ffffffff801749f4>{do_ioctl+24} <ffffffff80174c46>{vfs_ioctl+541}
       <ffffffff80174cb4>{sys_ioctl+89}
<ffffffff8010e4ba>{system_call+126}
       
Mem-info:
DMA per-cpu:
cpu 0 hot: low 0, high 0, batch 1 used:0
cpu 0 cold: low 0, high 0, batch 1 used:0
DMA32 per-cpu:
cpu 0 hot: low 0, high 186, batch 31 used:179
cpu 0 cold: low 0, high 62, batch 15 used:56
Normal per-cpu:
cpu 0 hot: low 0, high 186, batch 31 used:177
cpu 0 cold: low 0, high 62, batch 15 used:53
HighMem per-cpu: empty
Free pages:       87768kB (0kB HighMem)
Active:37781 inactive:903473 dirty:404070 writeback:39 unstable:0
free:21942 slab:37848 mapped:37226 pagetables:1509
DMA free:20kB min:24kB low:28kB high:36kB active:0kB inactive:0kB
present:12740kB pages_scanned:2 all_unreclaimable? yes
lowmem_reserve[]: 0 2999 4009 4009
DMA32 free:35300kB min:6052kB low:7564kB high:9076kB active:260kB
inactive:2845148kB present:3071904kB pages_scanned:0 all_unreclaimable?
no
lowmem_reserve[]: 0 0 1010 1010
Normal free:52448kB min:2036kB low:2544kB high:3052kB active:150864kB
inactive:768744kB present:1034240kB pages_scanned:0 all_unreclaimable?
no
lowmem_reserve[]: 0 0 0 0
HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB
present:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
DMA: 1*4kB 0*8kB 1*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB
0*2048kB 0*4096kB = 20kB
DMA32: 4483*4kB 61*8kB 43*16kB 36*32kB 37*64kB 11*128kB 0*256kB 0*512kB
1*1024kB 1*2048kB 2*4096kB = 35300kB
Normal: 1618*4kB 479*8kB 324*16kB 231*32kB 124*64kB 59*128kB 21*256kB
9*512kB 2*1024kB 1*2048kB 0*4096kB = 52448kB
HighMem: empty
Swap cache: add 38, delete 0, find 0/0, race 0+0
Free swap  = 9686816kB
Total swap = 9686968kB
Free swap:       9686816kB
1310720 pages of RAM
299467 reserved pages
936211 pages shared
38 pages swap cached
oom-killer: gfp_mask=0xd1, order=0

Call Trace:<ffffffff8014d7bc>{out_of_memory+48}
<ffffffff8014f4b0>{__alloc_pages+536}
       <ffffffff80169788>{bio_alloc_bioset+232}
<ffffffff80169d03>{bio_copy_user+218}
       <ffffffff801bd657>{blk_rq_map_user+136}
<ffffffff801c0008>{sg_io+328}
       <ffffffff801c047c>{scsi_cmd_ioctl+491}
<ffffffff88005e22>{:ide_core:generic_ide_ioctl+631}
       <ffffffff88202d0c>{:sd_mod:sd_ioctl+371}
<ffffffff802a6db6>{schedule_timeout+158}
       <ffffffff801bf165>{blkdev_ioctl+1365}
<ffffffff80243cb2>{sys_sendto+251}
       <ffffffff801751e5>{__pollwait+0}
<ffffffff8016b16a>{block_ioctl+25}
       <ffffffff801749f4>{do_ioctl+24} <ffffffff80174c46>{vfs_ioctl+541}
       <ffffffff80174cb4>{sys_ioctl+89}
<ffffffff8010e4ba>{system_call+126}
       
Mem-info:
DMA per-cpu:
cpu 0 hot: low 0, high 0, batch 1 used:0
cpu 0 cold: low 0, high 0, batch 1 used:0
DMA32 per-cpu:
cpu 0 hot: low 0, high 186, batch 31 used:179
cpu 0 cold: low 0, high 62, batch 15 used:56
Normal per-cpu:
cpu 0 hot: low 0, high 186, batch 31 used:176
cpu 0 cold: low 0, high 62, batch 15 used:53
HighMem per-cpu: empty
Free pages:       87768kB (0kB HighMem)
Active:37781 inactive:903473 dirty:404070 writeback:28 unstable:0
free:21942 slab:37848 mapped:37226 pagetables:1509
DMA free:20kB min:24kB low:28kB high:36kB active:0kB inactive:0kB
present:12740kB pages_scanned:2 all_unreclaimable? yes
lowmem_reserve[]: 0 2999 4009 4009
DMA32 free:35300kB min:6052kB low:7564kB high:9076kB active:260kB
inactive:2845148kB present:3071904kB pages_scanned:0 all_unreclaimable?
no
lowmem_reserve[]: 0 0 1010 1010
Normal free:52448kB min:2036kB low:2544kB high:3052kB active:150864kB
inactive:768744kB present:1034240kB pages_scanned:0 all_unreclaimable?
no
lowmem_reserve[]: 0 0 0 0
HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB
present:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
DMA: 1*4kB 0*8kB 1*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB
0*2048kB 0*4096kB = 20kB
DMA32: 4483*4kB 61*8kB 43*16kB 36*32kB 37*64kB 11*128kB 0*256kB 0*512kB
1*1024kB 1*2048kB 2*4096kB = 35300kB
Normal: 1618*4kB 479*8kB 324*16kB 231*32kB 124*64kB 59*128kB 21*256kB
9*512kB 2*1024kB 1*2048kB 0*4096kB = 52448kB
HighMem: empty
Swap cache: add 38, delete 0, find 0/0, race 0+0
Free swap  = 9686816kB
Total swap = 9686968kB
Free swap:       9686816kB
1310720 pages of RAM
299467 reserved pages
936213 pages shared
38 pages swap cached
oom-killer: gfp_mask=0xd1, order=0

Call Trace:<ffffffff8014d7bc>{out_of_memory+48}
<ffffffff8014f4b0>{__alloc_pages+536}
       <ffffffff80169788>{bio_alloc_bioset+232}
<ffffffff80169d03>{bio_copy_user+218}
       <ffffffff801bd657>{blk_rq_map_user+136}
<ffffffff801c0008>{sg_io+328}
       <ffffffff801c047c>{scsi_cmd_ioctl+491}
<ffffffff88005e22>{:ide_core:generic_ide_ioctl+631}
       <ffffffff88202d0c>{:sd_mod:sd_ioctl+371}
<ffffffff802a6db6>{schedule_timeout+158}
       <ffffffff801bf165>{blkdev_ioctl+1365}
<ffffffff80243cb2>{sys_sendto+251}
       <ffffffff801751e5>{__pollwait+0}
<ffffffff8016b16a>{block_ioctl+25}
       <ffffffff801749f4>{do_ioctl+24} <ffffffff80174c46>{vfs_ioctl+541}
       <ffffffff80174cb4>{sys_ioctl+89}
<ffffffff8010e4ba>{system_call+126}
       
Mem-info:
DMA per-cpu:
cpu 0 hot: low 0, high 0, batch 1 used:0
cpu 0 cold: low 0, high 0, batch 1 used:0
DMA32 per-cpu:
cpu 0 hot: low 0, high 186, batch 31 used:179
cpu 0 cold: low 0, high 62, batch 15 used:56
Normal per-cpu:
cpu 0 hot: low 0, high 186, batch 31 used:176
cpu 0 cold: low 0, high 62, batch 15 used:53
HighMem per-cpu: empty
Free pages:       87768kB (0kB HighMem)
Active:37781 inactive:903474 dirty:404070 writeback:13 unstable:0
free:21942 slab:37848 mapped:37226 pagetables:1509
DMA free:20kB min:24kB low:28kB high:36kB active:0kB inactive:0kB
present:12740kB pages_scanned:4 all_unreclaimable? yes
lowmem_reserve[]: 0 2999 4009 4009
DMA32 free:35300kB min:6052kB low:7564kB high:9076kB active:260kB
inactive:2845148kB present:3071904kB pages_scanned:0 all_unreclaimable?
no
lowmem_reserve[]: 0 0 1010 1010
Normal free:52448kB min:2036kB low:2544kB high:3052kB active:150864kB
inactive:768748kB present:1034240kB pages_scanned:0 all_unreclaimable?
no
lowmem_reserve[]: 0 0 0 0
HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB
present:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
DMA: 1*4kB 0*8kB 1*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB
0*2048kB 0*4096kB = 20kB
DMA32: 4483*4kB 61*8kB 43*16kB 36*32kB 37*64kB 11*128kB 0*256kB 0*512kB
1*1024kB 1*2048kB 2*4096kB = 35300kB
Normal: 1618*4kB 479*8kB 324*16kB 231*32kB 124*64kB 59*128kB 21*256kB
9*512kB 2*1024kB 1*2048kB 0*4096kB = 52448kB
HighMem: empty
Swap cache: add 38, delete 0, find 0/0, race 0+0
Free swap  = 9686816kB
Total swap = 9686968kB
Free swap:       9686816kB
1310720 pages of RAM
299467 reserved pages
936212 pages shared
38 pages swap cached
oom-killer: gfp_mask=0xd1, order=0

Call Trace:<ffffffff8014d7bc>{out_of_memory+48}
<ffffffff8014f4b0>{__alloc_pages+536}
       <ffffffff80169788>{bio_alloc_bioset+232}
<ffffffff80169d03>{bio_copy_user+218}
       <ffffffff801bd657>{blk_rq_map_user+136}
<ffffffff801c0008>{sg_io+328}
       <ffffffff801c047c>{scsi_cmd_ioctl+491}
<ffffffff88005e22>{:ide_core:generic_ide_ioctl+631}
       <ffffffff88202d0c>{:sd_mod:sd_ioctl+371}
<ffffffff802a6db6>{schedule_timeout+158}
       <ffffffff801bf165>{blkdev_ioctl+1365}
<ffffffff80243cb2>{sys_sendto+251}
       <ffffffff801751e5>{__pollwait+0}
<ffffffff8016b16a>{block_ioctl+25}
       <ffffffff801749f4>{do_ioctl+24} <ffffffff80174c46>{vfs_ioctl+541}
       <ffffffff80174cb4>{sys_ioctl+89}
<ffffffff8010e4ba>{system_call+126}
       
Mem-info:
DMA per-cpu:
cpu 0 hot: low 0, high 0, batch 1 used:0
cpu 0 cold: low 0, high 0, batch 1 used:0
DMA32 per-cpu:
cpu 0 hot: low 0, high 186, batch 31 used:179
cpu 0 cold: low 0, high 62, batch 15 used:56
Normal per-cpu:
cpu 0 hot: low 0, high 186, batch 31 used:174
cpu 0 cold: low 0, high 62, batch 15 used:53
HighMem per-cpu: empty
Free pages:       87768kB (0kB HighMem)
Active:37781 inactive:903474 dirty:404070 writeback:4 unstable:0
free:21942 slab:37848 mapped:37226 pagetables:1509
DMA free:20kB min:24kB low:28kB high:36kB active:0kB inactive:0kB
present:12740kB pages_scanned:4 all_unreclaimable? yes
lowmem_reserve[]: 0 2999 4009 4009
DMA32 free:35300kB min:6052kB low:7564kB high:9076kB active:260kB
inactive:2845148kB present:3071904kB pages_scanned:0 all_unreclaimable?
no
lowmem_reserve[]: 0 0 1010 1010
Normal free:52448kB min:2036kB low:2544kB high:3052kB active:150864kB
inactive:768748kB present:1034240kB pages_scanned:0 all_unreclaimable?
no
lowmem_reserve[]: 0 0 0 0
HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB
present:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
DMA: 1*4kB 0*8kB 1*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB
0*2048kB 0*4096kB = 20kB
DMA32: 4483*4kB 61*8kB 43*16kB 36*32kB 37*64kB 11*128kB 0*256kB 0*512kB
1*1024kB 1*2048kB 2*4096kB = 35300kB
Normal: 1618*4kB 479*8kB 324*16kB 231*32kB 124*64kB 59*128kB 21*256kB
9*512kB 2*1024kB 1*2048kB 0*4096kB = 52448kB
HighMem: empty
Swap cache: add 38, delete 0, find 0/0, race 0+0
Free swap  = 9686816kB
Total swap = 9686968kB
Free swap:       9686816kB
1310720 pages of RAM
299467 reserved pages
936216 pages shared
38 pages swap cached
Out of Memory: Killed process 6179 (nautilus).
Out of Memory: Killed process 6203 (clock-applet).
Out of Memory: Killed process 6141 (ssh-agent).
Out of Memory: Killed process 6957 (gnome-session).
Out of Memory: Killed process 6158 (gnome-settings-).
Out of Memory: Killed process 6166 (gnome-settings-).
Out of Memory: Killed process 6205 (mixer_applet2).
Out of Memory: Killed process 6175 (gnome-panel).
Out of Memory: Killed process 6181 (wnck-applet).
Out of Memory: Killed process 6190 (multiload-apple).

-- 
Andy, BlueArc Engineering

^ permalink raw reply	[flat|nested] 29+ messages in thread

* Re: Out of Memory: Killed process 16498 (java).
       [not found] <89E85E0168AD994693B574C80EDB9C2703555F8E@uk-email.terastack.bluearc.com>
@ 2006-01-19  9:44 ` Andrew Morton
  0 siblings, 0 replies; 29+ messages in thread
From: Andrew Morton @ 2006-01-19  9:44 UTC (permalink / raw)
  To: Andy Chittenden; +Cc: linux-kernel

"Andy Chittenden" <AChittenden@bluearc.com> wrote:
>
> > It'd be useful to see the dmesg output from that oom event.
> 
> Well, I've just run it again and loads of processes got killed (at least
> 3). Here's the output from dmesg:
>
> ...
>

> oom-killer: gfp_mask=0xd1, order=0
> Mem-info:
> DMA per-cpu:
> cpu 0 hot: low 0, high 0, batch 1 used:0
> cpu 0 cold: low 0, high 0, batch 1 used:0
> DMA32 per-cpu:
> cpu 0 hot: low 0, high 186, batch 31 used:162
> cpu 0 cold: low 0, high 62, batch 15 used:51
> Normal per-cpu:
> cpu 0 hot: low 0, high 186, batch 31 used:21
> cpu 0 cold: low 0, high 62, batch 15 used:41
> HighMem per-cpu: empty
> Free pages:     3116636kB (0kB HighMem)
> Active:52119 inactive:158700 dirty:108550 writeback:9545 unstable:0
> free:779159 slab:10643 mapped:44283 pagetables:1750
> DMA free:20kB min:24kB low:28kB high:36kB active:0kB inactive:0kB
> present:12740kB pages_scanned:18 all_unreclaimable? yes
> lowmem_reserve[]: 0 2999 4009 4009
> DMA32 free:2528816kB min:6052kB low:7564kB high:9076kB active:18312kB
> inactive:422164kB present:3071904kB pages_scanned:0 all_unreclaimable?
> no
> lowmem_reserve[]: 0 0 1010 1010
> Normal free:587800kB min:2036kB low:2544kB high:3052kB active:190164kB
> inactive:212636kB present:1034240kB pages_scanned:0 all_unreclaimable?
> no
> lowmem_reserve[]: 0 0 0 0
> HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB
> present:0kB pages_scanned:0 all_unreclaimable? no
> lowmem_reserve[]: 0 0 0 0
> DMA: 1*4kB 0*8kB 1*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB
> 0*2048kB 0*4096kB = 20kB
> DMA32: 3630*4kB 3233*8kB 3673*16kB 3553*32kB 2973*64kB 1729*128kB
> 763*256kB 418*512kB 458*1024kB 221*2048kB 140*4096kB = 2528816kB
> Normal: 17422*4kB 12118*8kB 7607*16kB 3846*32kB 1606*64kB 449*128kB
> 57*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 587800kB
> HighMem: empty
> Swap cache: add 38, delete 38, find 0/0, race 0+0
> Free swap  = 9686816kB
> Total swap = 9686968kB
> Free swap:       9686816kB
> 1310720 pages of RAM
> 299467 reserved pages
> 213833 pages shared
> 0 pages swap cached

You have 3G of free memory, 2.5G in the DMA32 zone, 0.5G in the NORMAL
zone.

> Out of Memory: Killed process 6179 (nautilus).
> oom-killer: gfp_mask=0xd1, order=0

What we've run out of is the old ZONE_DMA memory.

Can you please add this patch, retest?

--- devel/mm/oom_kill.c~a	2006-01-19 01:43:31.000000000 -0800
+++ devel-akpm/mm/oom_kill.c	2006-01-19 01:43:39.000000000 -0800
@@ -271,6 +271,7 @@ void out_of_memory(gfp_t gfp_mask, int o
 	if (printk_ratelimit()) {
 		printk("oom-killer: gfp_mask=0x%x, order=%d\n",
 			gfp_mask, order);
+		dump_stack();
 		show_mem();
 	}
 
_


^ permalink raw reply	[flat|nested] 29+ messages in thread

* Re: Out of Memory: Killed process 16498 (java).
  2006-01-19  9:40 Andy Chittenden
@ 2006-01-19  9:41 ` Con Kolivas
  0 siblings, 0 replies; 29+ messages in thread
From: Con Kolivas @ 2006-01-19  9:41 UTC (permalink / raw)
  To: Andy Chittenden; +Cc: Andrew Morton, linux-kernel

On Thursday 19 January 2006 20:40, Andy Chittenden wrote:
> > Are you using scsi? Someone just posted what looks to be a
> > scsi slab leak (Re:
> > scsi cmd slab leak? (Was Re: [ck] Anyone been having OOM
> > killer problems
> > lately?) that causes oom kills. Check your slabinfo for a large
> > scsi_cmd_cache.
>
> Not using scsi.  Using ide and sata. The target for the dd command was
> an ide disk.
>
> # cat /proc/slabinfo | grep scsi
> scsi_cmd_cache         4      8    448    8    1 : tunables   54   27
> 0 : slabdata      1      1      0
>
> Is that large?

Heh, absolutely not. It was a shot in the dark anyway. Good luck.

Cheers,
Con

^ permalink raw reply	[flat|nested] 29+ messages in thread

* RE: Out of Memory: Killed process 16498 (java).
@ 2006-01-19  9:40 Andy Chittenden
  2006-01-19  9:41 ` Con Kolivas
  0 siblings, 1 reply; 29+ messages in thread
From: Andy Chittenden @ 2006-01-19  9:40 UTC (permalink / raw)
  To: Con Kolivas, Andrew Morton; +Cc: linux-kernel

> Are you using scsi? Someone just posted what looks to be a 
> scsi slab leak (Re: 
> scsi cmd slab leak? (Was Re: [ck] Anyone been having OOM 
> killer problems 
> lately?) that causes oom kills. Check your slabinfo for a large 
> scsi_cmd_cache.

Not using scsi.  Using ide and sata. The target for the dd command was
an ide disk.

# cat /proc/slabinfo | grep scsi
scsi_cmd_cache         4      8    448    8    1 : tunables   54   27
0 : slabdata      1      1      0

Is that large?

-- 
Andy, BlueArc Engineering

^ permalink raw reply	[flat|nested] 29+ messages in thread

* Re: Out of Memory: Killed process 16498 (java).
  2006-01-19  8:56 ` Andrew Morton
@ 2006-01-19  9:36   ` Con Kolivas
  0 siblings, 0 replies; 29+ messages in thread
From: Con Kolivas @ 2006-01-19  9:36 UTC (permalink / raw)
  To: Andrew Morton; +Cc: Andy Chittenden, linux-kernel

On Thursday 19 January 2006 19:56, Andrew Morton wrote:
> "Andy Chittenden" <AChittenden@bluearc.com> wrote:
> > Why does running the following command cause processes to be killed:
> >
> > 	dd if=/dev/zero of=/u/u1/andyc/tmpfile bs=1M count=8k
> >
> > And I noticed one of my windows disappeared. Further investigation
> > showed that was my terminator window (java based app: see
> > http://software.jessies.org/terminator/). I found this in my syslog:
> >
> > Jan 17 11:12:58 boco kernel: Out of Memory: Killed process 16498 (java).
> >
> > My hardware: amd64 based machine (ASUS A8V Deluxe motherboard) with 4Gb
> > of memory.
> > My kernel: debian package linux-image-2.6.15-1-amd64-k8 package
> > installed. IE its running 2.6.15 compiled for amd64.
> >
> > This is repeatable. The above dd command also causes the machine to
> > become very unresponsive (eg windows don't focus).
>
> What type of filesytem is being written to?
>
> Has someone tuned /proc/sys/vm/swappiness, or something else under
> /proc/sys/vm?
>
> It'd be useful to see the dmesg output from that oom event.

Are you using scsi? Someone just posted what looks to be a scsi slab leak (Re: 
scsi cmd slab leak? (Was Re: [ck] Anyone been having OOM killer problems 
lately?) that causes oom kills. Check your slabinfo for a large 
scsi_cmd_cache.

Cheers,
Con

^ permalink raw reply	[flat|nested] 29+ messages in thread

* Re: Out of Memory: Killed process 16498 (java).
  2006-01-19  8:43 Andy Chittenden
@ 2006-01-19  8:56 ` Andrew Morton
  2006-01-19  9:36   ` Con Kolivas
  2006-01-19 20:54 ` Martin Bligh
  1 sibling, 1 reply; 29+ messages in thread
From: Andrew Morton @ 2006-01-19  8:56 UTC (permalink / raw)
  To: Andy Chittenden; +Cc: linux-kernel

"Andy Chittenden" <AChittenden@bluearc.com> wrote:
>
> Why does running the following command cause processes to be killed:
> 
> 	dd if=/dev/zero of=/u/u1/andyc/tmpfile bs=1M count=8k
> 
> And I noticed one of my windows disappeared. Further investigation
> showed that was my terminator window (java based app: see
> http://software.jessies.org/terminator/). I found this in my syslog:
> 
> Jan 17 11:12:58 boco kernel: Out of Memory: Killed process 16498 (java).
> 
> My hardware: amd64 based machine (ASUS A8V Deluxe motherboard) with 4Gb
> of memory.
> My kernel: debian package linux-image-2.6.15-1-amd64-k8 package
> installed. IE its running 2.6.15 compiled for amd64.
> 
> This is repeatable. The above dd command also causes the machine to
> become very unresponsive (eg windows don't focus).

What type of filesytem is being written to?

Has someone tuned /proc/sys/vm/swappiness, or something else under
/proc/sys/vm?

It'd be useful to see the dmesg output from that oom event.

^ permalink raw reply	[flat|nested] 29+ messages in thread

* Out of Memory: Killed process 16498 (java).
@ 2006-01-19  8:43 Andy Chittenden
  2006-01-19  8:56 ` Andrew Morton
  2006-01-19 20:54 ` Martin Bligh
  0 siblings, 2 replies; 29+ messages in thread
From: Andy Chittenden @ 2006-01-19  8:43 UTC (permalink / raw)
  To: linux-kernel

Why does running the following command cause processes to be killed:

	dd if=/dev/zero of=/u/u1/andyc/tmpfile bs=1M count=8k

And I noticed one of my windows disappeared. Further investigation
showed that was my terminator window (java based app: see
http://software.jessies.org/terminator/). I found this in my syslog:

Jan 17 11:12:58 boco kernel: Out of Memory: Killed process 16498 (java).

My hardware: amd64 based machine (ASUS A8V Deluxe motherboard) with 4Gb
of memory.
My kernel: debian package linux-image-2.6.15-1-amd64-k8 package
installed. IE its running 2.6.15 compiled for amd64.

This is repeatable. The above dd command also causes the machine to
become very unresponsive (eg windows don't focus).
 
-- 
Andy, BlueArc Engineering

^ permalink raw reply	[flat|nested] 29+ messages in thread

end of thread, other threads:[~2006-01-27 14:40 UTC | newest]

Thread overview: 29+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2006-01-23 10:03 Out of Memory: Killed process 16498 (java) Andy Chittenden
2006-01-23 10:11 ` Jens Axboe
  -- strict thread matches above, loose matches on Subject: below --
2006-01-27 11:53 Andy Chittenden
2006-01-27 14:21 ` Jens Axboe
2006-01-27 14:39   ` Anton Altaparmakov
2006-01-21 21:47 Andy Chittenden
2006-01-23  9:27 ` Jens Axboe
     [not found] <89E85E0168AD994693B574C80EDB9C27035561DE@uk-email.terastack.bluearc.com>
2006-01-20 17:39 ` Jens Axboe
     [not found] <89E85E0168AD994693B574C80EDB9C27035560E4@uk-email.terastack.bluearc.com>
2006-01-20 12:37 ` Larry Woodman
2006-01-19 15:11 Andy Chittenden
2006-01-19 19:48 ` Dave Jones
2006-01-19 22:15   ` Andrew Morton
2006-01-20  8:12     ` Jens Axboe
2006-01-20  8:23       ` Andrew Morton
2006-01-20 12:08         ` Jens Axboe
2006-01-20 12:17           ` Andrew Morton
2006-01-20 12:28             ` Jens Axboe
2006-01-20 12:51               ` Andrew Morton
2006-01-20  1:03   ` Andrew Morton
2006-01-20  1:28     ` Dave Jones
2006-01-20  1:49       ` Andrew Morton
2006-01-21 20:00         ` Dan Aloni
     [not found] <89E85E0168AD994693B574C80EDB9C2703555F8E@uk-email.terastack.bluearc.com>
2006-01-19  9:44 ` Andrew Morton
2006-01-19  9:40 Andy Chittenden
2006-01-19  9:41 ` Con Kolivas
2006-01-19  8:43 Andy Chittenden
2006-01-19  8:56 ` Andrew Morton
2006-01-19  9:36   ` Con Kolivas
2006-01-19 20:54 ` Martin Bligh

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).