linux-kernel.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
* 2.4.23pre6aa3 scsi oops
@ 2003-11-04  2:49 Jamie Clark
  2003-11-04 10:28 ` Andrea Arcangeli
  0 siblings, 1 reply; 15+ messages in thread
From: Jamie Clark @ 2003-11-04  2:49 UTC (permalink / raw)
  To: linux-kernel; +Cc: Andrea Arcangeli

[-- Attachment #1: Type: text/plain, Size: 823 bytes --]

Hi,

Consistent oops with 2.4.23pre6aa3 after 3-4 hours running bonnie on 
ext3 fs through qla2300 HBA. (w SMP, HIGHMEM) The test machine was 
completely wedged so I ended up transcribing the oops from the vga 
console. No typos I think.

As an aside: we're running 2.4.19pre10aa4 on a few high-load fileservers 
with 2.5G RAM and have been running into the same problem mentioned a 
month or so ago (Subject: "Strange dcache memory pressure when highmem 
enabled"). I had been suspicious of the dcache for quote a while so that 
thread rang true. I guessed that a later 2.4aa kernel would be a good 
approach. Anyone had success with a more recent one?   2.4.19pre10aa4 
has been rock-solid, and if it weren't for the dcache performance hit 
(and the ever increasing server load) I'd rather stick with it.

-Jamie Clark



[-- Attachment #2: ksymoops.out --]
[-- Type: text/plain, Size: 3110 bytes --]

ksymoops 2.4.5 on i686 2.4.23-pre6aa3.  Options used
     -V (default)
     -k /proc/ksyms (default)
     -l /proc/modules (default)
     -o /lib/modules/2.4.23-pre6aa3/ (default)
     -m /boot/System.map (specified)

Unable to handle kernel paging request at virtual address 0f0f0f47
c01abda5
*pde = 00000000
Oops: 0000 2.4.23-pre6aa3 #2 SMP Mon Nov 3 12:03:35 SGT 2003
CPU:    0
EIP:    0010:[<c01abda5>]    Not tainted
Using defaults from ksymoops -t elf32-i386 -a i386
EFLAGS: 00010096
eax: 00000f0f   ebx: c1000020   ecx: d93fa241   edx: afafbe1e
esi: 0f0f0f0f   edi: c03a7590   ebp: 00000014   esp: c02e1e28
ds: 0018   es: 00018   ss: 0018
Process swapper (pid: 0, stackpage=c02e1000)
Stack: 0f0f0f0f eb171800 c01d476b d93fa241 0f0f0f0f eb171800 f797ca18 eb171920
       f7ad8000 c03a7400 eb1718c8 c01d28ab eb171800 f794b618 00000246 00000000
       00000008 f797c9c0 eb1718c8 c02b2580 f794b600 00000000 c01d1ef8 f794b618
Call Trace:         [<c01d476b>] (36) [<c01d28ab>] (44) [<c01d1ef8>] (24)
  [<c01d21b0>] (36) [<c01d247f>] (40) [<c01cb681>] (28) [<c01ee58a>] (44)
  [<c01cbab8>] (20) [<c01cb8f7>] (32) [<c011e41d>] (40) [<c0108b2b>] (04)
  [<c01052c0>] (08) [<c01052c0>] (20) [<c01052c0>] (16) [<c01052c0>] (24)
  [<c01052ec>] (12) [<c0105342>] (12) [<c0105000>] (04) [<c0105048>] (24)
Code: 8b 46 38 8b 4e 34 29 d8 69 c0 ab aa aa aa c1 f8 04 c1 e0 0c


>>EIP; c01abda5 <blk_seg_merge_ok+31/64>   <=====

>>eax; 00000f0f Before first symbol
>>ebx; c1000020 <_end+c615ac/2cc158c>
>>ecx; d93fa241 <END_OF_CODE+1598972e/????>
>>edx; afafbe1e Before first symbol
>>esi; 0f0f0f0f Before first symbol
>>edi; c03a7590 <_end+8b1c/2cc158c>
>>esp; c02e1e28 <init_task_union+1e28/2000>

Trace; c01d476b <scsi_init_io_vc+cb/1b4>
Trace; c01d28ab <scsi_request_fn+1e7/3a8>
Trace; c01d1ef8 <scsi_queue_next_request+6c/168>
Trace; c01d21b0 <__scsi_end_request+1bc/1c8>
Trace; c01d247f <scsi_io_completion+1e7/3ec>
Trace; c01cb681 <scsi_done+d/a8>
Trace; c01ee58a <rw_intr+226/234>
Trace; c01cbab8 <scsi_finish_command+ac/b4>
Trace; c01cb8f7 <scsi_softirq+1db/26c>
Trace; c011e41d <do_softirq+dd/174>
Trace; c0108b2b <do_IRQ+db/ec>
Trace; c01052c0 <default_idle+0/34>
Trace; c01052c0 <default_idle+0/34>
Trace; c01052c0 <default_idle+0/34>
Trace; c01052c0 <default_idle+0/34>
Trace; c01052ec <default_idle+2c/34>
Trace; c0105342 <cpu_idle+2e/44>
Trace; c0105000 <_stext+0/0>
Trace; c0105048 <rest_init+48/4c>

Code;  c01abda5 <blk_seg_merge_ok+31/64>
00000000 <_EIP>:
Code;  c01abda5 <blk_seg_merge_ok+31/64>   <=====
   0:   8b 46 38                  mov    0x38(%esi),%eax   <=====
Code;  c01abda8 <blk_seg_merge_ok+34/64>
   3:   8b 4e 34                  mov    0x34(%esi),%ecx
Code;  c01abdab <blk_seg_merge_ok+37/64>
   6:   29 d8                     sub    %ebx,%eax
Code;  c01abdad <blk_seg_merge_ok+39/64>
   8:   69 c0 ab aa aa aa         imul   $0xaaaaaaab,%eax,%eax
Code;  c01abdb3 <blk_seg_merge_ok+3f/64>
   e:   c1 f8 04                  sar    $0x4,%eax
Code;  c01abdb6 <blk_seg_merge_ok+42/64>
  11:   c1 e0 0c                  shl    $0xc,%eax

 <0>Kernel panic: Aieee, killing interrupt handler!

[-- Attachment #3: dmesg.out --]
[-- Type: text/plain, Size: 14544 bytes --]

Linux version 2.4.23-pre6aa3 (root@vmclu2) (gcc version 2.95.4 20011002 (Debian prerelease)) #2 SMP Mon Nov 3 12:03:35 SGT 2003
BIOS-provided physical RAM map:
 BIOS-e820: 0000000000000000 - 000000000009fc00 (usable)
 BIOS-e820: 000000000009fc00 - 00000000000a0000 (reserved)
 BIOS-e820: 00000000000e0000 - 0000000000100000 (reserved)
 BIOS-e820: 0000000000100000 - 000000003fff0000 (usable)
 BIOS-e820: 000000003fff0000 - 000000003ffff000 (ACPI data)
 BIOS-e820: 000000003ffff000 - 0000000040000000 (ACPI NVS)
 BIOS-e820: 00000000fec00000 - 00000000fec02000 (reserved)
 BIOS-e820: 00000000fee00000 - 00000000fee01000 (reserved)
 BIOS-e820: 00000000fff80000 - 0000000100000000 (reserved)
127MB HIGHMEM available.
896MB LOWMEM available.
found SMP MP-table at 000ff780
hm, page 000ff000 reserved twice.
hm, page 00100000 reserved twice.
hm, page 000f0000 reserved twice.
hm, page 000f1000 reserved twice.
On node 0 totalpages: 262128
zone(0): 4096 pages.
zone(1): 225280 pages.
zone(2): 32752 pages.
ACPI disabled because your bios is from 2000 and too old
You can enable it with acpi=force
ACPI: RSDP (v000 AMI                                       ) @ 0x000ff8f0
ACPI: RSDT (v001 RCC    RCCNILE  0x00000001 MSFT 0x01000000) @ 0x3fff0000
ACPI: FADT (v001 RCC    RCCNILE  0x00000001 MSFT 0x01000000) @ 0x3fff0030
ACPI: MADT (v001 RCC    RCCNILE  0x00000001 MSFT 0x01000000) @ 0x3fff00b0
ACPI: DSDT (v001    RCC  CNB20HE 0x00000100 MSFT 0x0100000b) @ 0x00000000
ACPI: Local APIC address 0xfee00000
ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled)
Processor #0 Pentium(tm) Pro APIC version 17
ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled)
Processor #1 Pentium(tm) Pro APIC version 17
ACPI: LAPIC (acpi_id[0x02] lapic_id[0x00] disabled)
ACPI: LAPIC (acpi_id[0x03] lapic_id[0x00] disabled)
Using ACPI for processor (LAPIC) configuration information
Intel MultiProcessor Specification v1.4
    Virtual Wire compatibility mode.
OEM ID: AMI      Product ID: CNB20HE      APIC at: 0xFEE00000
I/O APIC #4 Version 17 at 0xFEC00000.
I/O APIC #5 Version 17 at 0xFEC01000.
Enabling APIC mode: Flat.	Using 2 I/O APICs
Processors: 2
Building zonelist for node : 0
Kernel command line: ro root=/dev/sda6
Initializing CPU#0
Detected 999.581 MHz processor.
Console: colour VGA+ 80x25
Calibrating delay loop... 1992.29 BogoMIPS
Memory: 1024736k/1048512k available (1323k kernel code, 15068k reserved, 596k data, 96k init, 131008k highmem)
Dentry cache hash table entries: 131072 (order: 8, 1048576 bytes)
Inode cache hash table entries: 65536 (order: 7, 524288 bytes)
Mount cache hash table entries: 512 (order: 0, 4096 bytes)
Buffer cache hash table entries: 65536 (order: 6, 262144 bytes)
Page-cache hash table entries: 262144 (order: 8, 1048576 bytes)
CPU: L1 I cache: 16K, L1 D cache: 16K
CPU: L2 cache: 256K
CPU:     After generic, caps: 0383fbff 00000000 00000000 00000000
CPU:             Common caps: 0383fbff 00000000 00000000 00000000
Enabling fast FPU save and restore... done.
Enabling unmasked SIMD FPU exception support... done.
Checking 'hlt' instruction... OK.
POSIX conformance testing by UNIFIX
mtrr: v1.40 (20010327) Richard Gooch (rgooch@atnf.csiro.au)
mtrr: detected mtrr type: Intel
CPU: L1 I cache: 16K, L1 D cache: 16K
CPU: L2 cache: 256K
CPU:     After generic, caps: 0383fbff 00000000 00000000 00000000
CPU:             Common caps: 0383fbff 00000000 00000000 00000000
CPU0: Intel Pentium III (Coppermine) stepping 06
per-CPU timeslice cutoff: 731.73 usecs.
task migration cache decay timeout: 10 msecs.
enabled ExtINT on CPU#0
ESR value before enabling vector: 00000004
ESR value after enabling vector: 00000000
Booting processor 1/1 eip 2000
Initializing CPU#1
masked ExtINT on CPU#1
ESR value before enabling vector: 00000000
ESR value after enabling vector: 00000000
Calibrating delay loop... 1998.84 BogoMIPS
CPU: L1 I cache: 16K, L1 D cache: 16K
CPU: L2 cache: 256K
CPU:     After generic, caps: 0383fbff 00000000 00000000 00000000
CPU:             Common caps: 0383fbff 00000000 00000000 00000000
CPU1: Intel Pentium III (Coppermine) stepping 06
Total of 2 processors activated (3991.14 BogoMIPS).
ENABLING IO-APIC IRQs
Setting 4 in the phys_id_present_map
...changing IO-APIC physical APIC ID to 4 ... ok.
Setting 5 in the phys_id_present_map
...changing IO-APIC physical APIC ID to 5 ... ok.
init IO_APIC IRQs
 IO-APIC (apicid-pin) 4-0, 4-5, 4-7, 4-9, 4-11, 5-0, 5-1, 5-2, 5-3, 5-4, 5-5, 5-7, 5-8, 5-9, 5-13 not connected.
..TIMER: vector=0x31 pin1=2 pin2=0
..MP-BIOS bug: 8254 timer not connected to IO-APIC
...trying to set up timer (IRQ0) through the 8259A ... 
..... (found pin 0) ...works.
number of MP IRQ sources: 18.
number of IO-APIC #4 registers: 16.
number of IO-APIC #5 registers: 16.
testing the IO APIC.......................

IO APIC #4......
.... register #00: 04000000
.......    : physical APIC id: 04
.......    : Delivery Type: 0
.......    : LTS          : 0
.... register #01: 000F0011
.......     : max redirection entries: 000F
.......     : PRQ implemented: 0
.......     : IO APIC version: 0011
.... register #02: 00000000
.......     : arbitration: 00
.... IRQ redirection table:
 NR Log Phy Mask Trig IRR Pol Stat Dest Deli Vect:   
 00 003 03  0    0    0   0   0    1    1    31
 01 003 03  0    0    0   0   0    1    1    39
 02 000 00  1    0    0   0   0    0    0    00
 03 003 03  0    0    0   0   0    1    1    41
 04 003 03  0    0    0   0   0    1    1    49
 05 000 00  1    0    0   0   0    0    0    00
 06 003 03  0    0    0   0   0    1    1    51
 07 000 00  1    0    0   0   0    0    0    00
 08 003 03  0    0    0   0   0    1    1    59
 09 000 00  1    0    0   0   0    0    0    00
 0a 003 03  1    1    0   1   0    1    1    61
 0b 000 00  1    0    0   0   0    0    0    00
 0c 003 03  0    0    0   0   0    1    1    69
 0d 003 03  0    0    0   0   0    1    1    71
 0e 003 03  0    0    0   0   0    1    1    79
 0f 003 03  0    0    0   0   0    1    1    81

IO APIC #5......
.... register #00: 05000000
.......    : physical APIC id: 05
.......    : Delivery Type: 0
.......    : LTS          : 0
.... register #01: 000F0011
.......     : max redirection entries: 000F
.......     : PRQ implemented: 0
.......     : IO APIC version: 0011
.... register #02: 01000000
.......     : arbitration: 01
.... IRQ redirection table:
 NR Log Phy Mask Trig IRR Pol Stat Dest Deli Vect:   
 00 000 00  1    0    0   0   0    0    0    00
 01 000 00  1    0    0   0   0    0    0    00
 02 000 00  1    0    0   0   0    0    0    00
 03 000 00  1    0    0   0   0    0    0    00
 04 000 00  1    0    0   0   0    0    0    00
 05 000 00  1    0    0   0   0    0    0    00
 06 003 03  1    1    0   1   0    1    1    89
 07 000 00  1    0    0   0   0    0    0    00
 08 000 00  1    0    0   0   0    0    0    00
 09 000 00  1    0    0   0   0    0    0    00
 0a 003 03  1    1    0   1   0    1    1    91
 0b 003 03  1    1    0   1   0    1    1    99
 0c 003 03  1    1    0   1   0    1    1    A1
 0d 000 00  1    0    0   0   0    0    0    00
 0e 003 03  1    1    0   1   0    1    1    A9
 0f 003 03  1    1    0   1   0    1    1    B1
IRQ to pin mappings:
IRQ0 -> 0:0
IRQ1 -> 0:1
IRQ3 -> 0:3
IRQ4 -> 0:4
IRQ6 -> 0:6
IRQ8 -> 0:8
IRQ10 -> 0:10
IRQ12 -> 0:12
IRQ13 -> 0:13
IRQ14 -> 0:14
IRQ15 -> 0:15
IRQ22 -> 1:6
IRQ26 -> 1:10
IRQ27 -> 1:11
IRQ28 -> 1:12
IRQ30 -> 1:14
IRQ31 -> 1:15
.................................... done.
Using local APIC timer interrupts.
calibrating APIC timer ...
..... CPU clock speed is 999.5537 MHz.
..... host bus clock speed is 133.2732 MHz.
cpu: 0, clocks: 1332732, slice: 444244
CPU0<T0:1332720,T1:888464,D:12,S:444244,C:1332732>
cpu: 1, clocks: 1332732, slice: 444244
CPU1<T0:1332720,T1:444224,D:8,S:444244,C:1332732>
checking TSC synchronization across CPUs: passed.
migration_task 0 on cpu=0
migration_task 1 on cpu=1
PCI: PCI BIOS revision 2.10 entry at 0xfdbb1, last bus=2
PCI: Using configuration type 1
PCI: Probing PCI hardware
PCI: Probing PCI hardware (bus 00)
PCI: Discovered primary peer bus 02 [IRQ]
PCI: Using IRQ router ServerWorks [1166/0200] at 00:0f.0
PCI->APIC IRQ transform: (B0,I4,P0) -> 28
PCI->APIC IRQ transform: (B0,I5,P0) -> 26
PCI->APIC IRQ transform: (B0,I5,P1) -> 27
PCI->APIC IRQ transform: (B0,I6,P0) -> 31
PCI->APIC IRQ transform: (B0,I15,P0) -> 10
PCI->APIC IRQ transform: (B1,I0,P0) -> 30
PCI->APIC IRQ transform: (B2,I1,P0) -> 22
PCI: Cannot allocate resource region 1 of device 00:00.0
Linux NET4.0 for Linux 2.4
Based upon Swansea University Computer Society NET3.039
Initializing RT netlink socket
Starting kswapd
bigpage subsystem: allocated 0 bigpages (=0MB).
allocated 32 pages and 32 bhs reserved for the highmem bounces
VFS: Disk quotas vdquot_6.5.1
aio_setup: num_physpages = 65532
aio_setup: sizeof(struct page) = 48
Journalled Block Device driver loaded
pty: 256 Unix98 ptys configured
Serial driver version 5.05c (2001-07-08) with MANY_PORTS SHARE_IRQ SERIAL_PCI enabled
ttyS00 at 0x03f8 (irq = 4) is a 16550A
ttyS01 at 0x02f8 (irq = 3) is a 16550A
Real Time Clock Driver v1.10e
Floppy drive(s): fd0 is 1.44M
FDC 0 is a National Semiconductor PC87306
Uniform Multi-Platform E-IDE driver Revision: 7.00beta4-2.4
ide: Assuming 33MHz system bus speed for PIO modes; override with idebus=xx
SvrWks OSB4: IDE controller at PCI slot 00:0f.1
SvrWks OSB4: chipset revision 0
SvrWks OSB4: not 100% native mode: will probe irqs later
    ide0: BM-DMA at 0xffa0-0xffa7, BIOS settings: hda:DMA, hdb:pio
    ide1: BM-DMA at 0xffa8-0xffaf, BIOS settings: hdc:pio, hdd:pio
hda: MATSHITA CR-177, ATAPI CD/DVD-ROM drive
ide0 at 0x1f0-0x1f7,0x3f6 on irq 14
hda: attached ide-cdrom driver.
hda: ATAPI 24X CD-ROM drive, 128kB Cache, (U)DMA
Uniform CD-ROM driver Revision: 3.12
SCSI subsystem driver Revision: 1.00
scsi0 : Adaptec AIC7XXX EISA/VLB/PCI SCSI HBA DRIVER, Rev 6.2.36
        <Adaptec aic7899 Ultra160 SCSI adapter>
        aic7899: Ultra160 Wide Channel A, SCSI Id=7, 32/253 SCBs

scsi1 : Adaptec AIC7XXX EISA/VLB/PCI SCSI HBA DRIVER, Rev 6.2.36
        <Adaptec aic7899 Ultra160 SCSI adapter>
        aic7899: Ultra160 Wide Channel B, SCSI Id=7, 32/253 SCBs

blk: queue f7939418, I/O limit 4095Mb (mask 0xffffffff)
(scsi0:A:0): 160.000MB/s transfers (80.000MHz DT, offset 63, 16bit)
  Vendor: SEAGATE   Model: ST318404LC        Rev: 0006
  Type:   Direct-Access                      ANSI SCSI revision: 03
blk: queue f7939218, I/O limit 4095Mb (mask 0xffffffff)
scsi0:A:0:0: Tagged Queuing enabled.  Depth 253
Attached scsi disk sda at scsi0, channel 0, id 0, lun 0
SCSI device sda: 35843670 512-byte hdwr sectors (18352 MB)
Partition check:
 sda: sda1 sda2 < sda5 sda6 sda7 sda8 >
NET4: Linux TCP/IP 1.0 for NET4.0
IP Protocols: ICMP, UDP, TCP, IGMP
IP: routing cache hash table of 8192 buckets, 64Kbytes
TCP: Hash tables configured (established 262144 bind 65536)
NET4: Unix domain sockets 1.0/SMP for Linux NET4.0.
EXT3-fs: INFO: recovery required on readonly filesystem.
EXT3-fs: write access will be enabled during recovery.
kjournald starting.  Commit interval 5 seconds
EXT3-fs: recovery complete.
EXT3-fs: mounted filesystem with ordered data mode.
VFS: Mounted root (ext3 filesystem) readonly.
Freeing unused kernel memory: 96k freed
Adding Swap: 1052216k swap-space (priority -1)
Adding Swap: 1052216k swap-space (priority -2)
EXT3 FS 2.4-0.9.19, 19 August 2002 on sd(8,6), internal journal
eepro100.c:v1.09j-t 9/29/99 Donald Becker http://www.scyld.com/network/eepro100.html
eepro100.c: $Revision: 1.36 $ 2000/11/17 Modified by Andrey V. Savochkin <saw@saw.sw.com.sg> and others
eth0: OEM i82557/i82558 10/100 Ethernet, 00:30:48:11:4A:32, IRQ 31.
  Board assembly 000000-000, Physical connectors present: RJ45
  Primary interface chip i82555 PHY #1.
  General self-test: passed.
  Serial sub-system self-test: passed.
  Internal registers self-test: passed.
  ROM checksum self-test: passed (0x04f4518b).
eth1: OEM i82557/i82558 10/100 Ethernet, 00:30:48:11:4A:33, IRQ 28.
  Board assembly 000000-000, Physical connectors present: RJ45
  Primary interface chip i82555 PHY #1.
  General self-test: passed.
  Serial sub-system self-test: passed.
  Internal registers self-test: passed.
  ROM checksum self-test: passed (0x04f4518b).
qla2x00_set_info starts at address = c3780060
qla2x00: Found  VID=1077 DID=2300 SSVID=1077 SSDID=9
scsi(2): Found a QLA2300  @ bus 2, device 0x1, irq 22, iobase 0xe800
scsi(2): Allocated 4096 SRB(s).
scsi(2): Configure NVRAM parameters...
scsi(2): 32 Bit PCI Addressing Enabled.
scsi(2): Verifying loaded RISC code...
scsi(2): Verifying chip...
scsi(2): Waiting for LIP to complete...
scsi(2): LIP reset occurred.
scsi(2): LIP occurred.
scsi(2): LOOP UP detected.
scsi(2): Topology - (Loop), Host Loop address 0x1
scsi2 : QLogic QLA2300 PCI to Fibre Channel Host Adapter: bus 2 device 1 irq 22
        Firmware version:  3.01.18, Driver version 6.05.60
blk: queue f794b818, I/O limit 4095Mb (mask 0xffffffff)
  Vendor: CNSi      Model: JFS224            Rev: L425
  Type:   Processor                          ANSI SCSI revision: 03
blk: queue f794b618, I/O limit 4095Mb (mask 0xffffffff)
  Vendor: CNSi      Model: JFS224            Rev: L425
  Type:   Direct-Access                      ANSI SCSI revision: 03
blk: queue f794b418, I/O limit 4095Mb (mask 0xffffffff)
scsi(2:0:0:0): Enabled tagged queuing, queue depth 16.
scsi(2:0:0:1): Enabled tagged queuing, queue depth 16.
Attached scsi generic sg1 at scsi2, channel 0, id 0, lun 0,  type 3
Attached scsi disk sdb at scsi2, channel 0, id 0, lun 1
SCSI device sdb: 70970368 512-byte hdwr sectors (36337 MB)
 sdb: sdb1 sdb2
LVM version 1.0.7(28/03/2003) module loaded
usb.c: registered new driver usbdevfs
usb.c: registered new driver hub
usb-ohci.c: USB OHCI at membase 0xf890e000, IRQ 10
usb-ohci.c: usb-00:0f.2, ServerWorks OSB4/CSB5 OHCI USB Controller
usb.c: new USB bus registered, assigned bus number 1
hub.c: USB hub found
hub.c: 4 ports detected
usb.c: registered new driver serial
usbserial.c: USB Serial Driver core v1.4
usbserial.c: USB Serial support registered for PL-2303
pl2303.c: Prolific PL2303 USB to serial adaptor driver v0.10
kjournald starting.  Commit interval 5 seconds
EXT3 FS 2.4-0.9.19, 19 August 2002 on sd(8,8), internal journal
EXT3-fs: mounted filesystem with ordered data mode.
kjournald starting.  Commit interval 5 seconds
EXT3 FS 2.4-0.9.19, 19 August 2002 on sd(8,1), internal journal
EXT3-fs: mounted filesystem with ordered data mode.
ip_tables: (C) 2000-2002 Netfilter core team
Installing knfsd (copyright (C) 1996 okir@monad.swb.de).

[-- Attachment #4: config --]
[-- Type: text/plain, Size: 23345 bytes --]

#
# Automatically generated by make menuconfig: don't edit
#
CONFIG_X86=y
# CONFIG_SBUS is not set
CONFIG_UID16=y

#
# Code maturity level options
#
CONFIG_EXPERIMENTAL=y

#
# Loadable module support
#
CONFIG_MODULES=y
CONFIG_MODVERSIONS=y
CONFIG_KMOD=y

#
# Processor type and features
#
# CONFIG_M386 is not set
# CONFIG_M486 is not set
# CONFIG_M586 is not set
# CONFIG_M586TSC is not set
# CONFIG_M586MMX is not set
# CONFIG_M686 is not set
CONFIG_MPENTIUMIII=y
# CONFIG_MPENTIUM4 is not set
# CONFIG_MK6 is not set
# CONFIG_MK7 is not set
# CONFIG_MK8 is not set
# CONFIG_MELAN is not set
# CONFIG_MCRUSOE is not set
# CONFIG_MWINCHIPC6 is not set
# CONFIG_MWINCHIP2 is not set
# CONFIG_MWINCHIP3D is not set
# CONFIG_MCYRIXIII is not set
# CONFIG_MVIAC3_2 is not set
CONFIG_X86_WP_WORKS_OK=y
CONFIG_X86_INVLPG=y
CONFIG_X86_CMPXCHG=y
CONFIG_X86_XADD=y
CONFIG_X86_BSWAP=y
CONFIG_X86_POPAD_OK=y
CONFIG_X86_CMPXCHG8=y
CONFIG_X86_L1_CACHE_SHIFT=5
CONFIG_X86_HAS_TSC=y
CONFIG_X86_GOOD_APIC=y
CONFIG_X86_PGE=y
CONFIG_X86_USE_PPRO_CHECKSUM=y
CONFIG_X86_F00F_WORKS_OK=y
# CONFIG_X86_MCE is not set
# CONFIG_TOSHIBA is not set
# CONFIG_I8K is not set
# CONFIG_THINKPAD is not set
# CONFIG_MICROCODE is not set
# CONFIG_X86_MSR is not set
# CONFIG_X86_CPUID is not set
# CONFIG_NOHIGHMEM is not set
CONFIG_HIGHMEM4G=y
# CONFIG_HIGHMEM64G is not set
CONFIG_HIGHMEM=y
CONFIG_FORCE_MAX_ZONEORDER=11
CONFIG_1GB=y
# CONFIG_2GB is not set
# CONFIG_3GB is not set
# CONFIG_05GB is not set
CONFIG_HIGHIO=y
# CONFIG_MATH_EMULATION is not set
CONFIG_MTRR=y
CONFIG_SMP=y
CONFIG_NR_CPUS=32
# CONFIG_X86_NUMA is not set
# CONFIG_X86_TSC_DISABLE is not set
CONFIG_X86_TSC=y
CONFIG_HAVE_DEC_LOCK=y

#
# General setup
#
CONFIG_NET=y
CONFIG_X86_IO_APIC=y
CONFIG_X86_LOCAL_APIC=y
CONFIG_PCI=y
# CONFIG_PCI_GOBIOS is not set
# CONFIG_PCI_GODIRECT is not set
CONFIG_PCI_GOANY=y
CONFIG_PCI_BIOS=y
CONFIG_PCI_DIRECT=y
CONFIG_ISA=y
CONFIG_PCI_NAMES=y
# CONFIG_EISA is not set
# CONFIG_MCA is not set
CONFIG_HOTPLUG=y

#
# PCMCIA/CardBus support
#
# CONFIG_PCMCIA is not set

#
# PCI Hotplug Support
#
# CONFIG_HOTPLUG_PCI is not set
# CONFIG_HOTPLUG_PCI_COMPAQ is not set
# CONFIG_HOTPLUG_PCI_COMPAQ_NVRAM is not set
# CONFIG_HOTPLUG_PCI_IBM is not set
CONFIG_SYSVIPC=y
CONFIG_BSD_PROCESS_ACCT=y
CONFIG_SYSCTL=y
CONFIG_MAX_USER_RT_PRIO=100
CONFIG_MAX_RT_PRIO=0
CONFIG_KCORE_ELF=y
# CONFIG_KCORE_AOUT is not set
# CONFIG_BINFMT_AOUT is not set
CONFIG_BINFMT_ELF=y
CONFIG_BINFMT_MISC=y
# CONFIG_PM is not set
# CONFIG_APM is not set

#
# ACPI Support
#
# CONFIG_ACPI is not set
CONFIG_ACPI_BOOT=y

#
# Memory Technology Devices (MTD)
#
# CONFIG_MTD is not set

#
# Parallel port support
#
# CONFIG_PARPORT is not set

#
# Plug and Play configuration
#
# CONFIG_PNP is not set
# CONFIG_ISAPNP is not set

#
# Block devices
#
CONFIG_BLK_DEV_FD=y
# CONFIG_BLK_DEV_XD is not set
# CONFIG_PARIDE is not set
# CONFIG_BLK_CPQ_DA is not set
# CONFIG_BLK_CPQ_CISS_DA is not set
# CONFIG_CISS_SCSI_TAPE is not set
# CONFIG_CISS_MONITOR_THREAD is not set
# CONFIG_BLK_DEV_DAC960 is not set
# CONFIG_BLK_DEV_UMEM is not set
CONFIG_BLK_DEV_LOOP=m
CONFIG_BLK_DEV_NBD=m
CONFIG_BLK_DEV_RAM=m
CONFIG_BLK_DEV_RAM_SIZE=4096
# CONFIG_BLK_DEV_INITRD is not set
# CONFIG_BLK_STATS is not set

#
# Multi-device support (RAID and LVM)
#
CONFIG_MD=y
CONFIG_BLK_DEV_MD=m
CONFIG_MD_LINEAR=m
CONFIG_MD_RAID0=m
CONFIG_MD_RAID1=m
CONFIG_MD_RAID5=m
CONFIG_MD_MULTIPATH=m
CONFIG_BLK_DEV_LVM=m
# CONFIG_BLK_DEV_DM is not set

#
# Networking options
#
CONFIG_PACKET=y
CONFIG_PACKET_MMAP=y
CONFIG_NETLINK_DEV=m
CONFIG_NETFILTER=y
# CONFIG_NETFILTER_DEBUG is not set
# CONFIG_FILTER is not set
CONFIG_UNIX=y
CONFIG_INET=y
# CONFIG_TUX is not set
CONFIG_IP_MULTICAST=y
CONFIG_IP_ADVANCED_ROUTER=y
CONFIG_IP_MULTIPLE_TABLES=y
CONFIG_IP_ROUTE_FWMARK=y
CONFIG_IP_ROUTE_NAT=y
# CONFIG_IP_ROUTE_MULTIPATH is not set
CONFIG_IP_ROUTE_TOS=y
# CONFIG_IP_ROUTE_VERBOSE is not set
# CONFIG_IP_PNP is not set
# CONFIG_NET_IPIP is not set
# CONFIG_NET_IPGRE is not set
# CONFIG_IP_MROUTE is not set
# CONFIG_ARPD is not set
# CONFIG_INET_ECN is not set
# CONFIG_SYN_COOKIES is not set

#
#   IP: Netfilter Configuration
#
CONFIG_IP_NF_CONNTRACK=m
# CONFIG_IP_NF_FTP is not set
# CONFIG_IP_NF_AMANDA is not set
# CONFIG_IP_NF_TFTP is not set
# CONFIG_IP_NF_IRC is not set
# CONFIG_IP_NF_QUEUE is not set
CONFIG_IP_NF_IPTABLES=m
CONFIG_IP_NF_MATCH_LIMIT=m
CONFIG_IP_NF_MATCH_MAC=m
# CONFIG_IP_NF_MATCH_PKTTYPE is not set
CONFIG_IP_NF_MATCH_MARK=m
CONFIG_IP_NF_MATCH_MULTIPORT=m
CONFIG_IP_NF_MATCH_TOS=m
# CONFIG_IP_NF_MATCH_RECENT is not set
# CONFIG_IP_NF_MATCH_ECN is not set
# CONFIG_IP_NF_MATCH_DSCP is not set
# CONFIG_IP_NF_MATCH_AH_ESP is not set
CONFIG_IP_NF_MATCH_LENGTH=m
CONFIG_IP_NF_MATCH_TTL=m
CONFIG_IP_NF_MATCH_TCPMSS=m
# CONFIG_IP_NF_MATCH_HELPER is not set
CONFIG_IP_NF_MATCH_STATE=m
# CONFIG_IP_NF_MATCH_CONNTRACK is not set
# CONFIG_IP_NF_MATCH_UNCLEAN is not set
# CONFIG_IP_NF_MATCH_OWNER is not set
CONFIG_IP_NF_FILTER=m
CONFIG_IP_NF_TARGET_REJECT=m
CONFIG_IP_NF_TARGET_MIRROR=m
CONFIG_IP_NF_NAT=m
CONFIG_IP_NF_NAT_NEEDED=y
CONFIG_IP_NF_TARGET_MASQUERADE=m
CONFIG_IP_NF_TARGET_REDIRECT=m
CONFIG_IP_NF_NAT_LOCAL=y
# CONFIG_IP_NF_NAT_SNMP_BASIC is not set
CONFIG_IP_NF_MANGLE=m
CONFIG_IP_NF_TARGET_TOS=m
# CONFIG_IP_NF_TARGET_ECN is not set
# CONFIG_IP_NF_TARGET_DSCP is not set
CONFIG_IP_NF_TARGET_MARK=m
CONFIG_IP_NF_TARGET_LOG=m
# CONFIG_IP_NF_TARGET_ULOG is not set
CONFIG_IP_NF_TARGET_TCPMSS=m
# CONFIG_IP_NF_ARPTABLES is not set
# CONFIG_IP_NF_COMPAT_IPCHAINS is not set
# CONFIG_IP_NF_COMPAT_IPFWADM is not set

#
#   IP: Virtual Server Configuration
#
# CONFIG_IP_VS is not set
# CONFIG_IPV6 is not set
# CONFIG_KHTTPD is not set

#
#    SCTP Configuration (EXPERIMENTAL)
#
CONFIG_IPV6_SCTP__=y
# CONFIG_IP_SCTP is not set
# CONFIG_ATM is not set
# CONFIG_VLAN_8021Q is not set
# CONFIG_IPX is not set
CONFIG_ATALK=m

#
# Appletalk devices
#
# CONFIG_DEV_APPLETALK is not set
# CONFIG_DECNET is not set
# CONFIG_BRIDGE is not set
# CONFIG_X25 is not set
# CONFIG_LAPB is not set
# CONFIG_LLC is not set
# CONFIG_NET_DIVERT is not set
# CONFIG_ECONET is not set
# CONFIG_WAN_ROUTER is not set
# CONFIG_NET_FASTROUTE is not set
# CONFIG_NET_HW_FLOWCONTROL is not set

#
# QoS and/or fair queueing
#
# CONFIG_NET_SCHED is not set

#
# Network testing
#
# CONFIG_NET_PKTGEN is not set

#
# Telephony Support
#
# CONFIG_PHONE is not set
# CONFIG_PHONE_IXJ is not set
# CONFIG_PHONE_IXJ_PCMCIA is not set

#
# ATA/IDE/MFM/RLL support
#
CONFIG_IDE=y

#
# IDE, ATA and ATAPI Block devices
#
CONFIG_BLK_DEV_IDE=y
# CONFIG_BLK_DEV_HD_IDE is not set
# CONFIG_BLK_DEV_HD is not set
CONFIG_BLK_DEV_IDEDISK=y
CONFIG_IDEDISK_MULTI_MODE=y
# CONFIG_IDEDISK_STROKE is not set
# CONFIG_BLK_DEV_IDECS is not set
CONFIG_BLK_DEV_IDECD=y
# CONFIG_BLK_DEV_IDETAPE is not set
# CONFIG_BLK_DEV_IDEFLOPPY is not set
# CONFIG_BLK_DEV_IDESCSI is not set
# CONFIG_IDE_TASK_IOCTL is not set
# CONFIG_BLK_DEV_CMD640 is not set
# CONFIG_BLK_DEV_CMD640_ENHANCED is not set
# CONFIG_BLK_DEV_ISAPNP is not set
CONFIG_BLK_DEV_IDEPCI=y
# CONFIG_BLK_DEV_GENERIC is not set
CONFIG_IDEPCI_SHARE_IRQ=y
CONFIG_BLK_DEV_IDEDMA_PCI=y
# CONFIG_BLK_DEV_OFFBOARD is not set
# CONFIG_BLK_DEV_IDEDMA_FORCED is not set
CONFIG_IDEDMA_PCI_AUTO=y
# CONFIG_IDEDMA_ONLYDISK is not set
CONFIG_BLK_DEV_IDEDMA=y
# CONFIG_IDEDMA_PCI_WIP is not set
# CONFIG_BLK_DEV_ADMA100 is not set
# CONFIG_BLK_DEV_AEC62XX is not set
# CONFIG_BLK_DEV_ALI15X3 is not set
# CONFIG_WDC_ALI15X3 is not set
# CONFIG_BLK_DEV_AMD74XX is not set
# CONFIG_AMD74XX_OVERRIDE is not set
# CONFIG_BLK_DEV_CMD64X is not set
# CONFIG_BLK_DEV_TRIFLEX is not set
# CONFIG_BLK_DEV_CY82C693 is not set
# CONFIG_BLK_DEV_CS5530 is not set
# CONFIG_BLK_DEV_HPT34X is not set
# CONFIG_HPT34X_AUTODMA is not set
# CONFIG_BLK_DEV_HPT366 is not set
CONFIG_BLK_DEV_PIIX=y
# CONFIG_BLK_DEV_NS87415 is not set
# CONFIG_BLK_DEV_OPTI621 is not set
# CONFIG_BLK_DEV_PDC202XX_OLD is not set
# CONFIG_PDC202XX_BURST is not set
# CONFIG_BLK_DEV_PDC202XX_NEW is not set
# CONFIG_BLK_DEV_RZ1000 is not set
# CONFIG_BLK_DEV_SC1200 is not set
CONFIG_BLK_DEV_SVWKS=y
# CONFIG_BLK_DEV_SIIMAGE is not set
# CONFIG_BLK_DEV_SIS5513 is not set
# CONFIG_BLK_DEV_SLC90E66 is not set
# CONFIG_BLK_DEV_TRM290 is not set
# CONFIG_BLK_DEV_VIA82CXXX is not set
# CONFIG_IDE_CHIPSETS is not set
CONFIG_IDEDMA_AUTO=y
# CONFIG_IDEDMA_IVB is not set
# CONFIG_DMA_NONPCI is not set
CONFIG_BLK_DEV_IDE_MODES=y
# CONFIG_BLK_DEV_ATARAID is not set
# CONFIG_BLK_DEV_ATARAID_PDC is not set
# CONFIG_BLK_DEV_ATARAID_HPT is not set
# CONFIG_BLK_DEV_ATARAID_SII is not set

#
# SCSI support
#
CONFIG_SCSI=y
CONFIG_BLK_DEV_SD=y
CONFIG_SD_EXTRA_DEVS=40
# CONFIG_CHR_DEV_ST is not set
# CONFIG_CHR_DEV_OSST is not set
# CONFIG_BLK_DEV_SR is not set
CONFIG_CHR_DEV_SG=y
CONFIG_SCSI_DEBUG_QUEUES=y
CONFIG_SCSI_MULTI_LUN=y
CONFIG_SCSI_CONSTANTS=y
CONFIG_SCSI_LOGGING=y

#
# SCSI low-level drivers
#
# CONFIG_BLK_DEV_3W_XXXX_RAID is not set
# CONFIG_SCSI_7000FASST is not set
# CONFIG_SCSI_ACARD is not set
# CONFIG_SCSI_AHA152X is not set
# CONFIG_SCSI_AHA1542 is not set
# CONFIG_SCSI_AHA1740 is not set
# CONFIG_SCSI_AACRAID is not set
CONFIG_SCSI_AIC7XXX=y
CONFIG_AIC7XXX_CMDS_PER_DEVICE=253
CONFIG_AIC7XXX_RESET_DELAY_MS=15000
# CONFIG_AIC7XXX_PROBE_EISA_VL is not set
# CONFIG_AIC7XXX_BUILD_FIRMWARE is not set
# CONFIG_AIC7XXX_DEBUG_ENABLE is not set
CONFIG_AIC7XXX_DEBUG_MASK=0
# CONFIG_AIC7XXX_REG_PRETTY_PRINT is not set
# CONFIG_SCSI_AIC79XX is not set
# CONFIG_SCSI_DPT_I2O is not set
# CONFIG_SCSI_ADVANSYS is not set
# CONFIG_SCSI_IN2000 is not set
# CONFIG_SCSI_AM53C974 is not set
# CONFIG_SCSI_MEGARAID is not set
# CONFIG_SCSI_BUSLOGIC is not set
# CONFIG_SCSI_CPQFCTS is not set
# CONFIG_SCSI_DMX3191D is not set
# CONFIG_SCSI_DTC3280 is not set
# CONFIG_SCSI_EATA is not set
# CONFIG_SCSI_EATA_DMA is not set
# CONFIG_SCSI_EATA_PIO is not set
# CONFIG_SCSI_FUTURE_DOMAIN is not set
# CONFIG_SCSI_GDTH is not set
# CONFIG_SCSI_GENERIC_NCR5380 is not set
# CONFIG_SCSI_IPS is not set
# CONFIG_SCSI_INITIO is not set
# CONFIG_SCSI_INIA100 is not set
# CONFIG_SCSI_NCR53C406A is not set
# CONFIG_SCSI_NCR53C7xx is not set
# CONFIG_SCSI_SYM53C8XX_2 is not set
# CONFIG_SCSI_NCR53C8XX is not set
# CONFIG_SCSI_SYM53C8XX is not set
# CONFIG_SCSI_PAS16 is not set
# CONFIG_SCSI_PCI2000 is not set
# CONFIG_SCSI_PCI2220I is not set
# CONFIG_SCSI_PSI240I is not set
# CONFIG_SCSI_QLOGIC_FAS is not set
# CONFIG_SCSI_QLOGIC_ISP is not set
# CONFIG_SCSI_QLOGIC_FC is not set
# CONFIG_SCSI_QLOGIC_1280 is not set
CONFIG_SCSI_QLOGIC_QLA2XXX=y
# CONFIG_SCSI_QLOGIC_QLA2XXX_QLA2100 is not set
# CONFIG_SCSI_QLOGIC_QLA2XXX_QLA2200 is not set
CONFIG_SCSI_QLOGIC_QLA2XXX_QLA2300=m
# CONFIG_SCSI_SEAGATE is not set
# CONFIG_SCSI_SIM710 is not set
# CONFIG_SCSI_SYM53C416 is not set
# CONFIG_SCSI_DC390T is not set
# CONFIG_SCSI_T128 is not set
# CONFIG_SCSI_U14_34F is not set
# CONFIG_SCSI_ULTRASTOR is not set
# CONFIG_SCSI_NSP32 is not set
# CONFIG_SCSI_DEBUG is not set

#
# Fusion MPT device support
#
# CONFIG_FUSION is not set
# CONFIG_FUSION_BOOT is not set
# CONFIG_FUSION_ISENSE is not set
# CONFIG_FUSION_CTL is not set
# CONFIG_FUSION_LAN is not set

#
# IEEE 1394 (FireWire) support (EXPERIMENTAL)
#
# CONFIG_IEEE1394 is not set

#
# I2O device support
#
# CONFIG_I2O is not set
# CONFIG_I2O_PCI is not set
# CONFIG_I2O_BLOCK is not set
# CONFIG_I2O_LAN is not set
# CONFIG_I2O_SCSI is not set
# CONFIG_I2O_PROC is not set

#
# Network device support
#
CONFIG_NETDEVICES=y

#
# ARCnet devices
#
# CONFIG_ARCNET is not set
CONFIG_DUMMY=m
# CONFIG_BONDING is not set
# CONFIG_EQUALIZER is not set
# CONFIG_TUN is not set
# CONFIG_ETHERTAP is not set

#
# Ethernet (10 or 100Mbit)
#
CONFIG_NET_ETHERNET=y
# CONFIG_SUNLANCE is not set
# CONFIG_HAPPYMEAL is not set
# CONFIG_SUNBMAC is not set
# CONFIG_SUNQE is not set
# CONFIG_SUNGEM is not set
# CONFIG_NET_VENDOR_3COM is not set
# CONFIG_LANCE is not set
# CONFIG_NET_VENDOR_SMC is not set
# CONFIG_NET_VENDOR_RACAL is not set
# CONFIG_AT1700 is not set
# CONFIG_DEPCA is not set
# CONFIG_HP100 is not set
# CONFIG_NET_ISA is not set
CONFIG_NET_PCI=y
# CONFIG_PCNET32 is not set
# CONFIG_AMD8111_ETH is not set
# CONFIG_ADAPTEC_STARFIRE is not set
# CONFIG_AC3200 is not set
# CONFIG_APRICOT is not set
# CONFIG_B44 is not set
# CONFIG_CS89x0 is not set
# CONFIG_TULIP is not set
# CONFIG_DE4X5 is not set
# CONFIG_DGRS is not set
# CONFIG_DM9102 is not set
CONFIG_EEPRO100=m
# CONFIG_EEPRO100_PIO is not set
# CONFIG_E100 is not set
# CONFIG_LNE390 is not set
# CONFIG_FEALNX is not set
# CONFIG_NATSEMI is not set
# CONFIG_NE2K_PCI is not set
# CONFIG_NE3210 is not set
# CONFIG_ES3210 is not set
# CONFIG_8139CP is not set
# CONFIG_8139TOO is not set
# CONFIG_8139TOO_PIO is not set
# CONFIG_8139TOO_TUNE_TWISTER is not set
# CONFIG_8139TOO_8129 is not set
# CONFIG_8139_OLD_RX_RESET is not set
# CONFIG_SIS900 is not set
# CONFIG_EPIC100 is not set
# CONFIG_SUNDANCE is not set
# CONFIG_SUNDANCE_MMIO is not set
# CONFIG_TLAN is not set
# CONFIG_VIA_RHINE is not set
# CONFIG_VIA_RHINE_MMIO is not set
# CONFIG_WINBOND_840 is not set
# CONFIG_NET_POCKET is not set

#
# Ethernet (1000 Mbit)
#
# CONFIG_ACENIC is not set
# CONFIG_DL2K is not set
# CONFIG_E1000 is not set
# CONFIG_MYRI_SBUS is not set
# CONFIG_NS83820 is not set
# CONFIG_HAMACHI is not set
# CONFIG_YELLOWFIN is not set
# CONFIG_R8169 is not set
# CONFIG_SK98LIN is not set
# CONFIG_TIGON3 is not set
# CONFIG_FDDI is not set
# CONFIG_NETCONSOLE is not set
# CONFIG_HIPPI is not set
# CONFIG_PLIP is not set
# CONFIG_PPP is not set
# CONFIG_SLIP is not set

#
# Wireless LAN (non-hamradio)
#
# CONFIG_NET_RADIO is not set

#
# Token Ring devices
#
# CONFIG_TR is not set
# CONFIG_NET_FC is not set
# CONFIG_RCPCI is not set
# CONFIG_SHAPER is not set

#
# Wan interfaces
#
# CONFIG_WAN is not set

#
# Amateur Radio support
#
# CONFIG_HAMRADIO is not set

#
# IrDA (infrared) support
#
# CONFIG_IRDA is not set

#
# ISDN subsystem
#
# CONFIG_ISDN is not set

#
# Old CD-ROM drivers (not SCSI, not IDE)
#
# CONFIG_CD_NO_IDESCSI is not set

#
# Input core support
#
# CONFIG_INPUT is not set
# CONFIG_INPUT_KEYBDEV is not set
# CONFIG_INPUT_MOUSEDEV is not set
# CONFIG_INPUT_JOYDEV is not set
# CONFIG_INPUT_EVDEV is not set

#
# Character devices
#
CONFIG_VT=y
CONFIG_VT_CONSOLE=y
CONFIG_SERIAL=y
CONFIG_SERIAL_CONSOLE=y
# CONFIG_SERIAL_EXTENDED is not set
# CONFIG_SERIAL_NONSTANDARD is not set
CONFIG_UNIX98_PTYS=y
CONFIG_UNIX98_PTY_COUNT=256

#
# I2C support
#
# CONFIG_I2C is not set

#
# Mice
#
# CONFIG_BUSMOUSE is not set
CONFIG_MOUSE=y
CONFIG_PSMOUSE=y
# CONFIG_82C710_MOUSE is not set
# CONFIG_PC110_PAD is not set
# CONFIG_MK712_MOUSE is not set

#
# Joysticks
#
# CONFIG_INPUT_GAMEPORT is not set
# CONFIG_QIC02_TAPE is not set
# CONFIG_IPMI_HANDLER is not set
# CONFIG_IPMI_PANIC_EVENT is not set
# CONFIG_IPMI_DEVICE_INTERFACE is not set
# CONFIG_IPMI_KCS is not set
# CONFIG_IPMI_WATCHDOG is not set

#
# Watchdog Cards
#
# CONFIG_WATCHDOG is not set
# CONFIG_SCx200_GPIO is not set
# CONFIG_AMD_RNG is not set
# CONFIG_INTEL_RNG is not set
# CONFIG_HW_RANDOM is not set
# CONFIG_AMD_PM768 is not set
# CONFIG_NVRAM is not set
CONFIG_RTC=y
# CONFIG_DTLK is not set
# CONFIG_R3964 is not set
# CONFIG_APPLICOM is not set
# CONFIG_SONYPI is not set

#
# Ftape, the floppy tape device driver
#
# CONFIG_FTAPE is not set
# CONFIG_AGP is not set
# CONFIG_ECC is not set

#
# Direct Rendering Manager (XFree86 DRI support)
#
# CONFIG_DRM is not set
# CONFIG_MWAVE is not set

#
# Multimedia devices
#
# CONFIG_VIDEO_DEV is not set

#
# File systems
#
CONFIG_QUOTA=y
# CONFIG_QFMT_V2 is not set
# CONFIG_AUTOFS_FS is not set
CONFIG_AUTOFS4_FS=y
# CONFIG_REISERFS_FS is not set
# CONFIG_REISERFS_CHECK is not set
# CONFIG_REISERFS_PROC_INFO is not set
# CONFIG_ADFS_FS is not set
# CONFIG_ADFS_FS_RW is not set
# CONFIG_AFFS_FS is not set
# CONFIG_HFS_FS is not set
# CONFIG_HFSPLUS_FS is not set
# CONFIG_BEFS_FS is not set
# CONFIG_BEFS_DEBUG is not set
# CONFIG_BFS_FS is not set
CONFIG_EXT3_FS=y
CONFIG_JBD=y
# CONFIG_JBD_DEBUG is not set
# CONFIG_FAT_FS is not set
# CONFIG_MSDOS_FS is not set
# CONFIG_UMSDOS_FS is not set
# CONFIG_VFAT_FS is not set
# CONFIG_EFS_FS is not set
# CONFIG_JFFS_FS is not set
# CONFIG_JFFS2_FS is not set
# CONFIG_CRAMFS is not set
CONFIG_TMPFS=y
CONFIG_RAMFS=y
CONFIG_ISO9660_FS=y
# CONFIG_JOLIET is not set
# CONFIG_ZISOFS is not set
# CONFIG_JFS_FS is not set
# CONFIG_JFS_DEBUG is not set
# CONFIG_JFS_STATISTICS is not set
# CONFIG_MINIX_FS is not set
# CONFIG_VXFS_FS is not set
# CONFIG_NTFS_FS is not set
# CONFIG_NTFS_RW is not set
# CONFIG_HPFS_FS is not set
CONFIG_PROC_FS=y
# CONFIG_DEVFS_FS is not set
# CONFIG_DEVFS_MOUNT is not set
# CONFIG_DEVFS_DEBUG is not set
CONFIG_DEVPTS_FS=y
# CONFIG_QNX4FS_FS is not set
# CONFIG_QNX4FS_RW is not set
# CONFIG_ROMFS_FS is not set
CONFIG_EXT2_FS=y
# CONFIG_SYSV_FS is not set
# CONFIG_UDF_FS is not set
# CONFIG_UDF_RW is not set
# CONFIG_UFS_FS is not set
# CONFIG_UFS_FS_WRITE is not set
# CONFIG_XFS_FS is not set
# CONFIG_XFS_POSIX_ACL is not set
# CONFIG_XFS_RT is not set
# CONFIG_XFS_QUOTA is not set
# CONFIG_XFS_DMAPI is not set
# CONFIG_XFS_DEBUG is not set
# CONFIG_PAGEBUF_DEBUG is not set

#
# Network File Systems
#
# CONFIG_CODA_FS is not set
# CONFIG_INTERMEZZO_FS is not set
CONFIG_NFS_FS=m
# CONFIG_NFS_V3 is not set
# CONFIG_NFS_DIRECTIO is not set
# CONFIG_ROOT_NFS is not set
CONFIG_NFSD=m
# CONFIG_NFSD_V3 is not set
# CONFIG_NFSD_TCP is not set
CONFIG_SUNRPC=m
CONFIG_LOCKD=m
CONFIG_SMB_FS=m
# CONFIG_SMB_NLS_DEFAULT is not set
# CONFIG_NCP_FS is not set
# CONFIG_NCPFS_PACKET_SIGNING is not set
# CONFIG_NCPFS_IOCTL_LOCKING is not set
# CONFIG_NCPFS_STRONG is not set
# CONFIG_NCPFS_NFS_NS is not set
# CONFIG_NCPFS_OS2_NS is not set
# CONFIG_NCPFS_SMALLDOS is not set
# CONFIG_NCPFS_NLS is not set
# CONFIG_NCPFS_EXTRAS is not set
# CONFIG_ZISOFS_FS is not set

#
# Partition Types
#
CONFIG_PARTITION_ADVANCED=y
# CONFIG_ACORN_PARTITION is not set
# CONFIG_OSF_PARTITION is not set
# CONFIG_AMIGA_PARTITION is not set
# CONFIG_ATARI_PARTITION is not set
# CONFIG_MAC_PARTITION is not set
CONFIG_MSDOS_PARTITION=y
# CONFIG_BSD_DISKLABEL is not set
# CONFIG_MINIX_SUBPARTITION is not set
# CONFIG_SOLARIS_X86_PARTITION is not set
# CONFIG_UNIXWARE_DISKLABEL is not set
# CONFIG_LDM_PARTITION is not set
# CONFIG_SGI_PARTITION is not set
# CONFIG_ULTRIX_PARTITION is not set
# CONFIG_SUN_PARTITION is not set
# CONFIG_EFI_PARTITION is not set
CONFIG_SMB_NLS=y
CONFIG_NLS=y

#
# Native Language Support
#
CONFIG_NLS_DEFAULT="iso8859-1"
# CONFIG_NLS_CODEPAGE_437 is not set
# CONFIG_NLS_CODEPAGE_737 is not set
# CONFIG_NLS_CODEPAGE_775 is not set
# CONFIG_NLS_CODEPAGE_850 is not set
# CONFIG_NLS_CODEPAGE_852 is not set
# CONFIG_NLS_CODEPAGE_855 is not set
# CONFIG_NLS_CODEPAGE_857 is not set
# CONFIG_NLS_CODEPAGE_860 is not set
# CONFIG_NLS_CODEPAGE_861 is not set
# CONFIG_NLS_CODEPAGE_862 is not set
# CONFIG_NLS_CODEPAGE_863 is not set
# CONFIG_NLS_CODEPAGE_864 is not set
# CONFIG_NLS_CODEPAGE_865 is not set
# CONFIG_NLS_CODEPAGE_866 is not set
# CONFIG_NLS_CODEPAGE_869 is not set
# CONFIG_NLS_CODEPAGE_936 is not set
# CONFIG_NLS_CODEPAGE_950 is not set
# CONFIG_NLS_CODEPAGE_932 is not set
# CONFIG_NLS_CODEPAGE_949 is not set
# CONFIG_NLS_CODEPAGE_874 is not set
# CONFIG_NLS_ISO8859_8 is not set
# CONFIG_NLS_CODEPAGE_1250 is not set
# CONFIG_NLS_CODEPAGE_1251 is not set
# CONFIG_NLS_ISO8859_1 is not set
# CONFIG_NLS_ISO8859_2 is not set
# CONFIG_NLS_ISO8859_3 is not set
# CONFIG_NLS_ISO8859_4 is not set
# CONFIG_NLS_ISO8859_5 is not set
# CONFIG_NLS_ISO8859_6 is not set
# CONFIG_NLS_ISO8859_7 is not set
# CONFIG_NLS_ISO8859_9 is not set
# CONFIG_NLS_ISO8859_13 is not set
# CONFIG_NLS_ISO8859_14 is not set
# CONFIG_NLS_ISO8859_15 is not set
# CONFIG_NLS_KOI8_R is not set
# CONFIG_NLS_KOI8_U is not set
# CONFIG_NLS_UTF8 is not set

#
# Console drivers
#
CONFIG_VGA_CONSOLE=y
CONFIG_VIDEO_SELECT=y
# CONFIG_MDA_CONSOLE is not set

#
# Frame-buffer support
#
# CONFIG_FB is not set

#
# Sound
#
# CONFIG_SOUND is not set

#
# USB support
#
CONFIG_USB=m
# CONFIG_USB_DEBUG is not set
CONFIG_USB_DEVICEFS=y
# CONFIG_USB_BANDWIDTH is not set
# CONFIG_USB_EHCI_HCD is not set
# CONFIG_USB_UHCI is not set
# CONFIG_USB_UHCI_ALT is not set
CONFIG_USB_OHCI=m
# CONFIG_USB_AUDIO is not set
# CONFIG_USB_EMI26 is not set
# CONFIG_USB_BLUETOOTH is not set
# CONFIG_USB_MIDI is not set
# CONFIG_USB_STORAGE is not set
# CONFIG_USB_STORAGE_DEBUG is not set
# CONFIG_USB_STORAGE_DATAFAB is not set
# CONFIG_USB_STORAGE_FREECOM is not set
# CONFIG_USB_STORAGE_ISD200 is not set
# CONFIG_USB_STORAGE_DPCM is not set
# CONFIG_USB_STORAGE_HP8200e is not set
# CONFIG_USB_STORAGE_SDDR09 is not set
# CONFIG_USB_STORAGE_SDDR55 is not set
# CONFIG_USB_STORAGE_JUMPSHOT is not set
# CONFIG_USB_ACM is not set
# CONFIG_USB_PRINTER is not set
# CONFIG_USB_HID is not set
# CONFIG_USB_HIDINPUT is not set
# CONFIG_USB_HIDDEV is not set
# CONFIG_USB_KBD is not set
# CONFIG_USB_MOUSE is not set
# CONFIG_USB_AIPTEK is not set
# CONFIG_USB_WACOM is not set
# CONFIG_USB_KBTAB is not set
# CONFIG_USB_POWERMATE is not set
# CONFIG_USB_DC2XX is not set
# CONFIG_USB_MDC800 is not set
# CONFIG_USB_SCANNER is not set
# CONFIG_USB_MICROTEK is not set
# CONFIG_USB_HPUSBSCSI is not set
# CONFIG_USB_PEGASUS is not set
# CONFIG_USB_RTL8150 is not set
# CONFIG_USB_KAWETH is not set
# CONFIG_USB_CATC is not set
# CONFIG_USB_AX8817X is not set
# CONFIG_USB_CDCETHER is not set
# CONFIG_USB_USBNET is not set
# CONFIG_USB_USS720 is not set

#
# USB Serial Converter support
#
CONFIG_USB_SERIAL=m
# CONFIG_USB_SERIAL_DEBUG is not set
# CONFIG_USB_SERIAL_GENERIC is not set
# CONFIG_USB_SERIAL_BELKIN is not set
# CONFIG_USB_SERIAL_WHITEHEAT is not set
# CONFIG_USB_SERIAL_DIGI_ACCELEPORT is not set
# CONFIG_USB_SERIAL_EMPEG is not set
# CONFIG_USB_SERIAL_FTDI_SIO is not set
# CONFIG_USB_SERIAL_VISOR is not set
# CONFIG_USB_SERIAL_IPAQ is not set
# CONFIG_USB_SERIAL_IR is not set
# CONFIG_USB_SERIAL_EDGEPORT is not set
# CONFIG_USB_SERIAL_EDGEPORT_TI is not set
# CONFIG_USB_SERIAL_KEYSPAN_PDA is not set
# CONFIG_USB_SERIAL_KEYSPAN is not set
# CONFIG_USB_SERIAL_MCT_U232 is not set
# CONFIG_USB_SERIAL_KLSI is not set
# CONFIG_USB_SERIAL_KOBIL_SCT is not set
CONFIG_USB_SERIAL_PL2303=m
# CONFIG_USB_SERIAL_CYBERJACK is not set
# CONFIG_USB_SERIAL_XIRCOM is not set
# CONFIG_USB_SERIAL_OMNINET is not set
# CONFIG_USB_RIO500 is not set
# CONFIG_USB_AUERSWALD is not set
# CONFIG_USB_TIGL is not set
# CONFIG_USB_BRLVGER is not set
# CONFIG_USB_LCD is not set

#
# Bluetooth support
#
# CONFIG_BLUEZ is not set

#
# Kernel hacking
#
CONFIG_DEBUG_KERNEL=y
# CONFIG_DEBUG_HZ is not set
# CONFIG_DEBUG_STACKOVERFLOW is not set
# CONFIG_DEBUG_HIGHMEM is not set
# CONFIG_DEBUG_SLAB is not set
# CONFIG_DEBUG_IOVIRT is not set
CONFIG_MAGIC_SYSRQ=y
# CONFIG_KMSGDUMP is not set
# CONFIG_DEBUG_SPINLOCK is not set
# CONFIG_FRAME_POINTER is not set
# CONFIG_HIGHMEM_EMULATION is not set
# CONFIG_X86_REMOTE_DEBUG is not set
# CONFIG_KERNEL_DEBUGGING is not set
CONFIG_LOG_BUF_SHIFT=0

#
# Cryptographic options
#
# CONFIG_CRYPTO is not set

#
# Library routines
#
# CONFIG_CRC32 is not set
# CONFIG_ZLIB_INFLATE is not set
# CONFIG_ZLIB_DEFLATE is not set
# CONFIG_FW_LOADER is not set
# CONFIG_QSORT is not set

^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: 2.4.23pre6aa3 scsi oops
  2003-11-04  2:49 2.4.23pre6aa3 scsi oops Jamie Clark
@ 2003-11-04 10:28 ` Andrea Arcangeli
  2003-11-04 11:52   ` Jamie Clark
  0 siblings, 1 reply; 15+ messages in thread
From: Andrea Arcangeli @ 2003-11-04 10:28 UTC (permalink / raw)
  To: Jamie Clark; +Cc: linux-kernel

On Tue, Nov 04, 2003 at 10:49:29AM +0800, Jamie Clark wrote:
> Hi,
> 
> Consistent oops with 2.4.23pre6aa3 after 3-4 hours running bonnie on 
> ext3 fs through qla2300 HBA. (w SMP, HIGHMEM) The test machine was 
> completely wedged so I ended up transcribing the oops from the vga 
> console. No typos I think.

I need to release an update that has a chance to fix it. Jens identified
problem in his last_merge scsi patch so I will back it out.

you can try to backout it by yourself in the meantime, it's called
\*elevator-merge-fast-path\* . or you can disable it with this patch:

--- xx/drivers/block/elevator.c.~1~	2003-10-17 21:49:49.000000000 +0200
+++ xx/drivers/block/elevator.c	2003-11-04 11:27:13.000000000 +0100
@@ -77,6 +77,7 @@ inline int bh_rq_in_between(struct buffe
 static int rq_mergeable(struct request *req, struct buffer_head *bh,
 			request_queue_t *q, int rw, int count, int max_sectors)
 {
+	return 0;
 	if (q->head_active && !q->plugged) {
 		struct request *next;
 		next = blkdev_entry_next_request(&q->queue_head);

Hope this helps.

^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: 2.4.23pre6aa3 scsi oops
  2003-11-04 10:28 ` Andrea Arcangeli
@ 2003-11-04 11:52   ` Jamie Clark
  2003-11-04 12:48     ` Andrea Arcangeli
                       ` (2 more replies)
  0 siblings, 3 replies; 15+ messages in thread
From: Jamie Clark @ 2003-11-04 11:52 UTC (permalink / raw)
  To: Andrea Arcangeli; +Cc: linux-kernel

I made the quick fix (disabling rq_mergeable) and started the load test.
Will let it run for a week or so.

Thanks for the help!
-Jamie

FYI an observation from my last test: the read latency seems to be much
improved and more consistent under this kernel (2.4.23pre6aa3, before
the oops and before this fix).  The maximum latency seemed steady over
the whole test without any of the longish pauses that showed up under
2.4.19. Quite a difference.

Andrea Arcangeli wrote:

>On Tue, Nov 04, 2003 at 10:49:29AM +0800, Jamie Clark wrote:
>
>>Hi,
>>
>>Consistent oops with 2.4.23pre6aa3 after 3-4 hours running bonnie on 
>>ext3 fs through qla2300 HBA. (w SMP, HIGHMEM) The test machine was 
>>completely wedged so I ended up transcribing the oops from the vga 
>>console. No typos I think.
>>
>I need to release an update that has a chance to fix it. Jens identified
>problem in his last_merge scsi patch so I will back it out.
>
>you can try to backout it by yourself in the meantime, it's called
>\*elevator-merge-fast-path\* . or you can disable it with this patch:
>



^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: 2.4.23pre6aa3 scsi oops
  2003-11-04 11:52   ` Jamie Clark
@ 2003-11-04 12:48     ` Andrea Arcangeli
  2003-11-27  3:21     ` 2.4.23pre6aa3 - possible ext3 deadlock Jamie Clark
  2003-12-06  1:05     ` 2.4.23pre6aa3 scsi oops Andrea Arcangeli
  2 siblings, 0 replies; 15+ messages in thread
From: Andrea Arcangeli @ 2003-11-04 12:48 UTC (permalink / raw)
  To: Jamie Clark; +Cc: linux-kernel

On Tue, Nov 04, 2003 at 07:52:40PM +0800, Jamie Clark wrote:
> the oops and before this fix).  The maximum latency seemed steady over
> the whole test without any of the longish pauses that showed up under
> 2.4.19. Quite a difference.

this is the effect of the elevator-lowlatency also merged in mainline
recently. Good to hear, thanks.

^ permalink raw reply	[flat|nested] 15+ messages in thread

* 2.4.23pre6aa3 - possible ext3 deadlock
  2003-11-04 11:52   ` Jamie Clark
  2003-11-04 12:48     ` Andrea Arcangeli
@ 2003-11-27  3:21     ` Jamie Clark
  2003-11-27 12:56       ` Jan Kara
  2003-12-06  1:05     ` 2.4.23pre6aa3 scsi oops Andrea Arcangeli
  2 siblings, 1 reply; 15+ messages in thread
From: Jamie Clark @ 2003-11-27  3:21 UTC (permalink / raw)
  To: Jan Kara, Andrea Arcangeli; +Cc: linux-kernel

[-- Attachment #1: Type: text/plain, Size: 1965 bytes --]

Followup to last test with 2.4.23pre6aa3. same config as before (ext3 
/ql2300 / bonnie++) I forgot to mention that I have quotas enabled on 
the partition under test.

There have been a few 2.4 / quota / ext3 deadlock-type  mails on the 
list recently so I'm sot sure if this is related. I can see an atime 
update in the call trace of one of the bonnies but no dquot_ calls 
anywhere.  I haven't applied Jan's patch as it came out before I could 
reliably reproduce this problem.

In the meantime I'll restart testing with noatime - which is how I'd 
usually mount such a fs anyhow.

The box seems to deadlock after 3-6 days of filesystem exercise. I have 
repeated this 3 times. The last run was 6 days and and then I see all 
disk I/O has ceased. The system remains pingable and sshd connects at 
the TCP layer but nothing happens after that. The root filesystem is 
ext3 but is a completely separate device (motherboard scsi) to the 
filesystems running bonnie (external FC RAID). It seems that all 
filesystem activity ceases.

I have run the alt-sysrq-t ouput through ksymoops and attached.  Hope 
this is useful to someone who knows.

-jamie

Jamie Clark wrote:

> I made the quick fix (disabling rq_mergeable) and started the load test.
> Will let it run for a week or so.
>
> Thanks for the help!
> -Jamie
>
> Andrea Arcangeli wrote:
>
>> On Tue, Nov 04, 2003 at 10:49:29AM +0800, Jamie Clark wrote:
>>
>>> Hi,
>>>
>>> Consistent oops with 2.4.23pre6aa3 after 3-4 hours running bonnie on 
>>> ext3 fs through qla2300 HBA. (w SMP, HIGHMEM) The test machine was 
>>> completely wedged so I ended up transcribing the oops from the vga 
>>> console. No typos I think.
>>>
>> I need to release an update that has a chance to fix it. Jens identified
>> problem in his last_merge scsi patch so I will back it out.
>>
>> you can try to backout it by yourself in the meantime, it's called
>> \*elevator-merge-fast-path\* . or you can disable it with this patch:
>

[-- Attachment #2: ksymoops.out --]
[-- Type: text/plain, Size: 36610 bytes --]

ksymoops 2.4.5 on i686 2.4.23-pre6aa3.  Options used
     -V (default)
     -k /proc/ksyms (default)
     -l /proc/modules (default)
     -o /lib/modules/2.4.23-pre6aa3/ (default)
     -m /boot/System.map (specified)

init          R F7907F14     0     1      0 12973               (NOTLB)
Using defaults from ksymoops -t elf32-i386 -a i386
Call Trace:         [<c01228b8>] (28) [<c012281c>] (12) [<c0151f0c>] (40) [<c0151bdc>] (20) [<c015238a>] (40) [<c0149454>] (44) [<c0107143>] (60)
migration_CPU S C1C1E000     0     2      1             3       (L-TLB)
Call Trace:         [<c011768f>] (16) [<c0117568>] (36) [<c01056d0>] (12)
migration_CPU S F790E346     0     3      1             4     2 (L-TLB)
Call Trace:         [<c011768f>] (16) [<c0117568>] (36) [<c01056d0>] (12) 
keventd       R 00000000     0     4      1             5     3 (L-TLB)
Call Trace:         [<c0127045>] (64) [<c01056d0>] (12)
ksoftirqd_CPU S F7908000     0     5      1             6     4 (L-TLB)
Call Trace:         [<c011e96f>] (20) [<c01056d0>] (12)
ksoftirqd_CPU S C1C1E000     0     6      1             7     5 (L-TLB)
Call Trace:         [<c011e96f>] (20) [<c01056d0>] (12)
kswapd        R C1C1C000  5332     7      1             8     6 (L-TLB)
Call Trace:         [<c0137ca9>] (36) [<c01056d0>] (12)
bdflush       S 00000286     0     8      1             9     7 (L-TLB)
Call Trace:         [<c0116351>] (36) [<c01459e3>] (16) [<c01056d0>] (12)
kupdated      R C1C19FB0  5224     9      1            10     8 (L-TLB)
Call Trace:         [<c01228b8>] (28) [<c012281c>] (12) [<c0145aaa>] (36) [<c01056d0>] (12)
ahc_dv_0      S F75C3F3C    12    10      1            11     9 (L-TLB)
Call Trace:         [<c0105e67>] (40) [<c0105f4b>] (12) [<c01dc79e>] (24) [<c01056d0>] (12)
ahc_dv_1      S F7524D1C  6188    11      1            12    10 (L-TLB)
Call Trace:         [<c0105e67>] (40) [<c0105f4b>] (12) [<c01dc79e>] (24) [<c01056d0>] (12)
scsi_eh_0     S C1C3BFDC  3304    12      1            13    11 (L-TLB)
Call Trace:         [<c0105e67>] (40) [<c0105f4b>] (12) [<c01d0d48>] (56) [<c01056d0>] (12)
scsi_eh_1     S C1C39FDC  6176    13      1            14    12 (L-TLB)
Call Trace:         [<c0105e67>] (40) [<c0105f4b>] (12) [<c01d0d48>] (56) [<c01056d0>] (12)
kjournald     R 00000286     0    14      1            57    13 (L-TLB)
Call Trace:         [<c0116351>] (36) [<c017de49>] (36) [<c017dca0>] (12) [<c01056d0>] (12)
qla2300_dpc2  S C2665FDC  2384    57      1            58    14 (L-TLB)
Call Trace:         [<c0105e67>] (40) [<c0105f4b>] (12) [<c2797943>] (120) [<c01056d0>] (12)
scsi_eh_2     S C2663FDC  2384    58      1            61    57 (L-TLB)
Call Trace:         [<c0105e67>] (40) [<c0105f4b>] (12) [<c01d0d48>] (56) [<c01056d0>] (12)
khubd         S 00000001  2384    61      1            99    58 (L-TLB)
Call Trace:         [<c26048bb>] (28) [<c260ddac>] (04) [<c260ddac>] (08) [<c01056d0>] (12)
kjournald     R 00000286  2384    99      1           100    61 (L-TLB)
Call Trace:         [<c0116351>] (36) [<c017de49>] (36) [<c017dca0>] (12) [<c01056d0>] (12)
kjournald     S 00000286    52   100      1           137    99 (L-TLB)
Call Trace:         [<c0116351>] (36) [<c017de49>] (36) [<c017dca0>] (12) [<c01056d0>] (12)
portmap       S 7FFFFFFF     0   137      1           208   100 (NOTLB)
Call Trace:         [<c012283c>] (24) [<c015267a>] (16) [<c01526b0>] (32) [<c0152884>] (64) [<c0151bdc>] (20) [<c0107143>] (60)
syslogd       R 7FFFFFFF     0   208      1           211   137 (NOTLB)
Call Trace:         [<c012283c>] (12) [<c01fe467>] (28) [<c0151f0c>] (40) [<c0151bdc>] (20) [<c015238a>] (84) [<c0107143>] (60)
klogd         R F7AA6000    52   211      1           219   208 (NOTLB)
Call Trace:         [<c01198bc>] (52) [<c016af45>] (16) [<c0140be7>] (36) [<c0107143>] (60)
rpc.statd     S 7FFFFFFF     0   219      1           225   211 (NOTLB)
Call Trace:         [<c012283c>] (12) [<c01fe467>] (28) [<c0151f0c>] (40) [<c0151bdc>] (20) [<c015238a>] (84) [<c0107143>] (60)
gpm           S F7A4BF14    52   225      1           230   219 (NOTLB)
Call Trace:         [<c01228b8>] (28) [<c012281c>] (12) [<c0151f0c>] (40) [<c0151bdc>] (20) [<c015238a>] (84) [<c0107143>] (60)
inetd         S 7FFFFFFF     0   230      1           249   225 (NOTLB)
Call Trace:         [<c012283c>] (12) [<c01fe467>] (28) [<c0151f0c>] (40) [<c0151bdc>] (20) [<c015238a>] (84) [<c0107143>] (60)
nfsd          R C23EBF54     0   241      1           253   242 (L-TLB)
Call Trace:         [<c01228b8>] (28) [<c012281c>] (12) [<c3228e99>] (72) [<f7a9029c>] (44) [<f7a9a520>] (12) [<c01056d0>] (12)
nfsd          R C23C9F54     0   242      1           241   243 (L-TLB)
Call Trace:         [<c01228b8>] (28) [<c012281c>] (12) [<c3228e99>] (72) [<f7a9029c>] (56) [<c01056d0>] (12)
nfsd          R C23A9F54    52   243      1           242   244 (L-TLB)
Call Trace:         [<c01228b8>] (28) [<c012281c>] (12) [<c3228e99>] (72) [<f7a9029c>] (56) [<c01056d0>] (12)
nfsd          R C238FF54     0   244      1           243   245 (L-TLB)
Call Trace:         [<c01228b8>] (28) [<c012281c>] (12) [<c3228e99>] (72) [<f7a9029c>] (56) [<c01056d0>] (12)
nfsd          R C236FF54    52   245      1           244   246 (L-TLB)
Call Trace:         [<c01228b8>] (28) [<c012281c>] (12) [<c3228e99>] (72) [<f7a9029c>] (56) [<c01056d0>] (12)
nfsd          R C234DF54     0   246      1           245   247 (L-TLB)
Call Trace:         [<c01228b8>] (28) [<c012281c>] (12) [<c3228e99>] (72) [<f7a9029c>] (56) [<c01056d0>] (12)
nfsd          R C232DF54    52   247      1           246   248 (L-TLB)
Call Trace:         [<c01228b8>] (28) [<c012281c>] (12) [<c3228e99>] (72) [<f7a9029c>] (56) [<c01056d0>] (12)
nfsd          R C2313F54    52   248      1           247   249 (L-TLB)
Call Trace:         [<c01228b8>] (28) [<c012281c>] (12) [<c3228e99>] (72) [<f7a9029c>] (48) [<f7a9a520>] (08) [<c01056d0>] (12)
lockd         S 7FFFFFFF  2384   249      1   250     248   230 (L-TLB)
Call Trace:         [<c012283c>] (20) [<c3225244>] (20) [<c3228e99>] (72) [<f7ab1cb3>] (36) [<c01056d0>] (12)
rpciod        S 00000001    52   250    249                     (L-TLB)
Call Trace:         [<c3225410>] (12) [<c322f74c>] (12) [<c322f74c>] (12) [<c322f744>] (04) [<c322f744>] (08) [<c01056d0>] (04) [<c322f74c>] (08)
rpc.mountd    S 7FFFFFFF    52   253      1           259   241 (NOTLB)
Call Trace:         [<c012283c>] (24) [<c015267a>] (16) [<c01526b0>] (32) [<c0152884>] (64) [<c0151bdc>] (20) [<c0107143>] (60)
sshd          S 7FFFFFFF    52   259      1 18848     263   253 (NOTLB)
Call Trace:         [<c012283c>] (12) [<c01fe467>] (28) [<c0151f0c>] (40) [<c0151bdc>] (20) [<c015238a>] (84) [<c0107143>] (60)
xfs           R C2287F14     0   263      1           268   259 (NOTLB)
Call Trace:         [<c01228b8>] (28) [<c012281c>] (12) [<c0151f0c>] (40) [<c0151bdc>] (20) [<c015238a>] (44) [<c011dd7d>] (40) [<c0107143>] (60)
atd           R F7A31F60    52   268      1           271   263 (NOTLB)
Call Trace:         [<c01228b8>] (28) [<c012281c>] (12) [<c01229ec>] (68) [<c0107143>] (60)
cron          R C2281F60     0   271      1 19062     275   268 (NOTLB)
Call Trace:         [<c01228b8>] (28) [<c012281c>] (12) [<c01229ec>] (68) [<c0107143>] (60)
apache        R C227DF14    52   275      1 18141     281   271 (NOTLB)
Call Trace:         [<c01228b8>] (28) [<c012281c>] (12) [<c0151f0c>] (40) [<c0151bdc>] (20) [<c015238a>] (84) [<c0107143>] (60)
getty         S 7FFFFFFF  5028   281      1           282   275 (NOTLB)
Call Trace:         [<c012283c>] (36) [<c0196084>] (04) [<c01960cf>] (88) [<c0191c3a>] (32) [<c0140be7>] (36) [<c0107143>] (60)
getty         S 7FFFFFFF     0   282      1           283   281 (NOTLB)
Call Trace:         [<c012283c>] (36) [<c0196084>] (04) [<c01960cf>] (88) [<c0191c3a>] (32) [<c0140be7>] (36) [<c0107143>] (60)
getty         S 7FFFFFFF     0   283      1           284   282 (NOTLB)
Call Trace:         [<c012283c>] (36) [<c0196084>] (04) [<c01960cf>] (88) [<c0191c3a>] (32) [<c0140be7>] (36) [<c0107143>] (60)
getty         S 7FFFFFFF    52   284      1           285   283 (NOTLB)
Call Trace:         [<c012283c>] (36) [<c0196084>] (04) [<c01960cf>] (88) [<c0191c3a>] (32) [<c0140be7>] (36) [<c0107143>] (60)
getty         S 7FFFFFFF    52   285      1           286   284 (NOTLB)
Call Trace:         [<c012283c>] (36) [<c0196084>] (04) [<c01960cf>] (88) [<c0191c3a>] (32) [<c0140be7>] (36) [<c0107143>] (60)
getty         S 7FFFFFFF    52   286      1           884   285 (NOTLB)
Call Trace:         [<c012283c>] (36) [<c0196084>] (04) [<c01960cf>] (88) [<c0191c3a>] (32) [<c0140be7>] (36) [<c0107143>] (60)
kjournald     D F7A9D294  4796   884      1           895   286 (L-TLB)
Call Trace:         [<c0105da8>] (36) [<c0105f40>] (16) [<c017c0b4>] (244) [<c0115f53>] (08) [<c0115f5f>] (76) [<c017de26>] (40) [<c017dca0>] (12) [<c01056d0>] (12)
screen        S 7FFFFFFF  2384   895      1   896     902   884 (NOTLB)
Call Trace:         [<c012283c>] (40) [<c0151f0c>] (40) [<c0151bdc>] (20) [<c015238a>] (84) [<c0107143>] (60)
rlbench.sh    S 00000000     0   896    895   898               (NOTLB)
Call Trace:         [<c011d6d6>] (60) [<c0107143>] (60)
rlbench       R C93DFF60  2384   897    896           898       (NOTLB)
Call Trace:         [<c01228b8>] (28) [<c012281c>] (12) [<c01229ec>] (68) [<c0107143>] (60)
tee           S C936E000  4888   898    896                 897 (NOTLB)
Call Trace:         [<c014afeb>] (60) [<c014b0c4>] (36) [<c0140be7>] (36) [<c0107143>] (60)
screen        S 7FFFFFFF  5524   902      1   903     907   895 (NOTLB)
Call Trace:         [<c012283c>] (40) [<c0151f0c>] (40) [<c0151bdc>] (20) [<c015238a>] (84) [<c0107143>] (60)
bash          S 00000000  2384   903    902  9385               (NOTLB)
Call Trace:         [<c011d6d6>] (60) [<c0107143>] (60)
screen        S 7FFFFFFF     0   907      1   908    9945   902 (NOTLB)
Call Trace:         [<c012283c>] (40) [<c0151f0c>] (40) [<c0151bdc>] (20) [<c015238a>] (84) [<c0107143>] (60)
bash          S 00000000     0   908    907  9387               (NOTLB)
Call Trace:         [<c011d6d6>] (60) [<c0107143>] (60)
bonnie++      D F7A9D294     0  9385    903  9400               (NOTLB)
Call Trace:         [<c0105da8>] (36) [<c0105f40>] (16) [<c017adfa>] (24) [<c0178ffd>] (24) [<c0173d48>] (24) [<c0156cbe>] (20) [<c01584aa>] (12) [<c012e882>] (52) [<c012ec64>] (16) [<c012eac0>] (24) [<c016f007>] (36) [<c0140be7>] (36) [<c0107143>] (60)
bonnie++      R C17FE560     0  9387    908  9394               (NOTLB)
bonnie++      R D7933F30     0  9394   9387                     (NOTLB)
Call Trace:         [<c01228b8>] (28) [<c012281c>] (12) [<c01526b0>] (32) [<c0152884>] (64) [<c0151bdc>] (20) [<c0107143>] (60)
bonnie++      R E157FF30     0  9400   9385                     (NOTLB)
Call Trace:         [<c01228b8>] (28) [<c012281c>] (12) [<c01526b0>] (32) [<c0152884>] (64) [<c0151bdc>] (20) [<c0107143>] (60)
screen        S 7FFFFFFF     0  9945      1  9946   12973   907 (NOTLB)
Call Trace:         [<c012283c>] (40) [<c0151f0c>] (40) [<c0151bdc>] (20) [<c015238a>] (84) [<c0107143>] (60)
bash          S 00000000     0  9946   9945  9947               (NOTLB)
Call Trace:         [<c011d6d6>] (60) [<c0107143>] (60)
bonnie++      D F7A9D294     0  9947   9946  9963               (NOTLB)
Call Trace:         [<c0105da8>] (36) [<c0105f40>] (16) [<c017adfa>] (24) [<c0178ffd>] (24) [<c0173d48>] (24) [<c0156cbe>] (20) [<c01584aa>] (12) [<c012e882>] (52) [<c012ec64>] (16) [<c012eac0>] (60) [<c0140be7>] (36) [<c0107143>] (60)
bonnie++      R DF49FF30     0  9963   9947                     (NOTLB)
Call Trace:         [<c01228b8>] (28) [<c012281c>] (12) [<c01526b0>] (32) [<c0152884>] (64) [<c0151bdc>] (20) [<c0107143>] (60)
lpd           S 7FFFFFFF     0 12973      1                9945 (NOTLB)
Call Trace:         [<c012283c>] (12) [<c01fe467>] (28) [<c0151f0c>] (40) [<c0151bdc>] (20) [<c015238a>] (44) [<c01ff657>] (40) [<c0107143>] (60)
apache        S 7FFFFFFF    52 18137    275         18138       (NOTLB)
Call Trace:         [<c012283c>] (40) [<c021f5ed>] (68) [<c021f77a>] (32) [<c0239a44>] (36) [<c01fec56>] (52) [<c0113d58>] (36) [<c012c729>] (52) [<c0124543>] (28) [<c01ff66c>] (44) [<c0107143>] (60)
apache        S 7FFFFFFF    52 18138    275         18139 18137 (NOTLB)
Call Trace:         [<c012283c>] (40) [<c021f5ed>] (68) [<c021f77a>] (32) [<c0239a44>] (36) [<c01fec56>] (52) [<c0113d58>] (36) [<c012c729>] (52) [<c0124543>] (28) [<c01ff66c>] (44) [<c0107143>] (60)
apache        S 7FFFFFFF    52 18139    275         18140 18138 (NOTLB)
Call Trace:         [<c012283c>] (40) [<c021f5ed>] (68) [<c021f77a>] (32) [<c0239a44>] (36) [<c01fec56>] (52) [<c0113d58>] (36) [<c012c729>] (52) [<c0124543>] (28) [<c01ff66c>] (44) [<c0107143>] (60)
apache        S 7FFFFFFF    52 18140    275         18141 18139 (NOTLB)
Call Trace:         [<c012283c>] (40) [<c021f5ed>] (68) [<c021f77a>] (32) [<c0239a44>] (36) [<c01fec56>] (52) [<c0113d58>] (36) [<c012c729>] (52) [<c0124543>] (28) [<c01ff66c>] (44) [<c0107143>] (60)
apache        S 7FFFFFFF    52 18141    275               18140 (NOTLB)
Call Trace:         [<c012283c>] (40) [<c021f5ed>] (68) [<c021f77a>] (32) [<c0239a44>] (36) [<c01fec56>] (52) [<c0113d58>] (36) [<c012c729>] (52) [<c0124543>] (28) [<c01ff66c>] (44) [<c0107143>] (60)
sshd          R 7FFFFFFF     0 18848    259 18850               (NOTLB)
Call Trace:         [<c012283c>] (40) [<c0151f0c>] (40) [<c0151bdc>] (20) [<c015238a>] (84) [<c0107143>] (60)
bash          S 7FFFFFFF     0 18850  18848                     (NOTLB)
Call Trace:         [<c012283c>] (36) [<c0196084>] (04) [<c01960cf>] (88) [<c0191c3a>] (32) [<c0140be7>] (36) [<c0107143>] (60)
cron          S E291A000     0 19062    271 19063               (NOTLB)
Call Trace:         [<c014afeb>] (60) [<c014b0c4>] (36) [<c0140be7>] (36) [<c0107143>] (60)
sh            S 00000000     0 19063  19062 19064               (NOTLB)
Call Trace:         [<c011d6d6>] (60) [<c0107143>] (60)
run-parts     S 7FFFFFFF     0 19064  19063 19091               (NOTLB)
Call Trace:         [<c012283c>] (40) [<c0151f0c>] (40) [<c0151bdc>] (20) [<c015238a>] (84) [<c0107143>] (60)
find          S 00000000     0 19091  19064 19092               (NOTLB)
Call Trace:         [<c011d6d6>] (60) [<c0107143>] (60)
updatedb      S 00000000     0 19092  19091 19109               (NOTLB)
Call Trace:         [<c011d6d6>] (60) [<c0107143>] (60)
updatedb      S 00000000     0 19106  19092 19107   19108       (NOTLB)
Call Trace:         [<c011d6d6>] (60) [<c0107143>] (60)
find          R F735CAC0  2384 19107  19106                     (NOTLB)
Call Trace:         [<c0141e75>] (40) [<c01715e2>] (20) [<c016eca6>] (84) [<c01493d4>] (52) [<c01510f4>] (12) [<c01516e0>] (24) [<c015182e>] (08) [<c01516e0>] (32) [<c015087b>] (24) [<c0107143>] (60)
sort          R current      0 19108  19092         19109 19106 (NOTLB)
frcode        S E20EA000    16 19109  19092               19108 (NOTLB)
Call Trace:         [<c014afeb>] (60) [<c014b0c4>] (36) [<c0140be7>] (36) [<c0107143>] (60)
Warning (Oops_read): Code line not seen, dumping what data is available

Proc;  init

>>EIP; f7907f14 <END_OF_CODE+17ce15/????>   <=====

Trace; c01228b8 <schedule_timeout+90/b0>
Trace; c012281c <process_timeout+0/c>
Trace; c0151f0c <do_select+1c8/204>
Trace; c0151bdc <__pollwait+0/c0>
Trace; c015238a <sys_select+41a/5fc>
Trace; c0149454 <sys_stat64+64/70>
Trace; c0107143 <system_call+33/38>
Proc;  migration_CPU

>>EIP; c1c1e000 <_end+187f58c/1dad58c>   <=====

Trace; c011768f <migration_thread+127/340>
Trace; c0117568 <migration_thread+0/340>
Trace; c01056d0 <arch_kernel_thread+28/38>
Proc;  migration_CPU

>>EIP; f790e346 <END_OF_CODE+183247/????>   <=====

Trace; c011768f <migration_thread+127/340>
Trace; c0117568 <migration_thread+0/340>
Trace; c01056d0 <arch_kernel_thread+28/38>
Proc;  keventd

>>EIP; 00000000 Before first symbol

Trace; c0127045 <context_thread+115/1cc>
Trace; c01056d0 <arch_kernel_thread+28/38>
Proc;  ksoftirqd_CPU

>>EIP; f7908000 <END_OF_CODE+17cf01/????>   <=====

Trace; c011e96f <ksoftirqd+93/c8>
Trace; c01056d0 <arch_kernel_thread+28/38>
Proc;  ksoftirqd_CPU

>>EIP; c1c1e000 <_end+187f58c/1dad58c>   <=====

Trace; c011e96f <ksoftirqd+93/c8>
Trace; c01056d0 <arch_kernel_thread+28/38>
Proc;  kswapd

>>EIP; c1c1c000 <_end+187d58c/1dad58c>   <=====

Trace; c0137ca9 <kswapd+75/ae>
Trace; c01056d0 <arch_kernel_thread+28/38>
Proc;  bdflush

>>EIP; 00000286 Before first symbol   <=====

Trace; c0116351 <interruptible_sleep_on+51/80>
Trace; c01459e3 <bdflush+cb/d0>
Trace; c01056d0 <arch_kernel_thread+28/38>
Proc;  kupdated

>>EIP; c1c19fb0 <_end+187b53c/1dad58c>   <=====

Trace; c01228b8 <schedule_timeout+90/b0>
Trace; c012281c <process_timeout+0/c>
Trace; c0145aaa <kupdate+c2/188>
Trace; c01056d0 <arch_kernel_thread+28/38>
Proc;  ahc_dv_0

>>EIP; f75c3f3c <[sunrpc].bss.end+34273429/343f34ed>   <=====

Trace; c0105e67 <__down_interruptible+5f/e4>
Trace; c0105f4b <__down_failed_interruptible+7/c>
Trace; c01dc79e <.text.lock.aic7xxx_osm+bb/21d>
Trace; c01056d0 <arch_kernel_thread+28/38>
Proc;  ahc_dv_1

>>EIP; f7524d1c <[sunrpc].bss.end+341d4209/343f34ed>   <=====

Trace; c0105e67 <__down_interruptible+5f/e4>
Trace; c0105f4b <__down_failed_interruptible+7/c>
Trace; c01dc79e <.text.lock.aic7xxx_osm+bb/21d>
Trace; c01056d0 <arch_kernel_thread+28/38>
Proc;  scsi_eh_0

>>EIP; c1c3bfdc <_end+189d568/1dad58c>   <=====

Trace; c0105e67 <__down_interruptible+5f/e4>
Trace; c0105f4b <__down_failed_interruptible+7/c>
Trace; c01d0d48 <.text.lock.scsi_error+dd/f5>
Trace; c01056d0 <arch_kernel_thread+28/38>
Proc;  scsi_eh_1

>>EIP; c1c39fdc <_end+189b568/1dad58c>   <=====

Trace; c0105e67 <__down_interruptible+5f/e4>
Trace; c0105f4b <__down_failed_interruptible+7/c>
Trace; c01d0d48 <.text.lock.scsi_error+dd/f5>
Trace; c01056d0 <arch_kernel_thread+28/38>
Proc;  kjournald

>>EIP; 00000286 Before first symbol   <=====

Trace; c0116351 <interruptible_sleep_on+51/80>
Trace; c017de49 <kjournald+199/250>
Trace; c017dca0 <commit_timeout+0/c>
Trace; c01056d0 <arch_kernel_thread+28/38>
Proc;  qla2300_dpc2

>>EIP; c2665fdc <[mii].rodata.end+2dc7dd/bf7801>   <=====

Trace; c0105e67 <__down_interruptible+5f/e4>
Trace; c0105f4b <__down_failed_interruptible+7/c>
Trace; c2797943 <[mii].rodata.end+40e144/bf7801>
Trace; c01056d0 <arch_kernel_thread+28/38>
Proc;  scsi_eh_2

>>EIP; c2663fdc <[mii].rodata.end+2da7dd/bf7801>   <=====

Trace; c0105e67 <__down_interruptible+5f/e4>
Trace; c0105f4b <__down_failed_interruptible+7/c>
Trace; c01d0d48 <.text.lock.scsi_error+dd/f5>
Trace; c01056d0 <arch_kernel_thread+28/38>
Proc;  khubd

>>EIP; 00000001 Before first symbol   <=====

Trace; c26048bb <[mii].rodata.end+27b0bc/bf7801>
Trace; c260ddac <[mii].rodata.end+2845ad/bf7801>
Trace; c260ddac <[mii].rodata.end+2845ad/bf7801>
Trace; c01056d0 <arch_kernel_thread+28/38>
Proc;  kjournald

>>EIP; 00000286 Before first symbol   <=====

Trace; c0116351 <interruptible_sleep_on+51/80>
Trace; c017de49 <kjournald+199/250>
Trace; c017dca0 <commit_timeout+0/c>
Trace; c01056d0 <arch_kernel_thread+28/38>
Proc;  kjournald

>>EIP; 00000286 Before first symbol   <=====

Trace; c0116351 <interruptible_sleep_on+51/80>
Trace; c017de49 <kjournald+199/250>
Trace; c017dca0 <commit_timeout+0/c>
Trace; c01056d0 <arch_kernel_thread+28/38>
Proc;  portmap

>>EIP; 7fffffff Before first symbol   <=====

Trace; c012283c <schedule_timeout+14/b0>
Trace; c015267a <do_poll+86/dc>
Trace; c01526b0 <do_poll+bc/dc>
Trace; c0152884 <sys_poll+1b4/2a3>
Trace; c0151bdc <__pollwait+0/c0>
Trace; c0107143 <system_call+33/38>
Proc;  syslogd

>>EIP; 7fffffff Before first symbol   <=====

Trace; c012283c <schedule_timeout+14/b0>
Trace; c01fe467 <sock_poll+23/28>
Trace; c0151f0c <do_select+1c8/204>
Trace; c0151bdc <__pollwait+0/c0>
Trace; c015238a <sys_select+41a/5fc>
Trace; c0107143 <system_call+33/38>
Proc;  klogd

>>EIP; f7aa6000 <END_OF_CODE+31af01/????>   <=====

Trace; c01198bc <do_syslog+ec/3e4>
Trace; c016af45 <kmsg_read+11/18>
Trace; c0140be7 <sys_read+8f/100>
Trace; c0107143 <system_call+33/38>
Proc;  rpc.statd

>>EIP; 7fffffff Before first symbol   <=====

Trace; c012283c <schedule_timeout+14/b0>
Trace; c01fe467 <sock_poll+23/28>
Trace; c0151f0c <do_select+1c8/204>
Trace; c0151bdc <__pollwait+0/c0>
Trace; c015238a <sys_select+41a/5fc>
Trace; c0107143 <system_call+33/38>
Proc;  gpm

>>EIP; f7a4bf14 <END_OF_CODE+2c0e15/????>   <=====

Trace; c01228b8 <schedule_timeout+90/b0>
Trace; c012281c <process_timeout+0/c>
Trace; c0151f0c <do_select+1c8/204>
Trace; c0151bdc <__pollwait+0/c0>
Trace; c015238a <sys_select+41a/5fc>
Trace; c0107143 <system_call+33/38>
Proc;  inetd

>>EIP; 7fffffff Before first symbol   <=====

Trace; c012283c <schedule_timeout+14/b0>
Trace; c01fe467 <sock_poll+23/28>
Trace; c0151f0c <do_select+1c8/204>
Trace; c0151bdc <__pollwait+0/c0>
Trace; c015238a <sys_select+41a/5fc>
Trace; c0107143 <system_call+33/38>
Proc;  nfsd

>>EIP; c23ebf54 <[mii].rodata.end+62755/bf7801>   <=====

Trace; c01228b8 <schedule_timeout+90/b0>
Trace; c012281c <process_timeout+0/c>
Trace; c3228e99 <[iptable_filter].data.end+2a779a/3be901>
Trace; f7a9029c <END_OF_CODE+30519d/????>
Trace; f7a9a520 <END_OF_CODE+30f421/????>
Trace; c01056d0 <arch_kernel_thread+28/38>
Proc;  nfsd

>>EIP; c23c9f54 <[mii].rodata.end+40755/bf7801>   <=====

Trace; c01228b8 <schedule_timeout+90/b0>
Trace; c012281c <process_timeout+0/c>
Trace; c3228e99 <[iptable_filter].data.end+2a779a/3be901>
Trace; f7a9029c <END_OF_CODE+30519d/????>
Trace; c01056d0 <arch_kernel_thread+28/38>
Proc;  nfsd

>>EIP; c23a9f54 <[mii].rodata.end+20755/bf7801>   <=====

Trace; c01228b8 <schedule_timeout+90/b0>
Trace; c012281c <process_timeout+0/c>
Trace; c3228e99 <[iptable_filter].data.end+2a779a/3be901>
Trace; f7a9029c <END_OF_CODE+30519d/????>
Trace; c01056d0 <arch_kernel_thread+28/38>
Proc;  nfsd

>>EIP; c238ff54 <[mii].rodata.end+6755/bf7801>   <=====

Trace; c01228b8 <schedule_timeout+90/b0>
Trace; c012281c <process_timeout+0/c>
Trace; c3228e99 <[iptable_filter].data.end+2a779a/3be901>
Trace; f7a9029c <END_OF_CODE+30519d/????>
Trace; c01056d0 <arch_kernel_thread+28/38>
Proc;  nfsd

>>EIP; c236ff54 <[qla2300].bss.end+39655/41701>   <=====

Trace; c01228b8 <schedule_timeout+90/b0>
Trace; c012281c <process_timeout+0/c>
Trace; c3228e99 <[iptable_filter].data.end+2a779a/3be901>
Trace; f7a9029c <END_OF_CODE+30519d/????>
Trace; c01056d0 <arch_kernel_thread+28/38>
Proc;  nfsd

>>EIP; c234df54 <[qla2300].bss.end+17655/41701>   <=====

Trace; c01228b8 <schedule_timeout+90/b0>
Trace; c012281c <process_timeout+0/c>
Trace; c3228e99 <[iptable_filter].data.end+2a779a/3be901>
Trace; f7a9029c <END_OF_CODE+30519d/????>
Trace; c01056d0 <arch_kernel_thread+28/38>
Proc;  nfsd

>>EIP; c232df54 <[qla2300]fw2300tp_code01+126e4/1804e>   <=====

Trace; c01228b8 <schedule_timeout+90/b0>
Trace; c012281c <process_timeout+0/c>
Trace; c3228e99 <[iptable_filter].data.end+2a779a/3be901>
Trace; f7a9029c <END_OF_CODE+30519d/????>
Trace; c01056d0 <arch_kernel_thread+28/38>
Proc;  nfsd

>>EIP; c2313f54 <[qla2300]qla2x00_cfg_build_path_tree+1dc/2a4>   <=====

Trace; c01228b8 <schedule_timeout+90/b0>
Trace; c012281c <process_timeout+0/c>
Trace; c3228e99 <[iptable_filter].data.end+2a779a/3be901>
Trace; f7a9029c <END_OF_CODE+30519d/????>
Trace; f7a9a520 <END_OF_CODE+30f421/????>
Trace; c01056d0 <arch_kernel_thread+28/38>
Proc;  lockd

>>EIP; 7fffffff Before first symbol   <=====

Trace; c012283c <schedule_timeout+14/b0>
Trace; c3225244 <[iptable_filter].data.end+2a3b45/3be901>
Trace; c3228e99 <[iptable_filter].data.end+2a779a/3be901>
Trace; f7ab1cb3 <END_OF_CODE+326bb4/????>
Trace; c01056d0 <arch_kernel_thread+28/38>
Proc;  rpciod

>>EIP; 00000001 Before first symbol   <=====

Trace; c3225410 <[iptable_filter].data.end+2a3d11/3be901>
Trace; c322f74c <[iptable_filter].data.end+2ae04d/3be901>
Trace; c322f74c <[iptable_filter].data.end+2ae04d/3be901>
Trace; c322f744 <[iptable_filter].data.end+2ae045/3be901>
Trace; c322f744 <[iptable_filter].data.end+2ae045/3be901>
Trace; c01056d0 <arch_kernel_thread+28/38>
Trace; c322f74c <[iptable_filter].data.end+2ae04d/3be901>
Proc;  rpc.mountd

>>EIP; 7fffffff Before first symbol   <=====

Trace; c012283c <schedule_timeout+14/b0>
Trace; c015267a <do_poll+86/dc>
Trace; c01526b0 <do_poll+bc/dc>
Trace; c0152884 <sys_poll+1b4/2a3>
Trace; c0151bdc <__pollwait+0/c0>
Trace; c0107143 <system_call+33/38>
Proc;  sshd

>>EIP; 7fffffff Before first symbol   <=====

Trace; c012283c <schedule_timeout+14/b0>
Trace; c01fe467 <sock_poll+23/28>
Trace; c0151f0c <do_select+1c8/204>
Trace; c0151bdc <__pollwait+0/c0>
Trace; c015238a <sys_select+41a/5fc>
Trace; c0107143 <system_call+33/38>
Proc;  xfs

>>EIP; c2287f14 <[lvm-mod].bss.end+fd355/175441>   <=====

Trace; c01228b8 <schedule_timeout+90/b0>
Trace; c012281c <process_timeout+0/c>
Trace; c0151f0c <do_select+1c8/204>
Trace; c0151bdc <__pollwait+0/c0>
Trace; c015238a <sys_select+41a/5fc>
Trace; c011dd7d <sys_gettimeofday+25/54>
Trace; c0107143 <system_call+33/38>
Proc;  atd

>>EIP; f7a31f60 <END_OF_CODE+2a6e61/????>   <=====

Trace; c01228b8 <schedule_timeout+90/b0>
Trace; c012281c <process_timeout+0/c>
Trace; c01229ec <sys_nanosleep+104/209>
Trace; c0107143 <system_call+33/38>
Proc;  cron

>>EIP; c2281f60 <[lvm-mod].bss.end+f73a1/175441>   <=====

Trace; c01228b8 <schedule_timeout+90/b0>
Trace; c012281c <process_timeout+0/c>
Trace; c01229ec <sys_nanosleep+104/209>
Trace; c0107143 <system_call+33/38>
Proc;  apache

>>EIP; c227df14 <[lvm-mod].bss.end+f3355/175441>   <=====

Trace; c01228b8 <schedule_timeout+90/b0>
Trace; c012281c <process_timeout+0/c>
Trace; c0151f0c <do_select+1c8/204>
Trace; c0151bdc <__pollwait+0/c0>
Trace; c015238a <sys_select+41a/5fc>
Trace; c0107143 <system_call+33/38>
Proc;  getty

>>EIP; 7fffffff Before first symbol   <=====

Trace; c012283c <schedule_timeout+14/b0>
Trace; c0196084 <read_chan+3b4/7e8>
Trace; c01960cf <read_chan+3ff/7e8>
Trace; c0191c3a <tty_read+da/128>
Trace; c0140be7 <sys_read+8f/100>
Trace; c0107143 <system_call+33/38>
Proc;  getty

>>EIP; 7fffffff Before first symbol   <=====

Trace; c012283c <schedule_timeout+14/b0>
Trace; c0196084 <read_chan+3b4/7e8>
Trace; c01960cf <read_chan+3ff/7e8>
Trace; c0191c3a <tty_read+da/128>
Trace; c0140be7 <sys_read+8f/100>
Trace; c0107143 <system_call+33/38>
Proc;  getty

>>EIP; 7fffffff Before first symbol   <=====

Trace; c012283c <schedule_timeout+14/b0>
Trace; c0196084 <read_chan+3b4/7e8>
Trace; c01960cf <read_chan+3ff/7e8>
Trace; c0191c3a <tty_read+da/128>
Trace; c0140be7 <sys_read+8f/100>
Trace; c0107143 <system_call+33/38>
Proc;  getty

>>EIP; 7fffffff Before first symbol   <=====

Trace; c012283c <schedule_timeout+14/b0>
Trace; c0196084 <read_chan+3b4/7e8>
Trace; c01960cf <read_chan+3ff/7e8>
Trace; c0191c3a <tty_read+da/128>
Trace; c0140be7 <sys_read+8f/100>
Trace; c0107143 <system_call+33/38>
Proc;  getty

>>EIP; 7fffffff Before first symbol   <=====

Trace; c012283c <schedule_timeout+14/b0>
Trace; c0196084 <read_chan+3b4/7e8>
Trace; c01960cf <read_chan+3ff/7e8>
Trace; c0191c3a <tty_read+da/128>
Trace; c0140be7 <sys_read+8f/100>
Trace; c0107143 <system_call+33/38>
Proc;  getty

>>EIP; 7fffffff Before first symbol   <=====

Trace; c012283c <schedule_timeout+14/b0>
Trace; c0196084 <read_chan+3b4/7e8>
Trace; c01960cf <read_chan+3ff/7e8>
Trace; c0191c3a <tty_read+da/128>
Trace; c0140be7 <sys_read+8f/100>
Trace; c0107143 <system_call+33/38>
Proc;  kjournald

>>EIP; f7a9d294 <END_OF_CODE+312195/????>   <=====

Trace; c0105da8 <__down+5c/bc>
Trace; c0105f40 <__down_failed+8/c>
Trace; c017c0b4 <.text.lock.commit+5/141>
Trace; c0115f53 <do_schedule+36f/3c4>
Trace; c0115f5f <do_schedule+37b/3c4>
Trace; c017de26 <kjournald+176/250>
Trace; c017dca0 <commit_timeout+0/c>
Trace; c01056d0 <arch_kernel_thread+28/38>
Proc;  screen

>>EIP; 7fffffff Before first symbol   <=====

Trace; c012283c <schedule_timeout+14/b0>
Trace; c0151f0c <do_select+1c8/204>
Trace; c0151bdc <__pollwait+0/c0>
Trace; c015238a <sys_select+41a/5fc>
Trace; c0107143 <system_call+33/38>
Proc;  rlbench.sh

>>EIP; 00000000 Before first symbol

Trace; c011d6d6 <sys_wait4+386/3bc>
Trace; c0107143 <system_call+33/38>
Proc;  rlbench

>>EIP; c93dff60 <[sunrpc].bss.end+608f44d/343f34ed>   <=====

Trace; c01228b8 <schedule_timeout+90/b0>
Trace; c012281c <process_timeout+0/c>
Trace; c01229ec <sys_nanosleep+104/209>
Trace; c0107143 <system_call+33/38>
Proc;  tee

>>EIP; c936e000 <[sunrpc].bss.end+601d4ed/343f34ed>   <=====

Trace; c014afeb <pipe_wait+5b/84>
Trace; c014b0c4 <pipe_read+b0/1fc>
Trace; c0140be7 <sys_read+8f/100>
Trace; c0107143 <system_call+33/38>
Proc;  screen

>>EIP; 7fffffff Before first symbol   <=====

Trace; c012283c <schedule_timeout+14/b0>
Trace; c0151f0c <do_select+1c8/204>
Trace; c0151bdc <__pollwait+0/c0>
Trace; c015238a <sys_select+41a/5fc>
Trace; c0107143 <system_call+33/38>
Proc;  bash

>>EIP; 00000000 Before first symbol

Trace; c011d6d6 <sys_wait4+386/3bc>
Trace; c0107143 <system_call+33/38>
Proc;  screen

>>EIP; 7fffffff Before first symbol   <=====

Trace; c012283c <schedule_timeout+14/b0>
Trace; c0151f0c <do_select+1c8/204>
Trace; c0151bdc <__pollwait+0/c0>
Trace; c015238a <sys_select+41a/5fc>
Trace; c0107143 <system_call+33/38>
Proc;  bash

>>EIP; 00000000 Before first symbol

Trace; c011d6d6 <sys_wait4+386/3bc>
Trace; c0107143 <system_call+33/38>
Proc;  bonnie++

>>EIP; f7a9d294 <END_OF_CODE+312195/????>   <=====

Trace; c0105da8 <__down+5c/bc>
Trace; c0105f40 <__down_failed+8/c>
Trace; c017adfa <.text.lock.transaction+5/26b>
Trace; c0178ffd <journal_start+95/c4>
Trace; c0173d48 <ext3_dirty_inode+74/108>
Trace; c0156cbe <__mark_inode_dirty+32/b0>
Trace; c01584aa <update_atime+4e/54>
Trace; c012e882 <__do_generic_file_read+4e2/4f0>
Trace; c012ec64 <generic_file_read+90/1d8>
Trace; c012eac0 <file_read_actor+0/114>
Trace; c016f007 <ext3_file_write+23/bc>
Trace; c0140be7 <sys_read+8f/100>
Trace; c0107143 <system_call+33/38>
Proc;  bonnie++

>>EIP; c17fe560 <_end+145faec/1dad58c>   <=====
Proc;  bonnie++

>>EIP; d7933f30 <[sunrpc].bss.end+145e341d/343f34ed>   <=====

Trace; c01228b8 <schedule_timeout+90/b0>
Trace; c012281c <process_timeout+0/c>
Trace; c01526b0 <do_poll+bc/dc>
Trace; c0152884 <sys_poll+1b4/2a3>
Trace; c0151bdc <__pollwait+0/c0>
Trace; c0107143 <system_call+33/38>
Proc;  bonnie++

>>EIP; e157ff30 <[sunrpc].bss.end+1e22f41d/343f34ed>   <=====

Trace; c01228b8 <schedule_timeout+90/b0>
Trace; c012281c <process_timeout+0/c>
Trace; c01526b0 <do_poll+bc/dc>
Trace; c0152884 <sys_poll+1b4/2a3>
Trace; c0151bdc <__pollwait+0/c0>
Trace; c0107143 <system_call+33/38>
Proc;  screen

>>EIP; 7fffffff Before first symbol   <=====

Trace; c012283c <schedule_timeout+14/b0>
Trace; c0151f0c <do_select+1c8/204>
Trace; c0151bdc <__pollwait+0/c0>
Trace; c015238a <sys_select+41a/5fc>
Trace; c0107143 <system_call+33/38>
Proc;  bash

>>EIP; 00000000 Before first symbol

Trace; c011d6d6 <sys_wait4+386/3bc>
Trace; c0107143 <system_call+33/38>
Proc;  bonnie++

>>EIP; f7a9d294 <END_OF_CODE+312195/????>   <=====

Trace; c0105da8 <__down+5c/bc>
Trace; c0105f40 <__down_failed+8/c>
Trace; c017adfa <.text.lock.transaction+5/26b>
Trace; c0178ffd <journal_start+95/c4>
Trace; c0173d48 <ext3_dirty_inode+74/108>
Trace; c0156cbe <__mark_inode_dirty+32/b0>
Trace; c01584aa <update_atime+4e/54>
Trace; c012e882 <__do_generic_file_read+4e2/4f0>
Trace; c012ec64 <generic_file_read+90/1d8>
Trace; c012eac0 <file_read_actor+0/114>
Trace; c0140be7 <sys_read+8f/100>
Trace; c0107143 <system_call+33/38>
Proc;  bonnie++

>>EIP; df49ff30 <[sunrpc].bss.end+1c14f41d/343f34ed>   <=====

Trace; c01228b8 <schedule_timeout+90/b0>
Trace; c012281c <process_timeout+0/c>
Trace; c01526b0 <do_poll+bc/dc>
Trace; c0152884 <sys_poll+1b4/2a3>
Trace; c0151bdc <__pollwait+0/c0>
Trace; c0107143 <system_call+33/38>
Proc;  lpd

>>EIP; 7fffffff Before first symbol   <=====

Trace; c012283c <schedule_timeout+14/b0>
Trace; c01fe467 <sock_poll+23/28>
Trace; c0151f0c <do_select+1c8/204>
Trace; c0151bdc <__pollwait+0/c0>
Trace; c015238a <sys_select+41a/5fc>
Trace; c01ff657 <sys_socketcall+9b/1fc>
Trace; c0107143 <system_call+33/38>
Proc;  apache

>>EIP; 7fffffff Before first symbol   <=====

Trace; c012283c <schedule_timeout+14/b0>
Trace; c021f5ed <wait_for_connect+105/1d8>
Trace; c021f77a <tcp_accept+ba/234>
Trace; c0239a44 <inet_accept+2c/1b8>
Trace; c01fec56 <sys_accept+66/fc>
Trace; c0113d58 <do_page_fault+0/5fb>
Trace; c012c729 <unmap_fixup+165/174>
Trace; c0124543 <sys_rt_sigaction+67/80>
Trace; c01ff66c <sys_socketcall+b0/1fc>
Trace; c0107143 <system_call+33/38>
Proc;  apache

>>EIP; 7fffffff Before first symbol   <=====

Trace; c012283c <schedule_timeout+14/b0>
Trace; c021f5ed <wait_for_connect+105/1d8>
Trace; c021f77a <tcp_accept+ba/234>
Trace; c0239a44 <inet_accept+2c/1b8>
Trace; c01fec56 <sys_accept+66/fc>
Trace; c0113d58 <do_page_fault+0/5fb>
Trace; c012c729 <unmap_fixup+165/174>
Trace; c0124543 <sys_rt_sigaction+67/80>
Trace; c01ff66c <sys_socketcall+b0/1fc>
Trace; c0107143 <system_call+33/38>
Proc;  apache

>>EIP; 7fffffff Before first symbol   <=====

Trace; c012283c <schedule_timeout+14/b0>
Trace; c021f5ed <wait_for_connect+105/1d8>
Trace; c021f77a <tcp_accept+ba/234>
Trace; c0239a44 <inet_accept+2c/1b8>
Trace; c01fec56 <sys_accept+66/fc>
Trace; c0113d58 <do_page_fault+0/5fb>
Trace; c012c729 <unmap_fixup+165/174>
Trace; c0124543 <sys_rt_sigaction+67/80>
Trace; c01ff66c <sys_socketcall+b0/1fc>
Trace; c0107143 <system_call+33/38>
Proc;  apache

>>EIP; 7fffffff Before first symbol   <=====

Trace; c012283c <schedule_timeout+14/b0>
Trace; c021f5ed <wait_for_connect+105/1d8>
Trace; c021f77a <tcp_accept+ba/234>
Trace; c0239a44 <inet_accept+2c/1b8>
Trace; c01fec56 <sys_accept+66/fc>
Trace; c0113d58 <do_page_fault+0/5fb>
Trace; c012c729 <unmap_fixup+165/174>
Trace; c0124543 <sys_rt_sigaction+67/80>
Trace; c01ff66c <sys_socketcall+b0/1fc>
Trace; c0107143 <system_call+33/38>
Proc;  apache

>>EIP; 7fffffff Before first symbol   <=====

Trace; c012283c <schedule_timeout+14/b0>
Trace; c021f5ed <wait_for_connect+105/1d8>
Trace; c021f77a <tcp_accept+ba/234>
Trace; c0239a44 <inet_accept+2c/1b8>
Trace; c01fec56 <sys_accept+66/fc>
Trace; c0113d58 <do_page_fault+0/5fb>
Trace; c012c729 <unmap_fixup+165/174>
Trace; c0124543 <sys_rt_sigaction+67/80>
Trace; c01ff66c <sys_socketcall+b0/1fc>
Trace; c0107143 <system_call+33/38>
Proc;  sshd

>>EIP; 7fffffff Before first symbol   <=====

Trace; c012283c <schedule_timeout+14/b0>
Trace; c0151f0c <do_select+1c8/204>
Trace; c0151bdc <__pollwait+0/c0>
Trace; c015238a <sys_select+41a/5fc>
Trace; c0107143 <system_call+33/38>
Proc;  bash

>>EIP; 7fffffff Before first symbol   <=====

Trace; c012283c <schedule_timeout+14/b0>
Trace; c0196084 <read_chan+3b4/7e8>
Trace; c01960cf <read_chan+3ff/7e8>
Trace; c0191c3a <tty_read+da/128>
Trace; c0140be7 <sys_read+8f/100>
Trace; c0107143 <system_call+33/38>
Proc;  cron

>>EIP; e291a000 <[sunrpc].bss.end+1f5c94ed/343f34ed>   <=====

Trace; c014afeb <pipe_wait+5b/84>
Trace; c014b0c4 <pipe_read+b0/1fc>
Trace; c0140be7 <sys_read+8f/100>
Trace; c0107143 <system_call+33/38>
Proc;  sh

>>EIP; 00000000 Before first symbol

Trace; c011d6d6 <sys_wait4+386/3bc>
Trace; c0107143 <system_call+33/38>
Proc;  run-parts

>>EIP; 7fffffff Before first symbol   <=====

Trace; c012283c <schedule_timeout+14/b0>
Trace; c0151f0c <do_select+1c8/204>
Trace; c0151bdc <__pollwait+0/c0>
Trace; c015238a <sys_select+41a/5fc>
Trace; c0107143 <system_call+33/38>
Proc;  find

>>EIP; 00000000 Before first symbol

Trace; c011d6d6 <sys_wait4+386/3bc>
Trace; c0107143 <system_call+33/38>
Proc;  updatedb

>>EIP; 00000000 Before first symbol

Trace; c011d6d6 <sys_wait4+386/3bc>
Trace; c0107143 <system_call+33/38>
Proc;  updatedb

>>EIP; 00000000 Before first symbol

Trace; c011d6d6 <sys_wait4+386/3bc>
Trace; c0107143 <system_call+33/38>
Proc;  find

>>EIP; f735cac0 <[sunrpc].bss.end+3400bfad/343f34ed>   <=====

Trace; c0141e75 <__wait_on_buffer+69/8c>
Trace; c01715e2 <ext3_bread+56/7c>
Trace; c016eca6 <ext3_readdir+96/390>
Trace; c01493d4 <cp_new_stat64+134/150>
Trace; c01510f4 <vfs_readdir+94/e0>
Trace; c01516e0 <filldir64+0/104>
Trace; c015182e <sys_getdents64+4a/96>
Trace; c01516e0 <filldir64+0/104>
Trace; c015087b <sys_fcntl64+7f/88>
Trace; c0107143 <system_call+33/38>
Proc;  sort

>>EIP; 0000000c Before first symbol   <=====
Proc;  frcode

>>EIP; e20ea000 <[sunrpc].bss.end+1ed994ed/343f34ed>   <=====

Trace; c014afeb <pipe_wait+5b/84>
Trace; c014b0c4 <pipe_read+b0/1fc>
Trace; c0140be7 <sys_read+8f/100>
Trace; c0107143 <system_call+33/38>


1 warning issued.  Results may not be reliable.

^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: 2.4.23pre6aa3 - possible ext3 deadlock
  2003-11-27  3:21     ` 2.4.23pre6aa3 - possible ext3 deadlock Jamie Clark
@ 2003-11-27 12:56       ` Jan Kara
  0 siblings, 0 replies; 15+ messages in thread
From: Jan Kara @ 2003-11-27 12:56 UTC (permalink / raw)
  To: Jamie Clark; +Cc: Andrea Arcangeli, linux-kernel

  Hi,

> Followup to last test with 2.4.23pre6aa3. same config as before (ext3 
> /ql2300 / bonnie++) I forgot to mention that I have quotas enabled on 
> the partition under test.
> 
> There have been a few 2.4 / quota / ext3 deadlock-type  mails on the 
> list recently so I'm sot sure if this is related. I can see an atime 
> update in the call trace of one of the bonnies but no dquot_ calls 
> anywhere.  I haven't applied Jan's patch as it came out before I could 
> reliably reproduce this problem.
  Yes, actually there's still a deadlock there for ext3+quotas (patch is
already written but not yet included in mainline I think) but as I've
looked through the trace it doesn't seem to be the case. About a week
ago I saw a report on deadlock of 2.4.22 with just plain ext3 without
quotas - so can you try to reproduce the problem with quotas not
compiled into the kernel (or just turning them off should be enough)?

> In the meantime I'll restart testing with noatime - which is how I'd 
> usually mount such a fs anyhow.
> 
> The box seems to deadlock after 3-6 days of filesystem exercise. I have 
> repeated this 3 times. The last run was 6 days and and then I see all 
> disk I/O has ceased. The system remains pingable and sshd connects at 
> the TCP layer but nothing happens after that. The root filesystem is 
> ext3 but is a completely separate device (motherboard scsi) to the 
> filesystems running bonnie (external FC RAID). It seems that all 
> filesystem activity ceases.
> 
> I have run the alt-sysrq-t ouput through ksymoops and attached.  Hope 
> this is useful to someone who knows.

						Thanks for report
								Honza

^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: 2.4.23pre6aa3 scsi oops
  2003-11-04 11:52   ` Jamie Clark
  2003-11-04 12:48     ` Andrea Arcangeli
  2003-11-27  3:21     ` 2.4.23pre6aa3 - possible ext3 deadlock Jamie Clark
@ 2003-12-06  1:05     ` Andrea Arcangeli
  2003-12-06  1:39       ` Jamie Clark
  2003-12-11  2:33       ` 2.4.23aa1 ext3 oops Jamie Clark
  2 siblings, 2 replies; 15+ messages in thread
From: Andrea Arcangeli @ 2003-12-06  1:05 UTC (permalink / raw)
  To: Jamie Clark; +Cc: linux-kernel

On Tue, Nov 04, 2003 at 07:52:40PM +0800, Jamie Clark wrote:
> I made the quick fix (disabling rq_mergeable) and started the load test.
> Will let it run for a week or so.

does your later recent email means it deadlocked again even with this
disabled?

Could you try again with 2.4.23aa1 again just in case?

> FYI an observation from my last test: the read latency seems to be much
> improved and more consistent under this kernel (2.4.23pre6aa3, before
> the oops and before this fix).  The maximum latency seemed steady over
> the whole test without any of the longish pauses that showed up under
> 2.4.19. Quite a difference.

nice to hear! thanks.

^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: 2.4.23pre6aa3 scsi oops
  2003-12-06  1:05     ` 2.4.23pre6aa3 scsi oops Andrea Arcangeli
@ 2003-12-06  1:39       ` Jamie Clark
  2003-12-11  2:33       ` 2.4.23aa1 ext3 oops Jamie Clark
  1 sibling, 0 replies; 15+ messages in thread
From: Jamie Clark @ 2003-12-06  1:39 UTC (permalink / raw)
  To: Andrea Arcangeli; +Cc: linux-kernel

Andrea Arcangeli wrote:

>On Tue, Nov 04, 2003 at 07:52:40PM +0800, Jamie Clark wrote:
>  
>
>>I made the quick fix (disabling rq_mergeable) and started the load test.
>>Will let it run for a week or so.
>>    
>>
>does your later recent email means it deadlocked again even with this
>disabled?
>  
>
Yes, although it appeared to be a different deadlock: ext3 perhaps.

>Could you try again with 2.4.23aa1 again just in case?
>  
>
I started another test on the same kernel with quotas disabled
as Jan suggested but I had to abort the test for unrelated reason
(needed spare hw). The interesting case for me is WITH quotas.

Will try 2.4.23aa1 as soon as I can re-assemble the test unit.

-Jamie



^ permalink raw reply	[flat|nested] 15+ messages in thread

* 2.4.23aa1 ext3 oops
  2003-12-06  1:05     ` 2.4.23pre6aa3 scsi oops Andrea Arcangeli
  2003-12-06  1:39       ` Jamie Clark
@ 2003-12-11  2:33       ` Jamie Clark
  2003-12-16 15:15         ` Marcelo Tosatti
  2003-12-17 11:42         ` David Woodhouse
  1 sibling, 2 replies; 15+ messages in thread
From: Jamie Clark @ 2003-12-11  2:33 UTC (permalink / raw)
  To: Andrea Arcangeli; +Cc: linux-kernel

[-- Attachment #1: Type: text/plain, Size: 1920 bytes --]

OK, no deadlock yet with 2.4.23aa1 however it oopsed under 
ext3_file_write() in __mark_inode_dirty().

Just to recap: this test is dual PIII, running several bonnie++ loads on 
an ext3+noatime+quota filesystem
mounted off

 From the oops the fault happens on the last instruction of:

        movl $0,8(%ebx)
        movl $0,4(%edx)
        movl 100(%edi),%eax 
        movl %edx,4(%eax)    <-- here

which appears to be this code in inode.c  [line 221+]

                if (!(inode->i_state & (I_LOCK|I_FREEING|I_CLEAR)) &&
                    !list_empty(&inode->i_hash)) {
                        list_del(&inode->i_list);
                        list_add(&inode->i_list, &sb->s_dirty);

After a quick browse of the assembler output the zeroing would appear to 
be part of the list_del inline,
and edi seems to equate to &sb. If I have read that correctly then the 
oops happens at the beginning of
the list_add() inline and eax is the head of the s_dirty list - pointing 
into oblivion.

__mark_inode_dirty() does not appear to take sb_lock before adding to 
the s_dirty list. Could that
be the culprit?   I'm completely unfamiliar with linux kernel so I might 
be way off here.

-Jamie

Andrea Arcangeli wrote:

>On Tue, Nov 04, 2003 at 07:52:40PM +0800, Jamie Clark wrote:
>  
>
>>I made the quick fix (disabling rq_mergeable) and started the load test.
>>Will let it run for a week or so.
>>    
>>
>
>does your later recent email means it deadlocked again even with this
>disabled?
>
>Could you try again with 2.4.23aa1 again just in case?
>
>  
>
>>FYI an observation from my last test: the read latency seems to be much
>>improved and more consistent under this kernel (2.4.23pre6aa3, before
>>the oops and before this fix).  The maximum latency seemed steady over
>>the whole test without any of the longish pauses that showed up under
>>2.4.19. Quite a difference.
>>    
>>
>
>nice to hear! thanks.
>  
>

[-- Attachment #2: ksymoops --]
[-- Type: text/plain, Size: 2845 bytes --]

ksymoops 2.4.5 on i686 2.4.23aa1.  Options used
     -V (default)
     -k /proc/ksyms (default)
     -l /proc/modules (default)
     -o /lib/modules/2.4.23aa1/ (default)
     -m /boot/System.map-2.4.23aa1 (default)

Warning: You did not tell me where to find symbol information.  I will
assume that the log matches the kernel and modules that are running
right now and I'll use the default options above for symbol resolution.
If the current kernel and/or modules do not match the log, you can get
more accurate output by telling me the kernel version and where to find
map, modules, ksyms etc.  ksymoops -h explains the options.

Unable to handle kernel paging request at virtual address d7ffbc0c
c0156eaf
*pde = 17c001e3
Oops: 0002 2.4.23aa1 #2 SMP Sat Dec 6 10:58:56 SGT 2003
CPU:    0
EIP:    0010:[<c0156eaf>]    Not tainted
Using defaults from ksymoops -t elf32-i386 -a i386
EFLAGS: 00010287
eax: d7ffbc08   ebx: e2ae6a00   ecx: c02a11f8   edx: e2ae6a08
esi: 00000001   edi: f769e000   ebp: 00002000   esp: f2799ef8
ds: 0018   es: 0018   ss: 0018
Process bonnie++ (pid: 330, stackpage=f2799000)
Stack: e2ae6a00 e2ae6a00 f5110420 c0130973 e2ae6a00 00000001 e2ae6a6c e2ae6a00
       f5110420 00002000 e2ae6a00 ffffffeb f785d660 c014bf57 e2ae6a00 00000000
       00000000 e2ae6a00 e2ae6ac4 00000000 00000000 00000000 f5110420 c013108f
Call Trace:         [<c0130973>] (40) [<c014bf57>] (40) [<c013108f>] (36) [<c016f377>] (36) [<c0140e57>] (36) [<c0107133>] (60)
Code: 89 50 04 89 43 08 8d 47 64 89 42 04 89 57 64 c6 05 10 12 2a


>>EIP; c0156eaf <__mark_inode_dirty+93/b0>   <=====

>>eax; d7ffbc08 <END_OF_CODE+1462abd5/????>
>>ebx; e2ae6a00 <END_OF_CODE+1f1159cd/????>
>>ecx; c02a11f8 <inode_in_use+0/8>
>>edx; e2ae6a08 <END_OF_CODE+1f1159d5/????>
>>edi; f769e000 <END_OF_CODE+33cccfcd/????>
>>ebp; 00002000 Before first symbol
>>esp; f2799ef8 <END_OF_CODE+2edc8ec5/????>

Trace; c0130973 <generic_file_write_nolock+d3/4c8>
Trace; c014bf57 <permission+7b/84>
Trace; c013108f <generic_file_write+13f/158>
Trace; c016f377 <ext3_file_write+23/bc>
Trace; c0140e57 <sys_write+8f/100>
Trace; c0107133 <system_call+33/38>

Code;  c0156eaf <__mark_inode_dirty+93/b0>
00000000 <_EIP>:
Code;  c0156eaf <__mark_inode_dirty+93/b0>   <=====
   0:   89 50 04                  mov    %edx,0x4(%eax)   <=====
Code;  c0156eb2 <__mark_inode_dirty+96/b0>
   3:   89 43 08                  mov    %eax,0x8(%ebx)
Code;  c0156eb5 <__mark_inode_dirty+99/b0>
   6:   8d 47 64                  lea    0x64(%edi),%eax
Code;  c0156eb8 <__mark_inode_dirty+9c/b0>
   9:   89 42 04                  mov    %eax,0x4(%edx)
Code;  c0156ebb <__mark_inode_dirty+9f/b0>
   c:   89 57 64                  mov    %edx,0x64(%edi)
Code;  c0156ebe <__mark_inode_dirty+a2/b0>
   f:   c6 05 10 12 2a 00 00      movb   $0x0,0x2a1210


1 warning issued.  Results may not be reliable.

^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: 2.4.23aa1 ext3 oops
  2003-12-11  2:33       ` 2.4.23aa1 ext3 oops Jamie Clark
@ 2003-12-16 15:15         ` Marcelo Tosatti
  2003-12-16 15:55           ` Jamie Clark
  2003-12-17 11:42         ` David Woodhouse
  1 sibling, 1 reply; 15+ messages in thread
From: Marcelo Tosatti @ 2003-12-16 15:15 UTC (permalink / raw)
  To: Jamie Clark; +Cc: Andrea Arcangeli, linux-kernel


Jamie, 

Did you try the patch I suggested for you to revert  ?

On Thu, 11 Dec 2003, Jamie Clark wrote:

> OK, no deadlock yet with 2.4.23aa1 however it oopsed under 
> ext3_file_write() in __mark_inode_dirty().
> 
> Just to recap: this test is dual PIII, running several bonnie++ loads on 
> an ext3+noatime+quota filesystem
> mounted off
> 
>  From the oops the fault happens on the last instruction of:
> 
>         movl $0,8(%ebx)
>         movl $0,4(%edx)
>         movl 100(%edi),%eax 
>         movl %edx,4(%eax)    <-- here
> 
> which appears to be this code in inode.c  [line 221+]
> 
>                 if (!(inode->i_state & (I_LOCK|I_FREEING|I_CLEAR)) &&
>                     !list_empty(&inode->i_hash)) {
>                         list_del(&inode->i_list);
>                         list_add(&inode->i_list, &sb->s_dirty);
> 
> After a quick browse of the assembler output the zeroing would appear to 
> be part of the list_del inline, and edi seems to equate to &sb. If I 
> have read that correctly then the 
> oops happens at the beginning of
> the list_add() inline and eax is the head of the s_dirty list - pointing 
> into oblivion.
> 
> __mark_inode_dirty() does not appear to take sb_lock before adding to 
> the s_dirty list. Could that
> be the culprit?   I'm completely unfamiliar with linux kernel so I might 
> be way off here.
> 
> -Jamie
> 
> Andrea Arcangeli wrote:
> 
> >On Tue, Nov 04, 2003 at 07:52:40PM +0800, Jamie Clark wrote:
> >  
> >
> >>I made the quick fix (disabling rq_mergeable) and started the load test.
> >>Will let it run for a week or so.
> >>    
> >>
> >
> >does your later recent email means it deadlocked again even with this
> >disabled?
> >
> >Could you try again with 2.4.23aa1 again just in case?
> >
> >  
> >
> >>FYI an observation from my last test: the read latency seems to be much
> >>improved and more consistent under this kernel (2.4.23pre6aa3, before
> >>the oops and before this fix).  The maximum latency seemed steady over
> >>the whole test without any of the longish pauses that showed up under
> >>2.4.19. Quite a difference.
> >>    
> >>
> >
> >nice to hear! thanks.
> >  
> >
> 


^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: 2.4.23aa1 ext3 oops
  2003-12-16 15:15         ` Marcelo Tosatti
@ 2003-12-16 15:55           ` Jamie Clark
  0 siblings, 0 replies; 15+ messages in thread
From: Jamie Clark @ 2003-12-16 15:55 UTC (permalink / raw)
  To: Marcelo Tosatti; +Cc: Jamie Clark, Andrea Arcangeli, linux-kernel

Hi Marcelo,

Sorry - I had started another test run after the previous crash (to 
collect more forensics) and have been waiting for the oops. The last one 
took 4 or 5 days. Just now I see I have another crash in the same place 
so I will apply the reversions you suggested and start again.

My apologies - this is slow work. I'm attempting to find a new kernel to 
keep our busy fileserver happy and I've had a spell of bad luck for the 
past month or two. I'll be happy when I get two weeks of uptime with a 
few bonnies running.


Marcelo Tosatti wrote:

>Jamie, 
>
>Did you try the patch I suggested for you to revert  ?
>
>On Thu, 11 Dec 2003, Jamie Clark wrote:
>
>  
>
>>OK, no deadlock yet with 2.4.23aa1 however it oopsed under 
>>ext3_file_write() in __mark_inode_dirty().
>>
>>Just to recap: this test is dual PIII, running several bonnie++ loads on 
>>an ext3+noatime+quota filesystem
>>mounted off
>>
>> From the oops the fault happens on the last instruction of:
>>
>>        movl $0,8(%ebx)
>>        movl $0,4(%edx)
>>        movl 100(%edi),%eax 
>>        movl %edx,4(%eax)    <-- here
>>
>>which appears to be this code in inode.c  [line 221+]
>>
>>                if (!(inode->i_state & (I_LOCK|I_FREEING|I_CLEAR)) &&
>>                    !list_empty(&inode->i_hash)) {
>>                        list_del(&inode->i_list);
>>                        list_add(&inode->i_list, &sb->s_dirty);
>>
>>After a quick browse of the assembler output the zeroing would appear to 
>>be part of the list_del inline, and edi seems to equate to &sb. If I 
>>have read that correctly then the 
>>oops happens at the beginning of
>>the list_add() inline and eax is the head of the s_dirty list - pointing 
>>into oblivion.
>>
>>__mark_inode_dirty() does not appear to take sb_lock before adding to 
>>the s_dirty list. Could that
>>be the culprit?   I'm completely unfamiliar with linux kernel so I might 
>>be way off here.
>>
>>-Jamie
>>
>>Andrea Arcangeli wrote:
>>
>>    
>>
>>>On Tue, Nov 04, 2003 at 07:52:40PM +0800, Jamie Clark wrote:
>>> 
>>>
>>>      
>>>
>>>>I made the quick fix (disabling rq_mergeable) and started the load test.
>>>>Will let it run for a week or so.
>>>>   
>>>>
>>>>        
>>>>
>>>does your later recent email means it deadlocked again even with this
>>>disabled?
>>>
>>>Could you try again with 2.4.23aa1 again just in case?
>>>
>>> 
>>>
>>>      
>>>
>>>>FYI an observation from my last test: the read latency seems to be much
>>>>improved and more consistent under this kernel (2.4.23pre6aa3, before
>>>>the oops and before this fix).  The maximum latency seemed steady over
>>>>the whole test without any of the longish pauses that showed up under
>>>>2.4.19. Quite a difference.
>>>>   
>>>>
>>>>        
>>>>
>>>nice to hear! thanks.
>>> 
>>>
>>>      
>>>
>
>  
>



^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: 2.4.23aa1 ext3 oops
  2003-12-11  2:33       ` 2.4.23aa1 ext3 oops Jamie Clark
  2003-12-16 15:15         ` Marcelo Tosatti
@ 2003-12-17 11:42         ` David Woodhouse
  2003-12-18  6:19           ` Jamie Clark
  1 sibling, 1 reply; 15+ messages in thread
From: David Woodhouse @ 2003-12-17 11:42 UTC (permalink / raw)
  To: Jamie Clark; +Cc: Andrea Arcangeli, linux-kernel

On Thu, 2003-12-11 at 10:33 +0800, Jamie Clark wrote:
> After a quick browse of the assembler output the zeroing would appear to 
> be part of the list_del inline, and edi seems to equate to &sb. 

Seems reasonable. It does look like something's stomped on sb->s_dirty.

> __mark_inode_dirty() does not appear to take sb_lock before adding to 
> the s_dirty list. Could that be the culprit?

I don't think so; it's holding the inode_lock which should be
sufficient. Besides -- in practice all updates to the 4-byte pointer
sb->s_dirty.next are going to be atomic, and there's no reason _ever_
for it to be set to d7ffbc08. It's hard to see how a simple locking
problem is going to cause such a thing.

How repeatable is this? Can you turn on slab poisoning?

-- 
dwmw2



^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: 2.4.23aa1 ext3 oops
  2003-12-17 11:42         ` David Woodhouse
@ 2003-12-18  6:19           ` Jamie Clark
  2003-12-18  6:25             ` David Woodhouse
  0 siblings, 1 reply; 15+ messages in thread
From: Jamie Clark @ 2003-12-18  6:19 UTC (permalink / raw)
  To: David Woodhouse; +Cc: Andrea Arcangeli, linux-kernel

David Woodhouse wrote:

>On Thu, 2003-12-11 at 10:33 +0800, Jamie Clark wrote:
>  
>
>>After a quick browse of the assembler output the zeroing would appear to 
>>be part of the list_del inline, and edi seems to equate to &sb. 
>>    
>>
>Seems reasonable. It does look like something's stomped on sb->s_dirty.
>  
>
>>__mark_inode_dirty() does not appear to take sb_lock before adding to 
>>the s_dirty list. Could that be the culprit?
>>    
>>
>
>I don't think so; it's holding the inode_lock which should be
>sufficient. Besides -- in practice all updates to the 4-byte pointer
>sb->s_dirty.next are going to be atomic, and there's no reason _ever_
>for it to be set to d7ffbc08. It's hard to see how a simple locking
>problem is going to cause such a thing.
>  
>
True.  I confess I didn't think too hard after narrowing down where it 
tripped up.

>How repeatable is this? 
>
The first oops ocurred after 4 or 5 days. My second run crashed in the 
first night, this time in filemap.c: precheck_file_write().
This oops seemed to be at or near the first dereference of inode, before 
the f_flags test.

        /* FIXME: this is for backwards compatibility with 2.4 */
        if (!S_ISBLK(inode->i_mode) && (file->f_flags & O_APPEND))
                *ppos = pos = inode->i_size;

 >>EIP; c01306fb <precheck_file_write+53/1f8>   <=====
Trace; c01308f8 <generic_file_write_nolock+58/4c8>
Trace; c013108f <generic_file_write+13f/158>
Trace; c016f3a7 <ext3_file_write+23/bc>
Trace; c0140e57 <sys_write+8f/100>
Trace; c0107133 <system_call+33/38>

>Can you turn on slab poisoning?
>  
>
Is CONFIG_DEBUG_SLAB all that I need?

I'm currently running 2.4.23aa1 with the inode.c patch reversions as 
Marcelo first suggested. When (IF) that crashes, or if I can get the 
test running on another SMP box I will revert to 2.4.23 + the qla2300 
driver that I need for the fibre-channel array.

-Jamie


^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: 2.4.23aa1 ext3 oops
  2003-12-18  6:19           ` Jamie Clark
@ 2003-12-18  6:25             ` David Woodhouse
  2003-12-18  7:33               ` Jamie Clark
  0 siblings, 1 reply; 15+ messages in thread
From: David Woodhouse @ 2003-12-18  6:25 UTC (permalink / raw)
  To: Jamie Clark; +Cc: Andrea Arcangeli, linux-kernel

On Thu, 2003-12-18 at 14:19 +0800, Jamie Clark wrote:
> The first oops ocurred after 4 or 5 days. My second run crashed in the 
> first night, this time in filemap.c: precheck_file_write().
> This oops seemed to be at or near the first dereference of inode, before 
> the f_flags test.
> 
>         /* FIXME: this is for backwards compatibility with 2.4 */
>         if (!S_ISBLK(inode->i_mode) && (file->f_flags & O_APPEND))
>                 *ppos = pos = inode->i_size;

Odd. And what's 'inode' then? What compiler are you using?

> Is CONFIG_DEBUG_SLAB all that I need?

Yes, I believe so.

-- 
dwmw2



^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: 2.4.23aa1 ext3 oops
  2003-12-18  6:25             ` David Woodhouse
@ 2003-12-18  7:33               ` Jamie Clark
  0 siblings, 0 replies; 15+ messages in thread
From: Jamie Clark @ 2003-12-18  7:33 UTC (permalink / raw)
  To: David Woodhouse; +Cc: Andrea Arcangeli, linux-kernel

David Woodhouse wrote:

>Odd. And what's 'inode' then? What compiler are you using?
>  
>
Debian woody  gcc 2.95.4

The code emitted at the start of precheck_file_write is:

precheck_file_write:
        pushl %ebp
        pushl %edi
        pushl %esi
        pushl %ebx
        movl 24(%esp),%ebp
        movl 32(%esp),%edx
        movl $-8192,%eax
[...]

I guessed inode to be ebp. the oops text copied from serial console (I 
lost the first line):

c01306fb
*pde = 00000000
Oops: 0002 2.4.23aa1 #12 SMP Thu Dec 11 11:25:47 SGT 2003
CPU:    1
EIP:    0010:[<c01306fb>]    Not tainted
EFLAGS: 00010206
eax: 00008000   ebx: 12ba4000   ecx: ffffffff   edx: f3a8ff48
esi: 00000000   edi: 00000000   ebp: c3526200   esp: f3a8feec
ds: 0018   es: 0018   ss: 0018
Process bonnie++ (pid: 316, stackpage=f3a8f000)
Stack: e7d715c0 c3526200 e7d715a0 00002000 c01308f8 e7d715a0 c3526200 
f3a8ff60
       f3a8ff48 c352626c c3526200 e7d715a0 00002000 f3a8ff44 0061a931 
00001000
       00000000 00001000 00001000 00000000 c3526200 c35262c4 00000000 
12ba4000
Call Trace:         [<c01308f8>] (88) [<c013108f>] (36) [<c016f3a7>] (36)
  [<c0140e57>] (36) [<c0107133>] (60)
Code: 00 60 74 15 8b 7c 24 14 f6 47 19 04 74 0b 8b 5d 44 8b 75 48

I'm now a little less confident about that last crash because I was a 
bit hasty in getting another run started. I have since trampled over the 
supporting evidence (kernel, map, build output).



^ permalink raw reply	[flat|nested] 15+ messages in thread

end of thread, other threads:[~2003-12-18  7:33 UTC | newest]

Thread overview: 15+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2003-11-04  2:49 2.4.23pre6aa3 scsi oops Jamie Clark
2003-11-04 10:28 ` Andrea Arcangeli
2003-11-04 11:52   ` Jamie Clark
2003-11-04 12:48     ` Andrea Arcangeli
2003-11-27  3:21     ` 2.4.23pre6aa3 - possible ext3 deadlock Jamie Clark
2003-11-27 12:56       ` Jan Kara
2003-12-06  1:05     ` 2.4.23pre6aa3 scsi oops Andrea Arcangeli
2003-12-06  1:39       ` Jamie Clark
2003-12-11  2:33       ` 2.4.23aa1 ext3 oops Jamie Clark
2003-12-16 15:15         ` Marcelo Tosatti
2003-12-16 15:55           ` Jamie Clark
2003-12-17 11:42         ` David Woodhouse
2003-12-18  6:19           ` Jamie Clark
2003-12-18  6:25             ` David Woodhouse
2003-12-18  7:33               ` Jamie Clark

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).