Re: sata_promise: random/intermittent errors

Re: sata_promise: random/intermittent errors

[Mikael Pettersson]

On Mon, 19 Feb 2007 12:43:50 +0800, Marc Marais wrote:
> I've decided to post this to the linux-ide list to see if I can get to the
> bottom of this problem I'm experiencing with sata_promise and my PATA drives.
> 
> I've pasted a thread from the linux-raid list where I was trying to
> troubleshoot/recover a destroyed raid5 array.
> 
> First a full history:
> 
> 1) 2.6.17.13: 3 drive PATA raid5 array with one drive starting to give read
> errors (legitimate according to SMART logs).
> 2) System lockups (no kernel panic seen) during load - I suspect due to the
> read error on the failing drive. 
> 3) Decide to upgrade to 2.6.20
> 4) Raid5 issues occur (handling of read errors caused md device to die). 
> 5) Patch from Neil to fix raid-5 error handling
> 6) Replace failed drive and add a new drive at the same time to create a 4
> drive PATA array.
> 7) Attempt to grow the array from 3 -> 4 devices which failed due to an error
> similar to this:
> 
> ata3: command timeout
> ata3: no sense translation for status: 0x40
> ata3: translated ATA stat/err 0x40/00 to SCSI SK/ASC/ASCQ 0xb/00/00
> ata4: status=0x40 { DriveReady }
> sd 3:0:0:0: SCSI error: return code = 0x08000002
> sdd: Current [descriptor]: sense key: Aborted Command
>      Additional sense: No additional sense information
> Descriptor sense data with sense descriptors (in hex):
>          72 0b 00 00 00 00 00 0c 00 0a 80 00 00 00 00 00
>          00 00 00 00
> end_request: I/O error, dev sdc, sector 260419647
> 
> 8) Raid array is trashed, rebuild array and restore from backup.
> 9) From this point on the system is up and running - restored to working
> state. However, I'm still getting errors similar to the above during array
> accesses (read/write). Not related to load. The array (being synced) manages
> to continue operation using another drive. My concern is that this may happen
> on a degraded array in future.
> 
> Note that the error I'm getting (shown above) has happened on sdc and sdd and
> at different sectors (i.e. not a consistent read error). Also, the SMART logs
> for both drives show NO error at all, short and long SMART tests complete
> successfully. I suspect this is an issue in the driver and/or my physical
> TX4000 card.

In the 2.6.20 kernel, 20619/TX4000 is still using the same driver
code and (old) error handling code it's been using for ages,
i.e., any 20619/TX4000 issues are unrelated to the SATAII and
new EH changes that I've done. Therefore I strongly suspect
either an old driver bug, or some hardware issue.

>From your dmesg log it seems you have at least 7 disks and a DVD
drive on two different controllers, an unused AIC7XXX, and an e1000
NIC, on a mainboard with a pair of Athlon MPs and 2GB RAM. All that
screams "power consumption" and "heat generation". Please make
absolutely sure that the PSU and cooling solutions are up to the job.
It doesn't hurt to check the cables and that the card is properly
seated as well. I'm assuming each drive is jumpered as master and
is connected at the far end of its cable?

It would be very useful if you could move the drives around,
so the sdc/sdd drives that experienced errors are moved to the
ports now used by sda/sdb. That should tell us if the errors
are tied to the drives or the ports.

/Mikael

Re: sata_promise: random/intermittent errors

[Marc Marais]

On Mon, 19 Feb 2007 11:26:24 +0100 (MET), Mikael Pettersson wrote
> On Mon, 19 Feb 2007 12:43:50 +0800, Marc Marais wrote:
> > I've decided to post this to the linux-ide list to see if I can get to the
> > bottom of this problem I'm experiencing with sata_promise and my PATA drives.
> > 
> > I've pasted a thread from the linux-raid list where I was trying to
> > troubleshoot/recover a destroyed raid5 array.
> > 
> > First a full history:
> > 
> > 1) 2.6.17.13: 3 drive PATA raid5 array with one drive starting to give read
> > errors (legitimate according to SMART logs).
> > 2) System lockups (no kernel panic seen) during load - I suspect due to the
> > read error on the failing drive. 
> > 3) Decide to upgrade to 2.6.20
> > 4) Raid5 issues occur (handling of read errors caused md device to die). 
> > 5) Patch from Neil to fix raid-5 error handling
> > 6) Replace failed drive and add a new drive at the same time to create a 4
> > drive PATA array.
> > 7) Attempt to grow the array from 3 -> 4 devices which failed due to an error
> > similar to this:
> > 
> > ata3: command timeout
> > ata3: no sense translation for status: 0x40
> > ata3: translated ATA stat/err 0x40/00 to SCSI SK/ASC/ASCQ 0xb/00/00
> > ata4: status=0x40 { DriveReady }
> > sd 3:0:0:0: SCSI error: return code = 0x08000002
> > sdd: Current [descriptor]: sense key: Aborted Command
> >      Additional sense: No additional sense information
> > Descriptor sense data with sense descriptors (in hex):
> >          72 0b 00 00 00 00 00 0c 00 0a 80 00 00 00 00 00
> >          00 00 00 00
> > end_request: I/O error, dev sdc, sector 260419647
> > 
> > 8) Raid array is trashed, rebuild array and restore from backup.
> > 9) From this point on the system is up and running - restored to working
> > state. However, I'm still getting errors similar to the above during array
> > accesses (read/write). Not related to load. The array (being synced) manages
> > to continue operation using another drive. My concern is that this may happen
> > on a degraded array in future.
> > 
> > Note that the error I'm getting (shown above) has happened on sdc and sdd and
> > at different sectors (i.e. not a consistent read error). Also, the SMART logs
> > for both drives show NO error at all, short and long SMART tests complete
> > successfully. I suspect this is an issue in the driver and/or my physical
> > TX4000 card.
> 
> In the 2.6.20 kernel, 20619/TX4000 is still using the same driver
> code and (old) error handling code it's been using for ages,
> i.e., any 20619/TX4000 issues are unrelated to the SATAII and
> new EH changes that I've done. Therefore I strongly suspect
> either an old driver bug, or some hardware issue.
> 
> >From your dmesg log it seems you have at least 7 disks and a DVD
> drive on two different controllers, an unused AIC7XXX, and an e1000
> NIC, on a mainboard with a pair of Athlon MPs and 2GB RAM. All that
> screams "power consumption" and "heat generation". Please make
> absolutely sure that the PSU and cooling solutions are up to the job.
> It doesn't hurt to check the cables and that the card is properly
> seated as well. I'm assuming each drive is jumpered as master and
> is connected at the far end of its cable?

I have been running this server for several years now in the same
configuration. I was originally running 4 80G drives and the only difference
now is they have been upgraded to 4 160G drives. The system is very well
cooled (CM Stacker case) and has a decent power supply which has been running
it for some time now.

However, I did reseat all cables and cards and also switched the IDE channels
around on the TX4000 card. I haven't had an error yet but, like I mentioned,
they are intermittent.
 
> It would be very useful if you could move the drives around,
> so the sdc/sdd drives that experienced errors are moved to the
> ports now used by sda/sdb. That should tell us if the errors
> are tied to the drives or the ports.

I will keep monitoring and check if the errors occur on the sda/sdb drives
since moving the drives around. 

Also, I saw a post on linux-kernel regarding another user seeing these
'command timeouts' (is that what they are?). If nothing can be done to prevent
occassional timeouts then at least they need to handled property by retrying
or whatever is best (I don't proclaim to have much inside knowledge of the
kernel so have no idea how libata handles errors). In my case, the md layer
was seeing the error and getting the data off another drive in the array which
could potential cause a problem if an array is already degraded when this happens.

Oh, and the aic7xxxx card IS being used - by an AIC tape drive ;) 

> /Mikael
> -

Thanks.

Regards,
Marc
--

Re: sata_promise: random/intermittent errors

[Marc Marais]

On Tue, 20 Feb 2007 12:48:12 +0800, Marc Marais wrote
> On Mon, 19 Feb 2007 11:26:24 +0100 (MET), Mikael Pettersson wrote
> > On Mon, 19 Feb 2007 12:43:50 +0800, Marc Marais wrote:
> > > I've decided to post this to the linux-ide list to see if I can get to 
the
> > > bottom of this problem I'm experiencing with sata_promise and my PATA 
drives.
> > > 
> > > I've pasted a thread from the linux-raid list where I was trying to
> > > troubleshoot/recover a destroyed raid5 array.
> > > 
> > > First a full history:
> > > 
> > > 1) 2.6.17.13: 3 drive PATA raid5 array with one drive starting to give 
read
> > > errors (legitimate according to SMART logs).
> > > 2) System lockups (no kernel panic seen) during load - I suspect due 
to the
> > > read error on the failing drive. 
> > > 3) Decide to upgrade to 2.6.20
> > > 4) Raid5 issues occur (handling of read errors caused md device to 
die). 
> > > 5) Patch from Neil to fix raid-5 error handling
> > > 6) Replace failed drive and add a new drive at the same time to create 
a 4
> > > drive PATA array.
> > > 7) Attempt to grow the array from 3 -> 4 devices which failed due to 
an error
> > > similar to this:
> > > 
> > > ata3: command timeout
> > > ata3: no sense translation for status: 0x40
> > > ata3: translated ATA stat/err 0x40/00 to SCSI SK/ASC/ASCQ 0xb/00/00
> > > ata4: status=0x40 { DriveReady }
> > > sd 3:0:0:0: SCSI error: return code = 0x08000002
> > > sdd: Current [descriptor]: sense key: Aborted Command
> > >      Additional sense: No additional sense information
> > > Descriptor sense data with sense descriptors (in hex):
> > >          72 0b 00 00 00 00 00 0c 00 0a 80 00 00 00 00 00
> > >          00 00 00 00
> > > end_request: I/O error, dev sdc, sector 260419647
> > > 
> > > 8) Raid array is trashed, rebuild array and restore from backup.
> > > 9) From this point on the system is up and running - restored to 
working
> > > state. However, I'm still getting errors similar to the above during 
array
> > > accesses (read/write). Not related to load. The array (being synced) 
manages
> > > to continue operation using another drive. My concern is that this may 
happen
> > > on a degraded array in future.
> > > 
> > > Note that the error I'm getting (shown above) has happened on sdc and 
sdd and
> > > at different sectors (i.e. not a consistent read error). Also, the 
SMART logs
> > > for both drives show NO error at all, short and long SMART tests 
complete
> > > successfully. I suspect this is an issue in the driver and/or my 
physical
> > > TX4000 card.
> > 
> > In the 2.6.20 kernel, 20619/TX4000 is still using the same driver
> > code and (old) error handling code it's been using for ages,
> > i.e., any 20619/TX4000 issues are unrelated to the SATAII and
> > new EH changes that I've done. Therefore I strongly suspect
> > either an old driver bug, or some hardware issue.
> > 
> > >From your dmesg log it seems you have at least 7 disks and a DVD
> > drive on two different controllers, an unused AIC7XXX, and an e1000
> > NIC, on a mainboard with a pair of Athlon MPs and 2GB RAM. All that
> > screams "power consumption" and "heat generation". Please make
> > absolutely sure that the PSU and cooling solutions are up to the job.
> > It doesn't hurt to check the cables and that the card is properly
> > seated as well. I'm assuming each drive is jumpered as master and
> > is connected at the far end of its cable?
> 
> I have been running this server for several years now in the same
> configuration. I was originally running 4 80G drives and the only 
difference
> now is they have been upgraded to 4 160G drives. The system is very well
> cooled (CM Stacker case) and has a decent power supply which has 
> been running it for some time now.
> 
> However, I did reseat all cables and cards and also switched the IDE 
> channels around on the TX4000 card. I haven't had an error yet but,
>  like I mentioned, they are intermittent.
> 
> > It would be very useful if you could move the drives around,
> > so the sdc/sdd drives that experienced errors are moved to the
> > ports now used by sda/sdb. That should tell us if the errors
> > are tied to the drives or the ports.
> 
> I will keep monitoring and check if the errors occur on the sda/sdb drives
> since moving the drives around.
> 
> Also, I saw a post on linux-kernel regarding another user seeing 
> these 'command timeouts' (is that what they are?). If nothing can be 
> done to prevent occassional timeouts then at least they need to 
> handled property by retrying or whatever is best (I don't proclaim 
> to have much inside knowledge of the kernel so have no idea how 
> libata handles errors). In my case, the md layer was seeing the 
> error and getting the data off another drive in the array which 
> could potential cause a problem if an array is already degraded when 
> this happens.
> 
> Oh, and the aic7xxxx card IS being used - by an AIC tape drive ;)
> 
> > /Mikael
> > -
> 
> Thanks.
> 
> Regards,
> Marc
> --

Replying to myself :)

Just an update. After switching the channels around I got some command 
timeouts and drives sda and sdb which implies a problem with the drives, 
however while examining the system I noticed the 6 pin aux power connector 
on the motherboard was loose - I'm not sure what effect that had but I 
noticed some MCE messages in the log (non-fatal correctable incident 
occurred on CPU x) before the system hang (which I think is ECC memory 
errors?). 

If I get more timeouts I'm going to replace the power supply. 

Anyway, sorry to burden the list with my problems, if you can take anything 
from this to improve the kernel/libata/sata_promise then at least I've made 
a contribution. Thanks for your time.

Regards,
Marc



--

sata_promise: random/intermittent errors

[Marc Marais]

I've decided to post this to the linux-ide list to see if I can get to the
bottom of this problem I'm experiencing with sata_promise and my PATA drives.

I've pasted a thread from the linux-raid list where I was trying to
troubleshoot/recover a destroyed raid5 array.

First a full history:

1) 2.6.17.13: 3 drive PATA raid5 array with one drive starting to give read
errors (legitimate according to SMART logs).
2) System lockups (no kernel panic seen) during load - I suspect due to the
read error on the failing drive. 
3) Decide to upgrade to 2.6.20
4) Raid5 issues occur (handling of read errors caused md device to die). 
5) Patch from Neil to fix raid-5 error handling
6) Replace failed drive and add a new drive at the same time to create a 4
drive PATA array.
7) Attempt to grow the array from 3 -> 4 devices which failed due to an error
similar to this:

ata3: command timeout
ata3: no sense translation for status: 0x40
ata3: translated ATA stat/err 0x40/00 to SCSI SK/ASC/ASCQ 0xb/00/00
ata4: status=0x40 { DriveReady }
sd 3:0:0:0: SCSI error: return code = 0x08000002
sdd: Current [descriptor]: sense key: Aborted Command
     Additional sense: No additional sense information
Descriptor sense data with sense descriptors (in hex):
         72 0b 00 00 00 00 00 0c 00 0a 80 00 00 00 00 00
         00 00 00 00
end_request: I/O error, dev sdc, sector 260419647

8) Raid array is trashed, rebuild array and restore from backup.
9) From this point on the system is up and running - restored to working
state. However, I'm still getting errors similar to the above during array
accesses (read/write). Not related to load. The array (being synced) manages
to continue operation using another drive. My concern is that this may happen
on a degraded array in future.

Note that the error I'm getting (shown above) has happened on sdc and sdd and
at different sectors (i.e. not a consistent read error). Also, the SMART logs
for both drives show NO error at all, short and long SMART tests complete
successfully. I suspect this is an issue in the driver and/or my physical
TX4000 card.

If you could shed any light on this I would appreciate it.

Thanks.
Regards.

------------- BEGIN DMESG DUMP -----------------

Linux version 2.6.20 (root@xerces) (gcc version 3.3.5 (Debian 1:3.3.5-13)) #2
SMP Mon Feb 12 09:28:29 GMT-9 2007 BIOS-provided physical RAM map:
sanitize start
sanitize end
copy_e820_map() start: 0000000000000000 size: 000000000009c800 end:
000000000009c800 type: 1
copy_e820_map() type is E820_RAM
copy_e820_map() start: 000000000009c800 size: 0000000000003800 end:
00000000000a0000 type: 2
copy_e820_map() start: 00000000000f0000 size: 0000000000010000 end:
0000000000100000 type: 2
copy_e820_map() start: 0000000000100000 size: 000000007feec000 end:
000000007ffec000 type: 1
copy_e820_map() type is E820_RAM
copy_e820_map() start: 000000007ffec000 size: 0000000000003000 end:
000000007ffef000 type: 3
copy_e820_map() start: 000000007ffef000 size: 0000000000010000 end:
000000007ffff000 type: 2
copy_e820_map() start: 000000007ffff000 size: 0000000000001000 end:
0000000080000000 type: 4
copy_e820_map() start: 00000000fec00000 size: 0000000000001000 end:
00000000fec01000 type: 2
copy_e820_map() start: 00000000fee00000 size: 0000000000001000 end:
00000000fee01000 type: 2
copy_e820_map() start: 00000000ffff0000 size: 0000000000010000 end:
0000000100000000 type: 2
 BIOS-e820: 0000000000000000 - 000000000009c800 (usable)
 BIOS-e820: 000000000009c800 - 00000000000a0000 (reserved)
 BIOS-e820: 00000000000f0000 - 0000000000100000 (reserved)
 BIOS-e820: 0000000000100000 - 000000007ffec000 (usable)
 BIOS-e820: 000000007ffec000 - 000000007ffef000 (ACPI data)
 BIOS-e820: 000000007ffef000 - 000000007ffff000 (reserved)
 BIOS-e820: 000000007ffff000 - 0000000080000000 (ACPI NVS)
 BIOS-e820: 00000000fec00000 - 00000000fec01000 (reserved)
 BIOS-e820: 00000000fee00000 - 00000000fee01000 (reserved)
 BIOS-e820: 00000000ffff0000 - 0000000100000000 (reserved) 1151MB HIGHMEM
available.
896MB LOWMEM available.
found SMP MP-table at 000f7ea0
Entering add_active_range(0, 0, 524268) 0 entries of 256 used Zone PFN ranges:
  DMA             0 ->     4096
  Normal       4096 ->   229376
  HighMem    229376 ->   524268
early_node_map[1] active PFN ranges
    0:        0 ->   524268
On node 0 totalpages: 524268
  DMA zone: 32 pages used for memmap
  DMA zone: 0 pages reserved
  DMA zone: 4064 pages, LIFO batch:0
  Normal zone: 1760 pages used for memmap
  Normal zone: 223520 pages, LIFO batch:31
  HighMem zone: 2303 pages used for memmap
  HighMem zone: 292589 pages, LIFO batch:31 DMI 2.3 present.
Intel MultiProcessor Specification v1.4
    Virtual Wire compatibility mode.
OEM ID: ASUS     Product ID: PROD00000000 APIC at: 0xFEE00000
Processor #0 6:10 APIC version 16
Processor #1 6:10 APIC version 16
I/O APIC #2 Version 17 at 0xFEC00000.
Enabling APIC mode:  Flat.  Using 1 I/O APICs
Processors: 2
Allocating PCI resources starting at 88000000 (gap: 80000000:7ec00000)
Detected 2133.464 MHz processor.
Built 1 zonelists.  Total pages: 520173
Kernel command line: auto BOOT_IMAGE=Linux ro root=901 acpi=off pci=noacpi
elevator=as mapped APIC to ffffd000 (fee00000) mapped IOAPIC to ffffc000
(fec00000) Enabling fast FPU save and restore... done.
Enabling unmasked SIMD FPU exception support... done.
Initializing CPU#0
PID hash table entries: 4096 (order: 12, 16384 bytes)
Console: colour VGA+ 80x50
Dentry cache hash table entries: 131072 (order: 7, 524288 bytes) Inode-cache
hash table entries: 65536 (order: 6, 262144 bytes)
Memory: 2072936k/2097072k available (1539k kernel code, 22916k reserved, 593k
data, 200k init, 1179568k highmem) virtual kernel memory layout:
    fixmap  : 0xfffa2000 - 0xfffff000   ( 372 kB)
    pkmap   : 0xff800000 - 0xffc00000   (4096 kB)
    vmalloc : 0xf8800000 - 0xff7fe000   ( 111 MB)
    lowmem  : 0xc0000000 - 0xf8000000   ( 896 MB)
      .init : 0xc031b000 - 0xc034d000   ( 200 kB)
      .data : 0xc0280c62 - 0xc0315230   ( 593 kB)
      .text : 0xc0100000 - 0xc0280c62   (1539 kB)
Checking if this processor honours the WP bit even in supervisor mode... Ok.
Calibrating delay using timer specific routine.. 4269.42 BogoMIPS
(lpj=2134710) Mount-cache hash table entries: 512
CPU: After generic identify, caps: 0383fbff c1cbfbff 00000000 00000000
00000000 00000000 00000000
CPU: CLK_CTL MSR was 60031223. Reprogramming to 20031223
CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
CPU: L2 Cache: 512K (64 bytes/line)
CPU: After all inits, caps: 0383fbff c1cbfbff 00000000 00000420 00000000
00000000 00000000 Intel machine check architecture supported.
Intel machine check reporting enabled on CPU#0.
Compat vDSO mapped to ffffe000.
Checking 'hlt' instruction... OK.
Freeing SMP alternatives: 10k freed
CPU0: AMD Athlon(TM) MP 2800+ stepping 00 Booting processor 1/1 eip 2000
Initializing CPU#1 Calibrating delay using timer specific routine.. 4266.31
BogoMIPS (lpj=2133156)
CPU: After generic identify, caps: 0383fbff c1cbfbff 00000000 00000000
00000000 00000000 00000000
CPU: CLK_CTL MSR was 60031223. Reprogramming to 20031223
CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
CPU: L2 Cache: 512K (64 bytes/line)
CPU: After all inits, caps: 0383fbff c1cbfbff 00000000 00000420 00000000
00000000 00000000 Intel machine check architecture supported.
Intel machine check reporting enabled on CPU#1.
CPU1: AMD Athlon(TM) MP 2800+ stepping 00 Total of 2 processors activated
(8535.73 BogoMIPS).
ExtINT not setup in hardware but reported by MP table ENABLING IO-APIC IRQs
..TIMER: vector=0x31 apic1=0 pin1=2 apic2=0 pin2=0 checking TSC
synchronization across 2 CPUs: passed.
Brought up 2 CPUs
migration_cost=1084
NET: Registered protocol family 16
PCI: PCI BIOS revision 2.10 entry at 0xf1f30, last bus=2
PCI: Using configuration type 1
Setting up standard PCI resources
mtrr: your CPUs had inconsistent fixed MTRR settings
mtrr: probably your BIOS does not setup all CPUs.
mtrr: corrected configuration.
Linux Plug and Play Support v0.97 (c) Adam Belay
PnPBIOS: Scanning system for PnP BIOS support...
PnPBIOS: Found PnP BIOS installation structure at 0xc00fc5f0
PnPBIOS: PnP BIOS version 1.0, entry 0xf0000:0xc620, dseg 0xf0000
PnPBIOS: 13 nodes reported by PnP BIOS; 13 recorded by driver SCSI subsystem
initialized libata version 2.00 loaded.
PCI: Probing PCI hardware
PCI: Probing PCI hardware (bus 00)
Boot video device is 0000:01:05.0
PCI: Using IRQ router AMD768 [1022/7443] at 0000:00:07.3
PCI->APIC IRQ transform: 0000:00:08.0[A] -> IRQ 16 APIC IRQ transform: 
PCI->0000:00:09.0[A] -> IRQ 17 APIC IRQ transform: 0000:01:05.0[A] -> 
PCI->IRQ 16 APIC IRQ transform: 0000:02:04.0[A] -> IRQ 17 APIC IRQ 
PCI->transform: 0000:02:05.0[A] -> IRQ 18 APIC IRQ transform: 
PCI->0000:02:05.1[B] -> IRQ 19 APIC IRQ transform: 0000:02:05.2[C] -> 
PCI->IRQ 16 APIC IRQ transform: 0000:02:06.0[A] -> IRQ 17 APIC IRQ 
PCI->transform: 0000:02:08.0[A] -> IRQ 19
pnp: 00:0f: ioport range 0xe400-0xe47f has been reserved
pnp: 00:0f: ioport range 0xe4e0-0xe4ff has been reserved
PCI: Bridge: 0000:00:01.0
  IO window: disabled.
  MEM window: ee000000-efcfffff
  PREFETCH window: eff00000-fb7fffff
PCI: Bridge: 0000:00:10.0
  IO window: a000-afff
  MEM window: e8800000-ebffffff
  PREFETCH window: efd00000-efdfffff
PCI: Setting latency timer of device 0000:00:01.0 to 64
NET: Registered protocol family 2
IP route cache hash table entries: 32768 (order: 5, 131072 bytes) TCP
established hash table entries: 131072 (order: 8, 1048576 bytes) TCP bind hash
table entries: 65536 (order: 7, 524288 bytes)
TCP: Hash tables configured (established 131072 bind 65536) TCP reno
registered checking if image is initramfs...it isn't (bad gzip magic numbers);
looks like an initrd Freeing initrd memory: 3072k freed Machine check
exception polling timer started.
highmem bounce pool size: 64 pages
VFS: Disk quotas dquot_6.5.1
Dquot-cache hash table entries: 1024 (order 0, 4096 bytes) io scheduler noop
registered io scheduler anticipatory registered (default) io scheduler
deadline registered io scheduler cfq registered BIOS failed to enable PCI
standards compliance, fixing this error.
isapnp: Scanning for PnP cards...
isapnp: No Plug & Play device found
Serial: 8250/16550 driver $Revision: 1.90 $ 2 ports, IRQ sharing enabled
serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
serial8250: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
00:02: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
00:03: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A RAMDISK driver initialized: 16
RAM disks of 8192K size 1024 blocksize
PNP: PS/2 Controller [PNP0303,PNP0f13] at 0x60,0x64 irq 1,12
serio: i8042 KBD port at 0x60,0x64 irq 1
mice: PS/2 mouse device common for all mice TCP cubic registered Starting
balanced_irq Using IPI Shortcut mode
input: AT Translated Set 2 keyboard as /class/input/input0
RAMDISK: cramfs filesystem found at block 0
RAMDISK: Loading 3072KiB [1 disk] into ram disk... 
VFS: Mounted root (cramfs filesystem) readonly.
Freeing unused kernel memory: 200k freed
NET: Registered protocol family 1
md: raid1 personality registered for level 1 Uniform Multi-Platform E-IDE
driver Revision: 7.00alpha2
ide: Assuming 33MHz system bus speed for PIO modes; override with idebus=xx
AMD7441: IDE controller at PCI slot 0000:00:07.1
AMD7441: chipset revision 4
AMD7441: not 100% native mode: will probe irqs later
AMD7441: 0000:00:07.1 (rev 04) UDMA100 controller
    ide0: BM-DMA at 0xd800-0xd807, BIOS settings: hda:DMA, hdb:DMA
    ide1: BM-DMA at 0xd808-0xd80f, BIOS settings: hdc:DMA, hdd:DMA Probing IDE
interface ide0...
hda: WDC WD800BB-00JHC0, ATA DISK drive
hdb: WDC WD2500JB-00GVC0, ATA DISK drive ide0 at 0x1f0-0x1f7,0x3f6 on irq 14
Probing IDE interface ide1...
hdc: WDC WD800BB-23DKA0, ATA DISK drive
hdd: HL-DT-STDVD-ROM GDR8163B, ATAPI CD/DVD-ROM drive
ide1 at 0x170-0x177,0x376 on irq 15
hda: max request size: 128KiB
hda: 156301488 sectors (80026 MB) w/2048KiB Cache, CHS=65535/16/63, UDMA(100)
hda: cache flushes supported
 hda: hda1 hda2 hda3 hda4 < hda5 hda6 hda7 hda8 >
hdb: max request size: 512KiB
hdb: 488397168 sectors (250059 MB) w/8192KiB Cache, CHS=30401/255/63, UDMA(100)
hdb: cache flushes supported
 hdb: hdb1
hdc: max request size: 512KiB
hdc: 156312576 sectors (80032 MB) w/2048KiB Cache, CHS=16383/255/63, UDMA(100)
hdc: cache flushes supported
 hdc: hdc1 hdc2 hdc3 hdc4 < hdc5 hdc6 hdc7 hdc8 >
md: md0 stopped.
md: bind<hda1>
md: bind<hdc1>
raid1: raid set md0 active with 2 out of 2 mirrors
md: md1 stopped.
md: bind<hda2>
md: bind<hdc2>
raid1: raid set md1 active with 2 out of 2 mirrors kjournald starting.  Commit
interval 5 seconds
EXT3-fs: mounted filesystem with ordered data mode.
hda: cache flushes supported
hdc: cache flushes supported
hdb: cache flushes supported
Adding 2007992k swap on /dev/md0.  Priority:-1 extents:1 across:2007992k
EXT3 FS on md1, internal journal
Real Time Clock Driver v1.12ac
hdd: ATAPI 52X DVD-ROM drive, 256kB Cache, UDMA(33) Uniform CD-ROM driver
Revision: 3.20
ieee1394: Initialized config rom entry `ip1394'
ieee1394: raw1394: /dev/raw1394 device initialized
ohci1394: fw-host0: OHCI-1394 1.0 (PCI): IRQ=[17]  MMIO=[e9800000-e98007ff] 
Max Packet=[2048]  IR/IT contexts=[4/8]
video1394: Installed video1394 module
AMD768 RNG detected
usbcore: registered new interface driver usbfs
usbcore: registered new interface driver hub
usbcore: registered new device driver usb
ohci_hcd: 2006 August 04 USB 1.1 'Open' Host Controller (OHCI) Driver (PCI)
ohci_hcd 0000:02:05.0: OHCI Host Controller ohci_hcd 0000:02:05.0: new USB bus
registered, assigned bus number 1 ohci_hcd 0000:02:05.0: irq 18, io mem
0xeb000000 usb usb1: configuration #1 chosen from 1 choice hub 1-0:1.0: USB
hub found hub 1-0:1.0: 3 ports detected ohci_hcd 0000:02:05.1: OHCI Host
Controller ohci_hcd 0000:02:05.1: new USB bus registered, assigned bus number
2 ohci_hcd 0000:02:05.1: irq 19, io mem 0xea800000
ieee1394: Host added: ID:BUS[0-00:1023]  GUID[005042f81010a4eb] usb usb2:
configuration #1 chosen from 1 choice hub 2-0:1.0: USB hub found hub 2-0:1.0:
2 ports detected
usbcore: registered new interface driver hiddev
usbcore: registered new interface driver usbhid
drivers/usb/input/hid-core.c: v2.6:USB HID core driver
Intel(R) PRO/1000 Network Driver - version 7.3.15-k2 Copyright (c) 1999-2006
Intel Corporation.
e1000: 0000:00:09.0: e1000_probe: (PCI:66MHz:32-bit) 00:0e:0c:a0:04:dd
e1000: eth0: e1000_probe: Intel(R) PRO/1000 Network Connection scsi0 : Adaptec
AIC7XXX EISA/VLB/PCI SCSI HBA DRIVER, Rev 7.0
        <Adaptec 2940 Ultra SCSI adapter>
        aic7880: Ultra Wide Channel A, SCSI Id=7, 16/253 SCBs

scsi 0:0:0:0: Sequential-Access SONY     SDX-500C         0101 PQ: 0 ANSI: 2
 target0:0:0: Beginning Domain Validation
 target0:0:0: wide asynchronous
 target0:0:0: FAST-10 WIDE SCSI 20.0 MB/s ST (100 ns, offset 8)
 target0:0:0: Domain Validation skipping write tests
 target0:0:0: Ending Domain Validation
sata_promise 0000:00:08.0: version 1.05
ata1: PATA max UDMA/133 cmd 0xF8AA6200 ctl 0xF8AA6238 bmdma 0x0 irq 16
ata2: PATA max UDMA/133 cmd 0xF8AA6280 ctl 0xF8AA62B8 bmdma 0x0 irq 16
ata3: PATA max UDMA/133 cmd 0xF8AA6300 ctl 0xF8AA6338 bmdma 0x0 irq 16
ata4: PATA max UDMA/133 cmd 0xF8AA6380 ctl 0xF8AA63B8 bmdma 0x0 irq 16
scsi1 : sata_promise
ata1.00: ATA-7, max UDMA/100, 312581808 sectors: LBA48
ata1.00: ata1: dev 0 multi count 0
ata1.00: configured for UDMA/100
scsi2 : sata_promise
ata2.00: ATA-7, max UDMA/100, 312581808 sectors: LBA48
ata2.00: ata2: dev 0 multi count 0
ata2.00: configured for UDMA/100
scsi3 : sata_promise
ata3.00: ATA-7, max UDMA/100, 312581808 sectors: LBA48
ata3.00: ata3: dev 0 multi count 0
ata3.00: configured for UDMA/100
scsi4 : sata_promise
ata4.00: ATA-6, max UDMA/100, 312581808 sectors: LBA48
ata4.00: ata4: dev 0 multi count 0
ata4.00: configured for UDMA/100
scsi 1:0:0:0: Direct-Access     ATA      WDC WD1600JB-00R 20.0 PQ: 0 ANSI: 5
scsi 2:0:0:0: Direct-Access     ATA      WDC WD1600JB-00R 20.0 PQ: 0 ANSI: 5
scsi 3:0:0:0: Direct-Access     ATA      WDC WD1600JB-00R 20.0 PQ: 0 ANSI: 5
scsi 4:0:0:0: Direct-Access     ATA      WDC WD1600JB-00E 15.0 PQ: 0 ANSI: 5
device-mapper: ioctl: 4.11.0-ioctl (2006-10-12) initialised: dm-devel@redhat.com
md: md2 stopped.
md: bind<hda3>
md: bind<hdc3>
raid1: raid set md2 active with 2 out of 2 mirrors
md: md3 stopped.
md: bind<hda5>
md: bind<hdc5>
raid1: raid set md3 active with 2 out of 2 mirrors
md: md4 stopped.
md: bind<hda6>
md: bind<hdc6>
raid1: raid set md4 active with 2 out of 2 mirrors
md: md5 stopped.
md: bind<hda7>
md: bind<hdc7>
raid1: raid set md5 active with 2 out of 2 mirrors
md: md6 stopped.
SCSI device sda: 312581808 512-byte hdwr sectors (160042 MB)
sda: Write Protect is off
sda: Mode Sense: 00 3a 00 00
SCSI device sda: write cache: enabled, read cache: enabled, doesn't support
DPO or FUA SCSI device sda: 312581808 512-byte hdwr sectors (160042 MB)
sda: Write Protect is off
sda: Mode Sense: 00 3a 00 00
SCSI device sda: write cache: enabled, read cache: enabled, doesn't support
DPO or FUA
 sda: sda1
sd 1:0:0:0: Attached scsi disk sda
SCSI device sdb: 312581808 512-byte hdwr sectors (160042 MB)
sdb: Write Protect is off
sdb: Mode Sense: 00 3a 00 00
SCSI device sdb: write cache: enabled, read cache: enabled, doesn't support
DPO or FUA SCSI device sdb: 312581808 512-byte hdwr sectors (160042 MB)
sdb: Write Protect is off
sdb: Mode Sense: 00 3a 00 00
SCSI device sdb: write cache: enabled, read cache: enabled, doesn't support
DPO or FUA
 sdb: sdb1
sd 2:0:0:0: Attached scsi disk sdb
SCSI device sdc: 312581808 512-byte hdwr sectors (160042 MB)
sdc: Write Protect is off
sdc: Mode Sense: 00 3a 00 00
SCSI device sdc: write cache: enabled, read cache: enabled, doesn't support
DPO or FUA SCSI device sdc: 312581808 512-byte hdwr sectors (160042 MB)
sdc: Write Protect is off
sdc: Mode Sense: 00 3a 00 00
SCSI device sdc: write cache: enabled, read cache: enabled, doesn't support
DPO or FUA
 sdc: sdc1
sd 3:0:0:0: Attached scsi disk sdc
SCSI device sdd: 312581808 512-byte hdwr sectors (160042 MB)
sdd: Write Protect is off
sdd: Mode Sense: 00 3a 00 00
SCSI device sdd: write cache: enabled, read cache: enabled, doesn't support
DPO or FUA SCSI device sdd: 312581808 512-byte hdwr sectors (160042 MB)
sdd: Write Protect is off
sdd: Mode Sense: 00 3a 00 00
SCSI device sdd: write cache: enabled, read cache: enabled, doesn't support
DPO or FUA
 sdd: sdd1
sd 4:0:0:0: Attached scsi disk sdd
md: bind<sdb1>
md: bind<sdc1>
md: bind<sda1>
raid5: automatically using best checksumming function: pIII_sse
   pIII_sse  :  4928.000 MB/sec
raid5: using function: pIII_sse (4928.000 MB/sec)
raid6: int32x1    855 MB/s
raid6: int32x2   1156 MB/s
raid6: int32x4    730 MB/s
raid6: int32x8    648 MB/s
raid6: mmxx1     1781 MB/s
raid6: mmxx2     3265 MB/s
raid6: sse1x1     464 MB/s
raid6: sse1x2     929 MB/s
raid6: using algorithm sse1x2 (929 MB/s)
md: raid6 personality registered for level 6
md: raid5 personality registered for level 5
md: raid4 personality registered for level 4
raid5: device sda1 operational as raid disk 0
raid5: device sdc1 operational as raid disk 2
raid5: device sdb1 operational as raid disk 1
raid5: allocated 4204kB for md6
raid5: raid level 5 set md6 active with 3 out of 4 devices, algorithm 2
RAID5 conf printout:
 --- rd:4 wd:3
 disk 0, o:1, dev:sda1
 disk 1, o:1, dev:sdb1
 disk 2, o:1, dev:sdc1
md: md7 stopped.
md: bind<hdc8>
md: bind<hda8>
raid1: raid set md7 active with 2 out of 2 mirrors
st: Version 20061107, fixed bufsize 32768, s/g segs 256 st 0:0:0:0: Attached
scsi tape st0 st 0:0:0:0: st0: try direct i/o: yes (alignment 512 B)
 target0:0:0: FAST-10 WIDE SCSI 20.0 MB/s ST (100 ns, offset 8)
st0: Block limits 2 - 16777215 bytes.
program stinit is using a deprecated SCSI ioctl, please convert it to SG_IO
kjournald starting.  Commit interval 5 seconds
EXT3 FS on md2, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
kjournald starting.  Commit interval 5 seconds
EXT3 FS on md3, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
kjournald starting.  Commit interval 5 seconds
EXT3 FS on md4, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
kjournald starting.  Commit interval 5 seconds
EXT3 FS on md5, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
kjournald starting.  Commit interval 5 seconds
EXT3 FS on md6, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
kjournald starting.  Commit interval 5 seconds
EXT3 FS on md7, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
kjournald starting.  Commit interval 5 seconds
EXT3 FS on hdb1, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
e1000: eth0: e1000_watchdog: NIC Link is Up 1000 Mbps Full Duplex
e1000: eth0: e1000_set_tso: TSO is Disabled
e1000: eth0: e1000_set_tso: TSO is Disabled
e1000: eth0: e1000_set_tso: TSO is Disabled process `syslogd' is using
obsolete setsockopt SO_BSDCOMPAT
ata1: no sense translation for status: 0x50
ata1: translated ATA stat/err 0x50/00 to SCSI SK/ASC/ASCQ 0xb/00/00
ata1: status=0x50 { DriveReady SeekComplete }
ata1: no sense translation for status: 0x50
ata1: translated ATA stat/err 0x50/00 to SCSI SK/ASC/ASCQ 0xb/00/00
ata1: status=0x50 { DriveReady SeekComplete }
ata1: no sense translation for status: 0x50
ata1: translated ATA stat/err 0x50/00 to SCSI SK/ASC/ASCQ 0xb/00/00
ata1: status=0x50 { DriveReady SeekComplete }
ata1: no sense translation for status: 0x50
ata1: translated ATA stat/err 0x50/00 to SCSI SK/ASC/ASCQ 0xb/00/00
ata1: status=0x50 { DriveReady SeekComplete }
ata1: no sense translation for status: 0x50
ata1: translated ATA stat/err 0x50/00 to SCSI SK/ASC/ASCQ 0xb/00/00
ata1: status=0x50 { DriveReady SeekComplete }
ata1: no sense translation for status: 0x50
ata1: translated ATA stat/err 0x50/00 to SCSI SK/ASC/ASCQ 0xb/00/00
ata1: status=0x50 { DriveReady SeekComplete }
ata1: no sense translation for status: 0x50
ata1: translated ATA stat/err 0x50/00 to SCSI SK/ASC/ASCQ 0xb/00/00
ata1: status=0x50 { DriveReady SeekComplete }
ata1: no sense translation for status: 0x50
ata1: translated ATA stat/err 0x50/00 to SCSI SK/ASC/ASCQ 0xb/00/00
ata1: status=0x50 { DriveReady SeekComplete }
ata1: no sense translation for status: 0x50
ata1: translated ATA stat/err 0x50/00 to SCSI SK/ASC/ASCQ 0xb/00/00
ata1: status=0x50 { DriveReady SeekComplete }
ata1: no sense translation for status: 0x50
ata1: translated ATA stat/err 0x50/00 to SCSI SK/ASC/ASCQ 0xb/00/00
ata1: status=0x50 { DriveReady SeekComplete }
ata1: no sense translation for status: 0x50
ata1: translated ATA stat/err 0x50/00 to SCSI SK/ASC/ASCQ 0xb/00/00
ata1: status=0x50 { DriveReady SeekComplete }
ata1: no sense translation for status: 0x50
ata1: translated ATA stat/err 0x50/00 to SCSI SK/ASC/ASCQ 0xb/00/00
ata1: status=0x50 { DriveReady SeekComplete }
ata2: no sense translation for status: 0x50
ata2: translated ATA stat/err 0x50/00 to SCSI SK/ASC/ASCQ 0xb/00/00
ata2: status=0x50 { DriveReady SeekComplete }
ata2: no sense translation for status: 0x50
ata2: translated ATA stat/err 0x50/00 to SCSI SK/ASC/ASCQ 0xb/00/00
ata2: status=0x50 { DriveReady SeekComplete }
ata2: no sense translation for status: 0x50
ata2: translated ATA stat/err 0x50/00 to SCSI SK/ASC/ASCQ 0xb/00/00
ata2: status=0x50 { DriveReady SeekComplete }
ata2: no sense translation for status: 0x50
ata2: translated ATA stat/err 0x50/00 to SCSI SK/ASC/ASCQ 0xb/00/00
ata2: status=0x50 { DriveReady SeekComplete }
ata2: no sense translation for status: 0x50
ata2: translated ATA stat/err 0x50/00 to SCSI SK/ASC/ASCQ 0xb/00/00
ata2: status=0x50 { DriveReady SeekComplete }
ata2: no sense translation for status: 0x50
ata2: translated ATA stat/err 0x50/00 to SCSI SK/ASC/ASCQ 0xb/00/00
ata2: status=0x50 { DriveReady SeekComplete }
ata2: no sense translation for status: 0x50
ata2: translated ATA stat/err 0x50/00 to SCSI SK/ASC/ASCQ 0xb/00/00
ata2: status=0x50 { DriveReady SeekComplete }
ata2: no sense translation for status: 0x50
ata2: translated ATA stat/err 0x50/00 to SCSI SK/ASC/ASCQ 0xb/00/00
ata2: status=0x50 { DriveReady SeekComplete }
ata2: no sense translation for status: 0x50
ata2: translated ATA stat/err 0x50/00 to SCSI SK/ASC/ASCQ 0xb/00/00
ata2: status=0x50 { DriveReady SeekComplete }
ata2: no sense translation for status: 0x50
ata2: translated ATA stat/err 0x50/00 to SCSI SK/ASC/ASCQ 0xb/00/00
ata2: status=0x50 { DriveReady SeekComplete }
ata2: no sense translation for status: 0x50
ata2: translated ATA stat/err 0x50/00 to SCSI SK/ASC/ASCQ 0xb/00/00
ata2: status=0x50 { DriveReady SeekComplete }
ata2: no sense translation for status: 0x50
ata2: translated ATA stat/err 0x50/00 to SCSI SK/ASC/ASCQ 0xb/00/00
ata2: status=0x50 { DriveReady SeekComplete }
ata3: no sense translation for status: 0x50
ata3: translated ATA stat/err 0x50/00 to SCSI SK/ASC/ASCQ 0xb/00/00
ata3: status=0x50 { DriveReady SeekComplete }
ata3: no sense translation for status: 0x50
ata3: translated ATA stat/err 0x50/00 to SCSI SK/ASC/ASCQ 0xb/00/00
ata3: status=0x50 { DriveReady SeekComplete }
ata3: no sense translation for status: 0x50
ata3: translated ATA stat/err 0x50/00 to SCSI SK/ASC/ASCQ 0xb/00/00
ata3: status=0x50 { DriveReady SeekComplete }
ata3: no sense translation for status: 0x50
ata3: translated ATA stat/err 0x50/00 to SCSI SK/ASC/ASCQ 0xb/00/00
ata3: status=0x50 { DriveReady SeekComplete }
ata3: no sense translation for status: 0x50
ata3: translated ATA stat/err 0x50/00 to SCSI SK/ASC/ASCQ 0xb/00/00
ata3: status=0x50 { DriveReady SeekComplete }
ata3: no sense translation for status: 0x50
ata3: translated ATA stat/err 0x50/00 to SCSI SK/ASC/ASCQ 0xb/00/00
ata3: status=0x50 { DriveReady SeekComplete }
ata3: no sense translation for status: 0x50
ata3: translated ATA stat/err 0x50/00 to SCSI SK/ASC/ASCQ 0xb/00/00
ata3: status=0x50 { DriveReady SeekComplete }
ata3: no sense translation for status: 0x50
ata3: translated ATA stat/err 0x50/00 to SCSI SK/ASC/ASCQ 0xb/00/00
ata3: status=0x50 { DriveReady SeekComplete }
ata3: no sense translation for status: 0x50
ata3: translated ATA stat/err 0x50/00 to SCSI SK/ASC/ASCQ 0xb/00/00
ata3: status=0x50 { DriveReady SeekComplete }
ata3: no sense translation for status: 0x50
ata3: translated ATA stat/err 0x50/00 to SCSI SK/ASC/ASCQ 0xb/00/00
ata3: status=0x50 { DriveReady SeekComplete }
ata3: no sense translation for status: 0x50
ata3: translated ATA stat/err 0x50/00 to SCSI SK/ASC/ASCQ 0xb/00/00
ata3: status=0x50 { DriveReady SeekComplete }
ata3: no sense translation for status: 0x50
ata3: translated ATA stat/err 0x50/00 to SCSI SK/ASC/ASCQ 0xb/00/00
ata3: status=0x50 { DriveReady SeekComplete }
ata4: no sense translation for status: 0x50
ata4: translated ATA stat/err 0x50/00 to SCSI SK/ASC/ASCQ 0xb/00/00
ata4: status=0x50 { DriveReady SeekComplete }
ata4: no sense translation for status: 0x50
ata4: translated ATA stat/err 0x50/00 to SCSI SK/ASC/ASCQ 0xb/00/00
ata4: status=0x50 { DriveReady SeekComplete }
ata4: no sense translation for status: 0x50
ata4: translated ATA stat/err 0x50/00 to SCSI SK/ASC/ASCQ 0xb/00/00
ata4: status=0x50 { DriveReady SeekComplete }
ata4: no sense translation for status: 0x50
ata4: translated ATA stat/err 0x50/00 to SCSI SK/ASC/ASCQ 0xb/00/00
ata4: status=0x50 { DriveReady SeekComplete }
ata4: no sense translation for status: 0x50
ata4: translated ATA stat/err 0x50/00 to SCSI SK/ASC/ASCQ 0xb/00/00
ata4: status=0x50 { DriveReady SeekComplete }
ata4: no sense translation for status: 0x50
ata4: translated ATA stat/err 0x50/00 to SCSI SK/ASC/ASCQ 0xb/00/00
ata4: status=0x50 { DriveReady SeekComplete }
ata4: no sense translation for status: 0x50
ata4: translated ATA stat/err 0x50/00 to SCSI SK/ASC/ASCQ 0xb/00/00
ata4: status=0x50 { DriveReady SeekComplete }
ata4: no sense translation for status: 0x50
ata4: translated ATA stat/err 0x50/00 to SCSI SK/ASC/ASCQ 0xb/00/00
ata4: status=0x50 { DriveReady SeekComplete }
ata4: no sense translation for status: 0x50
ata4: translated ATA stat/err 0x50/00 to SCSI SK/ASC/ASCQ 0xb/00/00
ata4: status=0x50 { DriveReady SeekComplete }
ata4: no sense translation for status: 0x50
ata4: translated ATA stat/err 0x50/00 to SCSI SK/ASC/ASCQ 0xb/00/00
ata4: status=0x50 { DriveReady SeekComplete }
ata4: no sense translation for status: 0x50
ata4: translated ATA stat/err 0x50/00 to SCSI SK/ASC/ASCQ 0xb/00/00
ata4: status=0x50 { DriveReady SeekComplete }
ata4: no sense translation for status: 0x50
ata4: translated ATA stat/err 0x50/00 to SCSI SK/ASC/ASCQ 0xb/00/00
ata4: status=0x50 { DriveReady SeekComplete }
st0: MTSETDRVBUFFER only allowed for root.
vmmon: module license 'unspecified' taints kernel.
/dev/vmmon[2331]: Module vmmon: registered with major=10 minor=165
/dev/vmmon[2331]: Module vmmon: initialized
/dev/vmnet: open called by PID 2366 (vmnet-bridge)
/dev/vmnet: hub 0 does not exist, allocating memory.
/dev/vmnet: port on hub 0 successfully opened
bridge-eth0: enabling the bridge
bridge-eth0: up
bridge-eth0: already up
bridge-eth0: attached
floppy0: no floppy controllers found
floppy0: no floppy controllers found
st 0:0:0:0: Attached scsi generic sg0 type 1 sd 1:0:0:0: Attached scsi generic
sg1 type 0 sd 2:0:0:0: Attached scsi generic sg2 type 0 sd 3:0:0:0: Attached
scsi generic sg3 type 0 sd 4:0:0:0: Attached scsi generic sg4 type 0
/dev/vmnet: open called by PID 2723 (vmware-vmx) device eth0 entered
promiscuous mode
bridge-eth0: enabled promiscuous mode
/dev/vmnet: port on hub 0 successfully opened
/dev/vmmon[2744]: host clock rate change request 0 -> 1001
/dev/vmnet: open called by PID 2972 (vmware-vmx)
/dev/vmnet: port on hub 0 successfully opened
md: bind<sdd1>
RAID5 conf printout:
 --- rd:4 wd:3
 disk 0, o:1, dev:sda1
 disk 1, o:1, dev:sdb1
 disk 2, o:1, dev:sdc1
 disk 3, o:1, dev:sdd1
md: recovery of RAID array md6
md: minimum _guaranteed_  speed: 1000 KB/sec/disk.
md: using maximum available idle IO bandwidth (but not more than 200000
KB/sec) for recovery.
md: using 128k window, over a total of 156288256 blocks.
md: md6: recovery done.
RAID5 conf printout:
 --- rd:4 wd:4
 disk 0, o:1, dev:sda1
 disk 1, o:1, dev:sdb1
 disk 2, o:1, dev:sdc1
 disk 3, o:1, dev:sdd1
/dev/vmnet: open called by PID 2989 (vmware-vmx)
/dev/vmnet: port on hub 0 successfully opened
/dev/vmnet: open called by PID 2989 (vmware-vmx)
/dev/vmnet: port on hub 0 successfully opened
/dev/vmmon[2744]: host clock rate change request 1001 -> 1002
/dev/vmmon[2744]: host clock rate change request 1002 -> 83
/dev/vmmon[2744]: host clock rate change request 83 -> 1001
/dev/vmmon[2744]: host clock rate change request 1001 -> 1002
/dev/vmmon[2744]: host clock rate change request 1002 -> 1001
/dev/vmnet: open called by PID 2988 (vmware-vmx)
/dev/vmnet: port on hub 0 successfully opened
/dev/vmnet: open called by PID 2989 (vmware-vmx)
/dev/vmnet: port on hub 0 successfully opened kjournald starting.  Commit
interval 5 seconds
EXT3 FS on dm-0, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
ata3: command timeout
ata3: no sense translation for status: 0x40
ata3: translated ATA stat/err 0x40/00 to SCSI SK/ASC/ASCQ 0xb/00/00
ata3: status=0x40 { DriveReady }
sd 3:0:0:0: SCSI error: return code = 0x08000002
sdc: Current [descriptor]: sense key: Aborted Command
    Additional sense: No additional sense information Descriptor sense data
with sense descriptors (in hex):
        72 0b 00 00 00 00 00 0c 00 0a 80 00 00 00 00 00 
        00 00 00 01
end_request: I/O error, dev sdc, sector 260419647
raid5:md6: read error corrected (8 sectors at 260419584 on sdc1)
ata4: command timeout
ata4: no sense translation for status: 0x40
ata4: translated ATA stat/err 0x40/00 to SCSI SK/ASC/ASCQ 0xb/00/00
ata4: status=0x40 { DriveReady }
sd 4:0:0:0: SCSI error: return code = 0x08000002
sdd: Current [descriptor]: sense key: Aborted Command
    Additional sense: No additional sense information Descriptor sense data
with sense descriptors (in hex):
        72 0b 00 00 00 00 00 0c 00 0a 80 00 00 00 00 00 
        00 00 00 00
end_request: I/O error, dev sdd, sector 277596095


------------- END DMESG DUMP -------------


---------- Forwarded Message -----------
On Sun, 18 Feb 2007 07:13:28 -0500 (EST), Justin Piszcz wrote
> On Sun, 18 Feb 2007, Marc Marais wrote:
> 
> > On Sun, 18 Feb 2007 20:39:09 +1100, Neil Brown wrote
> >> On Sunday February 18, marcm@liquid-nexus.net wrote:
> >>> Ok, I understand the risks which is why I did a full backup before 
doing
> >>> this. I have subsequently recreated the array and restored my data from
> >>> backup.
> >>
> >> Could you still please tell me exactly what kernel/mdadm version you
> >> were using?
> >>
> >> Thanks,
> >> NeilBrown
> >
> > 2.6.20 with the patch you supplied in response to the "md6_raid5 crash
> > email" I posted in linux-raid a few days ago. Just as background, I 
replaced
> > the failing drive and at the same time bought an additional drive in 
order
> > to increase the array size.
> >
> > mdadm -V = v2.6 - 21 December 2006. Compiled under Debian (stable).
> >
> > Also, I've just noticed another drive failure with the new array with a
> > similar error to what happened during the grow operation (although on a
> > different drive) - I wonder if I should post this to linux-ide?
> >
> > Feb 18 00:58:10 xerces kernel: ata4: command timeout
> > Feb 18 00:58:10 xerces kernel: ata4: no sense translation for status: 
0x40
> > Feb 18 00:58:10 xerces kernel: ata4: translated ATA stat/err 0x40/00 to 
SCSI
> > SK/ASC/ASCQ 0xb/00/00
> > Feb 18 00:58:10 xerces kernel: ata4: status=0x40 { DriveReady }
> > Feb 18 00:58:10 xerces kernel: sd 4:0:0:0: SCSI error: return code =
> > 0x08000002
> > Feb 18 00:58:10 xerces kernel: sdd: Current [descriptor]: sense key: 
Aborted
> > Command
> > Feb 18 00:58:10 xerces kernel:     Additional sense: No additional sense
> > information
> > Feb 18 00:58:10 xerces kernel: Descriptor sense data with sense 
descriptors
> > (in hex):
> > Feb 18 00:58:10 xerces kernel:         72 0b 00 00 00 00 00 0c 00 0a 80 
00
> > 00 00 00 00
> > Feb 18 00:58:10 xerces kernel:         00 00 00 00
> > Feb 18 00:58:10 xerces kernel: end_request: I/O error, dev sdd, sector
> > 35666775
> > Feb 18 00:58:10 xerces kernel: raid5: Disk failure on sdd1, disabling
> > device. Operation continuing on 3 devices
> >
> > Regards,
> > Marc
> >
> > -
> > To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >
> 
> Just out of curiosity:
> 
> Feb 18 00:58:10 xerces kernel: end_request: I/O error, dev sdd,
>  sector 35666775
> 
> Can you run:
> 
> smartctl -d ata -t short /dev/sdd
> wait 5 min
> smartctl -d ata -t long /dev/sdd
> wait 2-3 hr
> smartctl -d ata -a /dev/sdd
> 
> And then e-mail that output to the list?
> 
> Justin.

I have smartmontools performing regular short and long scans but I will run 
the tests immediately and send the output of smartctl -a when done.

Note I'm getting similar errors on sdc too (as in 5 minutes ago). 
Interestingly the SMART error logs for sdc and sdd show no errors at all.

ata3: command timeout
ata3: no sense translation for status: 0x40
ata3: translated ATA stat/err 0x40/00 to SCSI SK/ASC/ASCQ 0xb/00/00
ata4: status=0x40 { DriveReady }
sd 3:0:0:0: SCSI error: return code = 0x08000002
sdd: Current [descriptor]: sense key: Aborted Command
     Additional sense: No additional sense information
Descriptor sense data with sense descriptors (in hex):
         72 0b 00 00 00 00 00 0c 00 0a 80 00 00 00 00 00
         00 00 00 00
end_request: I/O error, dev sdc, sector 260419647
raid5:md6: read error corrected (8 sectors at 260419584 on sdc1)

Will post logs when done...

Marc

--
------- End of Forwarded Message -------


--