Scsi errors with Megaraid 300-8x

Scsi errors with Megaraid 300-8x

[Johan Groth]

Hi,
ever since I upgraded my server from a dual Opteron 244 (mobo Tyan 2885) 
system to a dual dual-core Opteron 285 (mobo Tyan 2895) system, I'm 
getting read errors that freezes the system which leads to my disk based 
backup software stopped working (faubackup). I think it is faubackup 
that triggers the bug.

I get these errors in the log:
Aug 20 06:35:08 jaguar kernel: sd 2:1:0:0: SCSI error: return code = 0x40001
Aug 20 06:35:56 jaguar kernel: end_request: I/O error, dev sda, sector 
616924530
Aug 20 06:36:03 jaguar kernel: sd 2:1:0:0: SCSI error: return code = 0x40001
Aug 20 06:36:03 jaguar kernel: end_request: I/O error, dev sda, sector 
616924538
Aug 20 06:36:03 jaguar kernel: sd 2:1:0:0: SCSI error: return code = 0x40001
Aug 20 06:36:03 jaguar kernel: end_request: I/O error, dev sda, sector 
616924546
Aug 20 06:36:03 jaguar kernel: sd 2:1:0:0: SCSI error: return code = 0x40001
Aug 20 06:36:03 jaguar kernel: end_request: I/O error, dev sda, sector 
616924554
Aug 20 06:36:07 jaguar kernel: sd 2:1:0:0: SCSI error: return code = 0x40001
Aug 20 06:36:07 jaguar kernel: end_request: I/O error, dev sda, sector 
616924562
Aug 20 06:36:07 jaguar kernel: sd 2:1:0:0: SCSI error: return code = 0x40001
Aug 20 06:36:07 jaguar kernel: end_request: I/O error, dev sda, sector 
616924570
Aug 20 06:36:07 jaguar kernel: sd 2:1:0:0: SCSI error: return code = 0x40001
Aug 20 06:36:07 jaguar kernel: end_request: I/O error, dev sda, sector 
616924578
Aug 20 06:36:07 jaguar kernel: sd 2:1:0:0: SCSI error: return code = 0x40001
Aug 20 06:36:07 jaguar kernel: end_request: I/O error, dev sda, sector 
616924538

The last sector is repeated until I reboot the machine. The only 
difference I've made to the raid configuration is that sdc is now 2x250 
MB instead of 4x120MB, but that array is the target not the source (sda).
The raid HW is an LSI Megaraid 300-8x with the following configuration:

Host: scsi0 Channel: 01 Id: 00 Lun: 00
   Vendor: MegaRAID Model: LD 0 RAID0  312G Rev: 814D
   Type:   Direct-Access                    ANSI SCSI revision: 02
Host: scsi0 Channel: 01 Id: 01 Lun: 00
   Vendor: MegaRAID Model: LD 1 RAID0  312G Rev: 814D
   Type:   Direct-Access                    ANSI SCSI revision: 02
Host: scsi0 Channel: 01 Id: 02 Lun: 00
   Vendor: MegaRAID Model: LD 2 RAID0  474G Rev: 814D
   Type:   Direct-Access                    ANSI SCSI revision: 02

I'm running debian sid stock kernel 2.6.17.

Other hw changes that may, may not affect the kernel:
CPU: dual core, hence the kernel sees 4 cpus, before 2.
Removed a sound card, a CS46xx card and am using internal sound instead.
Installed a DVB-T Freeview card (digital TV).
More RAM, 2GB instead 1.

Should mention that I used to use 2.6.16 on the old system, but I've 
checked in the kernel tree and no changes has been made to the megaraid 
drivers since April, I think.

Can anyone help me?
Please, CC me as I'm not subscribed.

Regards,
Johan

Re: Scsi errors with Megaraid 300-8x

[Mark Lord]

Johan Groth wrote:
> Hi,
> ever since I upgraded my server from a dual Opteron 244 (mobo Tyan 2885) 
> system to a dual dual-core Opteron 285 (mobo Tyan 2895) system, I'm 
> getting read errors that freezes the system which leads to my disk based 
> backup software stopped working (faubackup). I think it is faubackup 
> that triggers the bug.
> 
> I get these errors in the log:
> Aug 20 06:35:08 jaguar kernel: sd 2:1:0:0: SCSI error: return code = 
> 0x40001
> Aug 20 06:35:56 jaguar kernel: end_request: I/O error, dev sda, sector 
> 616924530
> Aug 20 06:36:03 jaguar kernel: sd 2:1:0:0: SCSI error: return code = 
> 0x40001
> Aug 20 06:36:03 jaguar kernel: end_request: I/O error, dev sda, sector 
> 616924538
..
> Aug 20 06:36:07 jaguar kernel: sd 2:1:0:0: SCSI error: return code = 
> 0x40001
> Aug 20 06:36:07 jaguar kernel: end_request: I/O error, dev sda, sector 
> 616924538
> 
> The last sector is repeated until I reboot the machine. The only 
> difference I've made to the raid configuration is that sdc is now 2x250 
> MB instead of 4x120MB, but that array is the target not the source (sda).
> The raid HW is an LSI Megaraid 300-8x with the following configuration:
..

That looks like the classic SCSI bad-sectory non-recovery bug.
The code in scsi_lib.c, scsi_error.c, and sd.c is currently a
bit of a mess here.  

Basically, given an I/O request for 200 sectors, with a bad sector
in the middle at number 100, what SCSI will often do is fail sectors
number 1 through 100, one at a time, retrying the entire remainder of
the request after each attempt.  This takes hours, and results in no
data for the first 99 good sectors.

What it needs to do *instead*, is retry each sector individually,
rather than the entire request.  This would result in sectors 1..99
and 101..200 succeeding, and retries/failure only for sector 100.

A slight optimization would be to fail the bio size around sector 100,
rather than just the one sector.

I've got patches that do exactly this, and they work quite well.
But they're probably not "pretty enough" for inclusion.

Cheers

Re: Scsi errors with Megaraid 300-8x

[Johan Groth]

Mark Lord wrote:
> Johan Groth wrote:

[snip]

> Basically, given an I/O request for 200 sectors, with a bad sector
> in the middle at number 100, what SCSI will often do is fail sectors
> number 1 through 100, one at a time, retrying the entire remainder of
> the request after each attempt.  This takes hours, and results in no
> data for the first 99 good sectors.

So what you are saying is that after the move to a new box and a new 
mobo a sector has gone bad on that raid slice? Weird, as I was very 
careful this those drives when I moved them.

I mean, the raid controller is the same, the cpus are the same, just 
more of them, the pci-x bus the same so I didn't expect any problems at 
all.

I was also under the impression that SATA raid controllers work like 
SCSI raid controllers in the way that if a bad sector is encountered the 
controller moves what it can and the mark the sector as bad. I might be 
very wrong about that, though.

However, if I have a bad sector I would like to have that one marked as 
bad so the kernel never tries to read it again. Any suggestions how I do 
that. I assume I have to boot something like Knoppix as sda is my system 
disk.

Regards,
Johan

Re: Scsi errors with Megaraid 300-8x

[Justin Piszcz]

On Wed, 23 Aug 2006, Johan Groth wrote:

> Mark Lord wrote:
>> Johan Groth wrote:
>
> [snip]
>
>> Basically, given an I/O request for 200 sectors, with a bad sector
>> in the middle at number 100, what SCSI will often do is fail sectors
>> number 1 through 100, one at a time, retrying the entire remainder of
>> the request after each attempt.  This takes hours, and results in no
>> data for the first 99 good sectors.
>
> So what you are saying is that after the move to a new box and a new mobo a 
> sector has gone bad on that raid slice? Weird, as I was very careful this 
> those drives when I moved them.
>
> I mean, the raid controller is the same, the cpus are the same, just more of 
> them, the pci-x bus the same so I didn't expect any problems at all.
>
> I was also under the impression that SATA raid controllers work like SCSI 
> raid controllers in the way that if a bad sector is encountered the 
> controller moves what it can and the mark the sector as bad. I might be very 
> wrong about that, though.
>
> However, if I have a bad sector I would like to have that one marked as bad 
> so the kernel never tries to read it again. Any suggestions how I do that. I 
> assume I have to boot something like Knoppix as sda is my system disk.
>
> Regards,
> Johan
>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
>

Run badblocks in r+w mode on the bad disk and it will force the disk to 
re-allocate the bad sector if it can.

Justin.

Re: Scsi errors with Megaraid 300-8x

[Johan Groth]

Justin Piszcz wrote:
> Run badblocks in r+w mode on the bad disk and it will force the disk to 
> re-allocate the bad sector if it can.
> 
> Justin.

Is that possible to do in a non-destructive way? I don't want to loose 
all data and apparently I can't back it up either :(.

Regards,
Johan

Re: Scsi errors with Megaraid 300-8x

[Justin Piszcz]

On Wed, 23 Aug 2006, Johan Groth wrote:

> Justin Piszcz wrote:
>> Run badblocks in r+w mode on the bad disk and it will force the disk to 
>> re-allocate the bad sector if it can.
>> 
>> Justin.
>
> Is that possible to do in a non-destructive way? I don't want to loose all 
> data and apparently I can't back it up either :(.
>
> Regards,
> Johan
>

Nope, r+w will write over everything on the disk, but I have found -the- 
most effective way to see if a disk is good or not.  I'd rather have the 
disk die to that test rather than using it in production and finding it 
dies with my data on it.

Justin.

Re: Scsi errors with Megaraid 300-8x

[Johan Groth]

Justin Piszcz wrote:
> 
> Nope, r+w will write over everything on the disk, but I have found -the- 
> most effective way to see if a disk is good or not.  I'd rather have the 
> disk die to that test rather than using it in production and finding it 
> dies with my data on it.
> 

Hmm, we both should read the man page of badblocks a bit better :).
I found this:

-n     Use non-destructive read-write mode.  By default only a 
non-destructive read-only test is done. This option must not be combined 
with the -w option, as they are mutually exclusive.


Cheers,
Johan

Re: Scsi errors with Megaraid 300-8x

[Justin Piszcz]

On Wed, 23 Aug 2006, Johan Groth wrote:

> Justin Piszcz wrote:
>> 
>> Nope, r+w will write over everything on the disk, but I have found -the- 
>> most effective way to see if a disk is good or not.  I'd rather have the 
>> disk die to that test rather than using it in production and finding it 
>> dies with my data on it.
>> 
>
> Hmm, we both should read the man page of badblocks a bit better :).
> I found this:
>
> -n     Use non-destructive read-write mode.  By default only a 
> non-destructive read-only test is done. This option must not be combined with 
> the -w option, as they are mutually exclusive.
>
>
> Cheers,
> Johan
>

I have not tested that option.  I wonder if it is as good as a real R+W 
mode.  What does smartctl -a /dev/sda say on the disk that you are having 
problems with?

Re: Scsi errors with Megaraid 300-8x

[Mark Lord]

Johan Groth wrote:
> Justin Piszcz wrote:
>> Run badblocks in r+w mode on the bad disk and it will force the disk 
>> to re-allocate the bad sector if it can.
>>
>> Justin.
> 
> Is that possible to do in a non-destructive way? I don't want to loose 
> all data and apparently I can't back it up either :(.

Yes, it is perfectly doable, but I don't think anyone has yet bothered
to release a utility that actually does it.

OPPORTUNITY FOR FAME AND FORTUNE! (okay, maybe just some fame):
=================================
Hack the existing smartctl code to read out the failed sector numbers,
and then issue single-sector read-overwrite to each of those bad sectors.

Very simple code.  I'll do it myself eventually, but please beat me to it!

Cheers

Re: Scsi errors with Megaraid 300-8x

[Johan Groth]

Mark Lord wrote:
> Johan Groth wrote:
>> Justin Piszcz wrote:
>>> Run badblocks in r+w mode on the bad disk and it will force the disk 
>>> to re-allocate the bad sector if it can.
>>>
>>> Justin.
>>
>> Is that possible to do in a non-destructive way? I don't want to loose 
>> all data and apparently I can't back it up either :(.
> 
> Yes, it is perfectly doable, but I don't think anyone has yet bothered
> to release a utility that actually does it.
> 
> OPPORTUNITY FOR FAME AND FORTUNE! (okay, maybe just some fame):
> =================================
> Hack the existing smartctl code to read out the failed sector numbers,
> and then issue single-sector read-overwrite to each of those bad sectors.
> 
> Very simple code.  I'll do it myself eventually, but please beat me to it!

Authors of badblocks already has with the -n option :). When I ran 
badblocks on the entire partition it wanted to check over 210 million 
blocks and when it finally came to the bad sector parts the controller 
lost the drive and the kernel started to spit out scsi errors! Buggy 
driver, hardware error? God knows. Unfortunately I don't have a log to 
show you as I was in single user mode.

I would like to run badblocks again but only around the damaged part. 
Thing is that I know which sector the kernel thinks is bad but badblocks 
wants to know which block to start and end at. How do I convert a sector 
number to block number. The partition is a standard XFS partition, ie I 
haven't made any changes to block sizes when I formatted it.

Got this from xfs_info:
meta-data=/dev/sda7              isize=256    agcount=16, agsize=3433014 
blks
          =                       sectsz=512   attr=1
data     =                       bsize=4096   blocks=54928224, imaxpct=25
          =                       sunit=0      swidth=0 blks, unwritten=1
naming   =version 2              bsize=4096
log      =internal               bsize=4096   blocks=26820, version=1
          =                       sectsz=512   sunit=0 blks
realtime =none                   extsz=65536  blocks=0, rtextents=0

There's another thing I would like know. How do I find out what file a 
sector belongs to?

Regards,
Johan

Re: Scsi errors with Megaraid 300-8x

[Mark Lord]

Johan Groth wrote:
> Mark Lord wrote:
>
>> OPPORTUNITY FOR FAME AND FORTUNE! (okay, maybe just some fame):
>> =================================
>> Hack the existing smartctl code to read out the failed sector numbers,
>> and then issue single-sector read-overwrite to each of those bad sectors.
>>
>> Very simple code.  I'll do it myself eventually, but please beat me to 
>> it!
> 
> Authors of badblocks already has with the -n option :)

Not quite.  As you pointed out:

> I would like to run badblocks again but only around the damaged part. 

The drive *knows* which sectors are bad -- it's mostly all in the S.M.A.R.T.
logs and such.  Which smartctl already knows how to read.

So now we just need a script-kiddie to do a nice little awk script for you,
which extracts the bad sectors info from the output of smartctl, and then
feeds this as input to badblocks.

Cheers