when does it become faulty disk

when does it become faulty disk

[Raz Ben-Jehuda(caro)]

I have managed to make the kernel remove a disk 
from my raid even if this raid is "/" . I did it by adding a line 
in ata_scsi_error that remove the ata disk from the raid array.
This means that when the first error ocurrs on a disk It is removed 
from the array. 
Well, this is not the best thing to do..
Question is :
When does a disk become faulty ?
Is when you have N errors in T time ?
New ideas would be welcomed.

-- 
Raz
Long Live the Penguin

Re: when does it become faulty disk

[Molle Bestefich]

Raz Ben-Jehuda(caro) wrote:
> I have managed to make the kernel remove a disk
> from my raid even if this raid is "/" . I did it by adding a line
> in ata_scsi_error that remove the ata disk from the raid array.
> This means that when the first error ocurrs on a disk It is removed
> from the array.
> Well, this is not the best thing to do..
> Question is :
> When does a disk become faulty ?

When trying to read sectors from a disk and the disk fails the read:
 1.)  Read the data from the other disks in the RAID and
 2.)  Overwrite the sectors where the read error occur.
If this write also fails, then the disk has used up it's spare sectors
area.  The RAID array is now by definition in a degraded state since
that sector no longer exists in a redundant (readable at least) way. 
The disk should therefore be kicked, in order to notify the user that
it should be replaced immediately.

> Is when you have N errors in T time ?

Nah, it's when you run out of spare sectors and your data redundancy
is thereby lost that you have to fault the disk to prevent future data
loss.  Don't try to second-guess when the disk is going to get faulty
based on how many errors occurred.  If you want to do something like
that, read out the SMART data from the disk.  The manufacturer's data
about the disks health should be your data source.

> New ideas would be welcomed.

HTH..

Re: when does it become faulty disk

[raz ben jehuda]

So, what about write errors ?
from what you are saying i understand that when a write error occurs
the disk is faulty.


On Sun, 2005-06-19 at 22:10, Molle Bestefich wrote:
> Raz Ben-Jehuda(caro) wrote:
> > I have managed to make the kernel remove a disk
> > from my raid even if this raid is "/" . I did it by adding a line
> > in ata_scsi_error that remove the ata disk from the raid array.
> > This means that when the first error ocurrs on a disk It is removed
> > from the array.
> > Well, this is not the best thing to do..
> > Question is :
> > When does a disk become faulty ?
> 
> When trying to read sectors from a disk and the disk fails the read:
>  1.)  Read the data from the other disks in the RAID and
>  2.)  Overwrite the sectors where the read error occur.
> If this write also fails, then the disk has used up it's spare sectors
> area.  The RAID array is now by definition in a degraded state since
> that sector no longer exists in a redundant (readable at least) way. 
> The disk should therefore be kicked, in order to notify the user that
> it should be replaced immediately.


> > Is when you have N errors in T time ?
> 
> Nah, it's when you run out of spare sectors and your data redundancy
> is thereby lost that you have to fault the disk to prevent future data
> loss.  Don't try to second-guess when the disk is going to get faulty
> based on how many errors occurred.  If you want to do something like
> that, read out the SMART data from the disk.  The manufacturer's data
> about the disks health should be your data source.
> 
> > New ideas would be welcomed.
> 
> HTH..
> -
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
-- 
Raz
Long Live The Penguin

Re: when does it become faulty disk

[Molle Bestefich]

raz ben jehuda wrote:
> So, what about write errors ?
> from what you are saying i understand that when a write error occurs
> the disk is faulty.

Yes..  If you are serious about your data redundancy, yes.

A sector _read_ error is a notification from the disk that a sector
has gone bad and that some particular data is lost.

A sector _write_ error is the disk telling you that:
 1. The sector has gone bad.
 2. The disk failed to relocate the sector to the spare area, probably
because it's full.

The above are slight simplifications, since other kinds of read and
write errors may in very rare cases occur.  That's OK though, since
you DO want sick disks with strange internal errors that are causing
read or write errors to get kicked.

In rare cases a disk could get sick in a way where writes to a bad
sector succeeds but subsequent reads fail.  Never seen it happen...
But just in case, you might want to re-read a failed sector after you
have written to it, just to check that the disk actually correctly
relocated it.

Once a disk has been kicked, you may want to instruct the user to
check that the disk's spare sector count has indeed reached 0, by
using smartctl -a /dev/xxx.  That command will also tell of other disk
failures.

Re: when does it become faulty disk

[raz ben jehuda]

thank you molle.
This was a very helpful information.
On Mon, 2005-06-20 at 10:55, Molle Bestefich wrote:
> raz ben jehuda wrote:
> > So, what about write errors ?
> > from what you are saying i understand that when a write error occurs
> > the disk is faulty.
> 
> Yes..  If you are serious about your data redundancy, yes.
> 
> A sector _read_ error is a notification from the disk that a sector
> has gone bad and that some particular data is lost.
> 
> A sector _write_ error is the disk telling you that:
>  1. The sector has gone bad.
>  2. The disk failed to relocate the sector to the spare area, probably
> because it's full.
> 
> The above are slight simplifications, since other kinds of read and
> write errors may in very rare cases occur.  That's OK though, since
> you DO want sick disks with strange internal errors that are causing
> read or write errors to get kicked.
> 
> In rare cases a disk could get sick in a way where writes to a bad
> sector succeeds but subsequent reads fail.  Never seen it happen...
> But just in case, you might want to re-read a failed sector after you
> have written to it, just to check that the disk actually correctly
> relocated it.
> 
> Once a disk has been kicked, you may want to instruct the user to
> check that the disk's spare sector count has indeed reached 0, by
> using smartctl -a /dev/xxx.  That command will also tell of other disk
> failures.
-- 
Raz
Long Live The Penguin

Re: when does it become faulty disk

[Michael Tokarev]

Molle Bestefich wrote:
> Raz Ben-Jehuda(caro) wrote:
> 
>>I have managed to make the kernel remove a disk
>>from my raid even if this raid is "/" . I did it by adding a line
>>in ata_scsi_error that remove the ata disk from the raid array.
>>This means that when the first error ocurrs on a disk It is removed
>>from the array.
>>Well, this is not the best thing to do..
>>Question is :
>>When does a disk become faulty ?
> 
> 
> When trying to read sectors from a disk and the disk fails the read:
>  1.)  Read the data from the other disks in the RAID and
>  2.)  Overwrite the sectors where the read error occur.

Note: this is NOT how current linux softraid code works, it's
how it *supposed* to work.  And right now, linux raid code
kicks a drive out of the array after *any* error (read or
write), without trying to "understand" what happened.

/mjt

Re: when does it become faulty disk

[raz ben jehuda]

On Mon, 2005-06-20 at 16:45, Michael Tokarev wrote:
> Molle Bestefich wrote:
> > Raz Ben-Jehuda(caro) wrote:
> > 
> >>I have managed to make the kernel remove a disk
> >>from my raid even if this raid is "/" . I did it by adding a line
> >>in ata_scsi_error that remove the ata disk from the raid array.
> >>This means that when the first error ocurrs on a disk It is removed
> >>from the array.
> >>Well, this is not the best thing to do..
> >>Question is :
> >>When does a disk become faulty ?
> > 
> > 
> > When trying to read sectors from a disk and the disk fails the read:
> >  1.)  Read the data from the other disks in the RAID and
> >  2.)  Overwrite the sectors where the read error occur.
> 
> Note: this is NOT how current linux softraid code works, it's
> how it *supposed* to work.  And right now, linux raid code
> kicks a drive out of the array after *any* error (read or
> write), without trying to "understand" what happened.
no accurate.
I have raid1 over "/" and whenever I pull out one of the disks
the system hangs. in order to fix it I added to ata_scsi_error()
ata_port_disable(). 
> /mjt
> -
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
-- 
Raz
Long Live The Penguin

More tales of horror from the linux (HW) raid crypt

[Harry Mangalam]

Hi All,

From the traffic, this list seems to be heavily slanted towards the SW aspect 
of Linux RAID, but there have been a few postings (other than mine) about the 
HW aspects of it.  So, apologies for the verbarea on the HW aspects, but at 
least a few people have told me that this running monologue of raid troubles 
has been useful, so herein, some more.  If I'm reiterating what is part of a 
FAQ, please let me know, but I read a lot of them and didn't stumble across 
much of this.


short version:  test ALL your disks before you use them, especially in a RAID 
set, especially the 'recertified' ones.

long version: An FYI for anyone who uses disks in their computers... and 
definitely for those thinking of setting up either software or hardware raids

I refer you to previous posts to this list about the detailed background, but 
it's briefly alluded to at the end of this list - more failures of the HW 
RAID on a dual opteron running ubuntu linux amd64-k8-SMP.

I pulled the 3ware controller and used the onboard Silicon Image controller to 
run diagnostic SMART tests on all the 'recertified' SD-series disks that came 
back from WD.  It's probably possible to use the 3ware controller (someone of 
the linux raid list indicated he did so), but I wanted the 3ware controller 
out of the loop because we suspected it as well.  I used the linux command 
'smartctl -t long /dev/sdx (where x=a-d).  smartctl is part of the 
smartmontools package and can be used to test SATA disks as well as PATA 
disks (altho I've been told that the kernel has to be patched to do this - 
I'm using a Ubuntu-supplied kernel which works out-of-the-box).  The long test 
lasts about 90 minutes for a 250GB disk and can be performed in parallel on 
each disk.

 5 (FIVE!) of them (out of 9 returned from WD) failed that test - either they 
already had logged errors (SMART devices store their last 5 errors to their 
onboard memory), or they failed the short test (~2 minutes) or they failed 
the long test with unrecoverable errors.  They're on their way back to WD for 
other replacements

However, these are a superset of the disks that the 3ware controller failed on 
when being used for an array (see the message below) - I now think that the 
problem is either with the power supply (possible, but unlikely) or the disks 
(definitely), as well as the hotswap cages (definitely).  I'm pretty sure 
that the controller is fine - it's been running with 5 disks in RAID5 for 
several days now with no errors or warnings at all.

That makes me extremely suspicious of WD's 'recertified' drives, but that's 
the only avenue we have to get replacements right now.  And I'll be dang sure 
to test ALL of them before I store data on them.

I do have to reiterate, now that I've been running bonnie++ on both the SW 
RAID5 (on 4 disks - all that the onboard controller would control) and on the 
3ware-controlled RAID5 that the SW is slightly faster and actually seems to 
use about as much CPU time as the 3ware in these tests.  It's also more 
flexible in terms of how you set up and partition the devices.  It's also so 
MUCH cheaper - using the onboard SATA controller and a $20 4 port SATA 
controller, I could control the same number of disks as the 3ware (8) but the 
3ware costs $500.  The big advantage of the 3ware controller is (relative) 
simplicity.  Plug in controller, plug in disks, hit power switch, go into 
3ware BIOS, allocate disks to a RAID unit, boot the OS, make a filesystem on 
the /dev/sdx and mount it.  You can set/get some basic configuration and 
information from the 3ware utilities but not to the extent that you can with 
mdadm and the related utils.


post below is to a 3ware support tech.

> > As a reminder of the system, it's currently
> > - dual opteron IWILL DK8X Mobo, Gb ethernet
> > - Silicon image 4port SATA controller onboard (now disabled),
> > - 3ware 9500S 8 port card running 8x250GB WD 2500SD disks in RAID5.
> > - Disks are in 2x 4slot Chenbro hotswap RAID cages.
> > - running kubuntu Linux in pure 64bit mode (altho bringing up KDE
> >   currently locks the system in some configurations)
> > - using kernel image 2.6.11-1-amd64-k8-smp as a kubuntu debian install
> >     (NOT custom-compiled)
> > - OS is running from a separate WD 200GB IDE disk
> >     (which recently bonked at 3 months old, replaced by WD without
> > complaint.) - on an APC UPS (runnning apcupsd communicating thru a usb
> > cable)
> >
> > The 9500 that you sent to me was put into service as soon as we got
> > enough SD disks to make a raid5 - 3 of them, on ports 0-2, in the 1st
> > hotswap cage.
> >
> > During that time, the array stayed up and seemed to be stable over about
> > 1 week of heavy testing.  Once we got all the disks replace with SD
> > disks, I set it up as 8 disks in a RAID5 and things seemed to be fine for
> > about a day. Then, the disk on port 3 had problems.  I replaced it and
> > again it appeared to go bad.  I then disconnected it from the hotswap
> > cage and connected it directly to the controller.  That seemed to solve
> > that problem, so there definitely is a problem with one hotswap cage -
> > it's being replaced.
> >
> >
> > However after that incident, there have been 3 more with disks on the
> > other hotswap cage, on different ports, one of port 6 (4 warnings of:
> > Sector repair completed: port=6, LBA=0x622CE39, and then the error:
> > Degraded unit detected: unit=0, port=6.  I wasn't sure if it was a
> > seating error or a real disk error, so I pulled the disk and re-seated it
> > (and the controller accepted it fine) but then after it rebuilt the
> > array, it failed again on that port.  OK, I replaced the drive.  Then
> > port 7 reported:(0x04:0x0023): Sector repair completed: port=7,
> > LBA=0x2062320
> >
> > I started a series of copy and read/write tests to make sure the array
> > was stable under load, and then just as the array filled up, it failed
> > again, this time again on port 3: (0x04:0x0002): Degraded unit detected:
> > unit=0, port=3 (this port is connected directly to the controller).
> >
> > And this morning, I saw that yet another drive looks like it has failed
> > or at least is unresponsive:(0x04:0x0009): Drive timeout detected: port=5
> >
> > Discounting the incidents that seem to be related to the bad hotswap
> > cage, that's still 4 disks (with MTBF of 1Mhr) that have gone bad in 2
> > days.
> >
> > I then connected all the disks directly to the controller to remove all
> > hotswap cage influence, and the disk on port 3 almost immediately was
> > marked bad - I have to say that this again sounds like a controller
> > problem.  An amazing statistical convergence of random disk failures?
> > Electrical failure (the system is on a relatively good APC UPS (SMART UPS
> > 1000), so the voltage supply should be good, and no other problems have
> > been seen.  I guess I could throw the Power supply on a scope to see if
> > it's stable, but there have been no other such glitches (unless it's an
> > uneven power supply that is causing the disks to die).
> >
> > Currently most of the disks that were marked bad by the 3ware controller
> > are being tested under the onboard silicon image controller in a raid5
> > config. I'll test over the weekend to see what they do.

At this point I tested the disks using the smartctl, and found out the bad 
ones.  The SW RAID stayed up without errors until I brought it down to 
install the 3ware controller.

-- 
Cheers, Harry
Harry J Mangalam - 949 856 2847 (vox; email for fax) - hjm@tacgi.com 
            <<plain text preferred>>
_______________________________________________
List-Info: https://maillists.uci.edu/mailman/listinfo/uci-linux
-- 
Cheers, Harry
Harry J Mangalam - 949 856 2847 (vox; email for fax) - hjm@tacgi.com 
            <<plain text preferred>>

Re: More tales of horror from the linux (HW) raid crypt

[Mike Hardy]

Not that I wish bad luck on anyone, but I always enjoy reading about
problems others have had and how they solved them, so I don't have to
learn the hard way. Hopefully your situation gets sorted out.

I can only second (and third, etc) the motion to do SMART tests on all
drives before putting them in service, and add that you should really do
a short test daily and a long test at least weekly if possibly.
Basically you just can't trust consumer drives at all these days.
smartmontools and rsync are probably my most-loved open source packages
these days. I usually get 1 out of 10 bad out of the box now (some
remappable at least) and a handful then fail quickly too. Most of them
haven't gone three years so I can't say if they fail completely, but
they seem to be lasting ok with occasional bad blocks.

I'm very interested in the relative SW raid / HW raid performance. I
have both in service (two raid 5 sets are actually the same size with
the same number of components) and see roughly the same as you mention.
One difference that I see is that HW raid should generate fewer
interrupts and lower bus traffic.

In the one area I used HW raid (a 3ware 8 port PATA, 8x250GB Maxtor,
2xOpteron), it was specifically because the motherboard chipset (or its
interaction with Linux at least) was crap, and couldn't handle the
interrupt load under bonnie++. So this could be a factor.

It also goes to show that burning the machine in (with bonnie++ or
similar) is a very good step. At least you catch these things before
they're in service...

Anyway, good luck with the new drives.

-Mike

Harry Mangalam wrote:
> Hi All,
> 
>>From the traffic, this list seems to be heavily slanted towards the SW aspect 
> of Linux RAID, but there have been a few postings (other than mine) about the 
> HW aspects of it.  So, apologies for the verbarea on the HW aspects, but at 
> least a few people have told me that this running monologue of raid troubles 
> has been useful, so herein, some more.  If I'm reiterating what is part of a 
> FAQ, please let me know, but I read a lot of them and didn't stumble across 
> much of this.
> 
> 
> short version:  test ALL your disks before you use them, especially in a RAID 
> set, especially the 'recertified' ones.

Re: More tales of horror from the linux (HW) raid crypt

[Harry Mangalam]

Thanks for the note - I'd also be interested in what drives people people use 
that seem to be robust and which are having problems.  The 3ware tech that I 
spoke to recommended the WD-SD series, but a local senior sysadmin has warned 
his entire dept off WD drives for the reasons I've mentioned.  He rec'ed 
Seagate (5 year warr) and Hitachi.  So far the only good thing that I can say 
about WD is that their return policy has been relatively smooth (tho they 
smoothly keep returning bad drives :( )

I personally buy IBM (now Hitachi I think) and Seagate and have always avoided 
WD as being cheap and cheesy.  5 years ago when I was a sysadmin, I was 
replacing WD at probably 5 times the rate of anything else.  It was probably 
strongly influenced by the fact that more people were using them than the 
others but I don't think I've ever had an IBM or seagate 3.5" die (have a 
teetering stack of working ones that just got too small to bother with).  I 
figured that since WD was still in business (tho seeming always on the verge 
of going bust) their quality must have improved, but maybe not.

Happy to be contradicted.  If there's a 'Consumer's Reports' for HD 
reliability on the web, I couldn't find it.  Maybe it's time to set one up - 
with SMART disk records, it's possible to get more reliable runtime records 
and data than just the usual single point or small data set that individuals 
can provide (and you can bet that most vendors aren't going to make their 
failures public..).

Perhaps something like the CDDB, where a you can run an applet on a cronjob 
that will upload your disk's SMART data cache to a remote DB on a regular 
basis, so the millions of disks out there can be profiled.  Hmmm - sounds 
like a good undergrad project. Did someone say Summer of Code? 
(http://code.google.com/summerofcode.html)

hjm




On Wednesday 22 June 2005 12:33 pm, Mike Hardy wrote:
> Not that I wish bad luck on anyone, but I always enjoy reading about
> problems others have had and how they solved them, so I don't have to
> learn the hard way. Hopefully your situation gets sorted out.

-- 
Cheers, Harry
Harry J Mangalam - 949 856 2847 (vox; email for fax) - hjm@tacgi.com 
            <<plain text preferred>>

Re: More tales of horror from the linux (HW) raid crypt

[Jure Pecar]

On Wed, 22 Jun 2005 13:16:33 -0700
Harry Mangalam <hjm@tacgi.com> wrote:

> Perhaps something like the CDDB, where a you can run an applet on a
> cronjob  that will upload your disk's SMART data cache to a remote DB on a
> regular  basis, so the millions of disks out there can be profiled.  Hmmm
> - sounds  like a good undergrad project. Did someone say Summer of Code? 
> (http://code.google.com/summerofcode.html)

Actually if you look at the storagereview.com, they already have a crude
form of this online under "reliability survey". You have to login, then you
can enter disk model that you have expirience with and what were those
expiriences. There's quite some info already available ... 

-- 

Jure Pečar
http://jure.pecar.org/
-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: More tales of horror from the linux (HW) raid crypt

[Dan Stromberg]

[-- Attachment #1: Type: text/plain, Size: 1146 bytes --]

On Wed, 2005-06-22 at 12:33 -0700, Mike Hardy wrote:

> I'm very interested in the relative SW raid / HW raid performance. I
> have both in service (two raid 5 sets are actually the same size with
> the same number of components) and see roughly the same as you mention.
> One difference that I see is that HW raid should generate fewer
> interrupts and lower bus traffic.

In the early days of RAID, people always used to say that for speed, you
had to get a hardware RAID controller instead of doing software RAID.

However, I saw an interesting comment on a local linux user group
mailing list recently.  That comment was to the effect that hardware
RAID controllers tend to have no where near the horsepower of a modern
desktop CPU - so the claim was (I've not verified this!) that using
software RAID would be much faster.

I'm thinking that if you can do software RAID with a dedicated box
that's doing nothing but RAID and a little network, then yes, that
likely would be faster than some scaled-down hardware RAID controller -
but it would also likely cost more, be less likely to have good hot
swap, and so on.



[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: More tales of horror from the linux (HW) raid crypt

[Brad Dameron]

On Wed, 2005-06-22 at 13:16 -0700, Harry Mangalam wrote:
> Thanks for the note - I'd also be interested in what drives people people use 
> that seem to be robust and which are having problems.  The 3ware tech that I 
> spoke to recommended the WD-SD series, but a local senior sysadmin has warned 
> his entire dept off WD drives for the reasons I've mentioned.  He rec'ed 
> Seagate (5 year warr) and Hitachi.  So far the only good thing that I can say 
> about WD is that their return policy has been relatively smooth (tho they 
> smoothly keep returning bad drives :( )
> 
> I personally buy IBM (now Hitachi I think) and Seagate and have always avoided 
> WD as being cheap and cheesy.  5 years ago when I was a sysadmin, I was 
> replacing WD at probably 5 times the rate of anything else.  It was probably 
> strongly influenced by the fact that more people were using them than the 
> others but I don't think I've ever had an IBM or seagate 3.5" die (have a 
> teetering stack of working ones that just got too small to bother with).  I 
> figured that since WD was still in business (tho seeming always on the verge 
> of going bust) their quality must have improved, but maybe not.
> 
> Happy to be contradicted.  If there's a 'Consumer's Reports' for HD 
> reliability on the web, I couldn't find it.  Maybe it's time to set one up - 
> with SMART disk records, it's possible to get more reliable runtime records 
> and data than just the usual single point or small data set that individuals 
> can provide (and you can bet that most vendors aren't going to make their 
> failures public..).
> 
> Perhaps something like the CDDB, where a you can run an applet on a cronjob 
> that will upload your disk's SMART data cache to a remote DB on a regular 
> basis, so the millions of disks out there can be profiled.  Hmmm - sounds 
> like a good undergrad project. Did someone say Summer of Code? 
> (http://code.google.com/summerofcode.html)
> 
> hjm

Everyone has had issues with one brand or another. Sometimes the
shipping can make all the difference. As I have had 4-5 out of 16 drives
shipped come up bad out of the box. And these were Seagate 73GB SCSI
drives. Not some cheap SATA/PATA drives. Now when you come to comparing
drives you have to remember that not all manufacturers make their own
drives. It is know that WD has sold IBM drives as theirs and vice-versa.
I currently have a 12 drive 3ware 9500S-12 with 73GB WD 10k raptor
drives in it that has been online for a year and a month without issues.
And the controller has the very first firmware and linux drivers out. 
I also have several hundred Seagate SCSI 10k drives from 36gb to 146gb. 
And a few dozen Fujitsu 15k 73gb SCSI drives. And out of all these
drives Seagate is the only ones I have to keep sending in for RMA. Now
mind you I do have about 5 times as many drives. 

I used to not be partial to Maxtor either due to a issue several years
ago with their PATA drives dieing on me in bulk. But I am currently
purchasing 16 of their Maxline III SATA 300GB drives for a large storage
array. Why am I going with them? Because they have the best reported
performance for this type of drive. Not because they have a record of
lasting longer. It is too hard to tell in a production/semi-production
environment which brand will last the longest. So go with what you think
is the best. And have spares/backups and monitoring programs to be
prepared for when one does die. The odds of multiple failures at the
same time are pretty high. 

And just for the record, look at the Areca SATA RAID cards. They have 16
and even 24 port cards if you want very large arrays. The cards greatly
out perform the 3ware cards in READ/WRITE performance. They even offer
RAID6. Just look at the processor on the ARC-12xx series.

Just my 2 cents. Hope it helps.

Brad Dameron
SeaTab Software
www.seatab.com

Re: More tales of horror from the linux (HW) raid crypt

[Brad Dameron]

On Wed, 2005-06-22 at 13:54 -0700, Dan Stromberg wrote:
> On Wed, 2005-06-22 at 12:33 -0700, Mike Hardy wrote:
> 
> > I'm very interested in the relative SW raid / HW raid performance. I
> > have both in service (two raid 5 sets are actually the same size with
> > the same number of components) and see roughly the same as you mention.
> > One difference that I see is that HW raid should generate fewer
> > interrupts and lower bus traffic.
> 
> In the early days of RAID, people always used to say that for speed, you
> had to get a hardware RAID controller instead of doing software RAID.
> 
> However, I saw an interesting comment on a local linux user group
> mailing list recently.  That comment was to the effect that hardware
> RAID controllers tend to have no where near the horsepower of a modern
> desktop CPU - so the claim was (I've not verified this!) that using
> software RAID would be much faster.
> 
> I'm thinking that if you can do software RAID with a dedicated box
> that's doing nothing but RAID and a little network, then yes, that
> likely would be faster than some scaled-down hardware RAID controller -
> but it would also likely cost more, be less likely to have good hot
> swap, and so on.


This is very true. Especially with 64bit machines. The software RAID is
much faster and efficient. However with servers that are using a lot of
CPU resources it can become a bottleneck. Our inhouse application is
proven faster with hardware RAID than with software RAID. Mainly due to
the 100% CPU utilization. It being able to offload the IO to the
MegaRAID card helps. I can't wait to test it with the new 320-2E. The
CPU is twice as fast as the 320-2X we use now.

Brad Dameron
SeaTab Software
www.seatab.com

Re: More tales of horror from the linux (HW) raid crypt

[Harry Mangalam]

Is this too OT? Let me know..

The Storage Review DB would be a decent 1st approximation (includes data on 
~35K drives), but relies on entering the info on only drives that the DB 
knows about, and also relies on info that the person entering the data 
remembers - how many hours in use, etc. It's also available only to 
registered users who might be entering spurious data to gain access to the 
DB.

The SMART data (granted, it would only be possible to collect data from SMART 
drives) would tell you considerably more info:
(including that I was mistaken when I previously said I only bought IBM's and 
Seagates) :)

This is the kind of info that wold be quite useful to have available in a 
CDDB-like DB.  It would also point out which drives are susceptible to 
failure under continuous-on conditions versus frequent power-cycling, etc.

an example from my home system:

1062 $ smartctl -a /dev/hda
smartctl version 5.32 Copyright (C) 2002-4 Bruce Allen
Home page is http://smartmontools.sourceforge.net/

=== START OF INFORMATION SECTION ===
Device Model:     WDC WD800JB-00CRA1
Serial Number:    WD-WMA8E4088773
Firmware Version: 17.07W17
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   5
ATA Standard is:  Exact ATA specification draft version not indicated
Local Time is:    Wed Jun 22 14:24:56 2005 PDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x84) Offline data collection activity
                                        was suspended by an interrupting 
command from host.
                                        Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0) The previous self-test routine 
completed
                                        without error or no self-test has ever
                                        been run.
Total time to complete Offline
data collection:                 (3120) seconds.
Offline data collection
capabilities:                    (0x3b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off 
support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        Conveyance Self-test supported.
                                        No Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        No General Purpose Logging support.
Short self-test routine
recommended polling time:        (   2) minutes.
Extended self-test routine
recommended polling time:        (  58) minutes.
Conveyance self-test routine
recommended polling time:        (   5) minutes.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  
WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000b   200   200   051    Pre-fail  Always       
-       0
  3 Spin_Up_Time            0x0007   099   095   021    Pre-fail  Always       
-       4141
  4 Start_Stop_Count        0x0032   100   100   040    Old_age   Always       
-       122
  5 Reallocated_Sector_Ct   0x0033   199   199   140    Pre-fail  Always       
-       2
  7 Seek_Error_Rate         0x000b   200   200   051    Pre-fail  Always       
-       0
  9 Power_On_Hours          0x0032   075   075   000    Old_age   Always       
-       18940
 10 Spin_Retry_Count        0x0013   100   100   051    Pre-fail  Always       
-       0
 11 Calibration_Retry_Count 0x0013   100   100   051    Pre-fail  Always       
-       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       
-       120
196 Reallocated_Event_Count 0x0032   198   198   000    Old_age   Always       
-       2
197 Current_Pending_Sector  0x0012   200   200   000    Old_age   Always       
-       0
198 Offline_Uncorrectable   0x0012   200   200   000    Old_age   Always       
-       0
199 UDMA_CRC_Error_Count    0x000a   200   253   000    Old_age   Always       
-       0
200 Multi_Zone_Error_Rate   0x0009   200   200   051    Pre-fail  Offline      
-       0

SMART Error Log Version: 1
No Errors Logged


On Wednesday 22 June 2005 1:38 pm, Jure Pecar wrote:
> On Wed, 22 Jun 2005 13:16:33 -0700
>
> Harry Mangalam <hjm@tacgi.com> wrote:
> > Perhaps something like the CDDB, where a you can run an applet on a
> > cronjob  that will upload your disk's SMART data cache to a remote DB on
> > a regular  basis, so the millions of disks out there can be profiled. 
> > Hmmm - sounds  like a good undergrad project. Did someone say Summer of
> > Code? (http://code.google.com/summerofcode.html)
>
> Actually if you look at the storagereview.com, they already have a crude
> form of this online under "reliability survey". You have to login, then you
> can enter disk model that you have expirience with and what were those
> expiriences. There's quite some info already available ...

-- 
Cheers, Harry
Harry J Mangalam - 949 856 2847 (vox; email for fax) - hjm@tacgi.com 
            <<plain text preferred>>

Re: More tales of horror from the linux (HW) raid crypt

[Harry Mangalam]

Good points and good info; I'll look into the areca products a bit more. Just 
yesterday I looked at the 16 tray Chenbro/Areca combo:
 http://www.chenbro.com.tw/Chenbro_Newsletter/Vol21_rm321.htm
I'll read up on the Areca controllers.
Thanks
hjm

On Wednesday 22 June 2005 2:09 pm, Brad Dameron wrote:
> And just for the record, look at the Areca SATA RAID cards. They have 16
> and even 24 port cards if you want very large arrays. The cards greatly
> out perform the 3ware cards in READ/WRITE performance. They even offer
> RAID6. Just look at the processor on the ARC-12xx series.
>
> Just my 2 cents. Hope it helps.
>
> Brad Dameron
> SeaTab Software
> www.seatab.com
>
>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

-- 
Cheers, Harry
Harry J Mangalam - 949 856 2847 (vox; email for fax) - hjm@tacgi.com 
            <<plain text preferred>>

Re: More tales of horror from the linux (HW) raid crypt

[Ming Zhang]

[-- Attachment #1: Type: text/plain, Size: 1074 bytes --]

will this 24 port card itself will be a bottleneck?

ming

On Wed, 2005-06-22 at 14:43 -0700, Harry Mangalam wrote:
> Good points and good info; I'll look into the areca products a bit more. Just 
> yesterday I looked at the 16 tray Chenbro/Areca combo:
>  http://www.chenbro.com.tw/Chenbro_Newsletter/Vol21_rm321.htm
> I'll read up on the Areca controllers.
> Thanks
> hjm
> 
> On Wednesday 22 June 2005 2:09 pm, Brad Dameron wrote:
> > And just for the record, look at the Areca SATA RAID cards. They have 16
> > and even 24 port cards if you want very large arrays. The cards greatly
> > out perform the 3ware cards in READ/WRITE performance. They even offer
> > RAID6. Just look at the processor on the ARC-12xx series.
> >
> > Just my 2 cents. Hope it helps.
> >
> > Brad Dameron
> > SeaTab Software
> > www.seatab.com
> >
> >
> > -
> > To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: More tales of horror from the linux (HW) raid crypt

[John Madden]

> will this 24 port card itself will be a bottleneck?

Yes.  The U320 SCSI bus on which it resides will be if nothing else.

John




-- 
John Madden
UNIX Systems Engineer
Ivy Tech Community College of Indiana
jmadden@ivytech.edu

Re: More tales of horror from the linux (HW) raid crypt

[Ming Zhang]

[-- Attachment #1: Type: text/plain, Size: 270 bytes --]

it is a sata card right?

why u320 here?

ming

On Wed, 2005-06-22 at 17:11 -0500, John Madden wrote:
> > will this 24 port card itself will be a bottleneck?
> 
> Yes.  The U320 SCSI bus on which it resides will be if nothing else.
> 
> John
> 
> 
> 
> 

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: More tales of horror from the linux (HW) raid crypt

[Harry Mangalam]

Also a good point - I was just discussing this in another thread.
Areca makes a number of SATA RAID controllers - some are SCSI-SATA, some are 
PCI-X-SATA, some are PCIe-SATA.  

Areca supplied this URL to a relatively long review of recent SATA and SCSI 
controllers - it's worth a read.  And surprisingly, the SATA controllers do 
pretty well against the SCSI controllers.

http://www.tweakers.net/reviews/557/1

hjm

On Wednesday 22 June 2005 3:00 pm, Ming Zhang wrote:
> will this 24 port card itself will be a bottleneck?
>
> ming
>

-- 
Cheers, Harry
Harry J Mangalam - 949 856 2847 (vox; email for fax) - hjm@tacgi.com 
            <<plain text preferred>>

Re: More tales of horror from the linux (HW) raid crypt

[Ming Zhang]

[-- Attachment #1: Type: text/plain, Size: 764 bytes --]

On Wed, 2005-06-22 at 15:45 -0700, Harry Mangalam wrote:
> Also a good point - I was just discussing this in another thread.
> Areca makes a number of SATA RAID controllers - some are SCSI-SATA, some are 
> PCI-X-SATA, some are PCIe-SATA.  

24 port with 40MB/s per disk is 960MB/s, that is a huge number. :P


> 
> Areca supplied this URL to a relatively long review of recent SATA and SCSI 
> controllers - it's worth a read.  And surprisingly, the SATA controllers do 
> pretty well against the SCSI controllers.
> 
> http://www.tweakers.net/reviews/557/1

thanks for this very good link!


> 
> hjm
> 
> On Wednesday 22 June 2005 3:00 pm, Ming Zhang wrote:
> > will this 24 port card itself will be a bottleneck?
> >
> > ming
> >
> 

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

SMART, was Re: More tales of horror from the linux (HW) raid crypt

[Konstantin Olchanski]

On Wed, Jun 22, 2005 at 02:33:00PM -0700, Harry Mangalam wrote:
> The SMART data ... would tell you considerably more info:
> an example from my home system:
> 1062 $ smartctl -a /dev/hda ...

But does anybody know what all these SMART parameters *mean* ?!?

What is this "Raw_Read_Error_Rate"?
What is this "Seek_Error_Rate"?

AFAIK, only a handful of parameters ("temperature", "reallocated sectors",
and "start stop count") seem to measure something. All the other
stuff appears to be complete gibberish.

Then for the SMART "status", it is completely useless- I had SMART
complain about perfectly good disks ("this disk will fail soon!!!"- a few
120GB WDs that sometimes stop seeking, probably through overheating)
and I had completely unusable disks (i.e. develops about 10 unreadable
sectors per day) that SMART was perfectly happy with.

K.O.


> Vendor Specific SMART Attributes with Thresholds:
> ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  
> WHEN_FAILED RAW_VALUE
>   1 Raw_Read_Error_Rate     0x000b   200   200   051    Pre-fail  Always       
> -       0
>   3 Spin_Up_Time            0x0007   099   095   021    Pre-fail  Always       
> -       4141
>   4 Start_Stop_Count        0x0032   100   100   040    Old_age   Always       
> -       122
>   5 Reallocated_Sector_Ct   0x0033   199   199   140    Pre-fail  Always       
> -       2
>   7 Seek_Error_Rate         0x000b   200   200   051    Pre-fail  Always       
> -       0
>   9 Power_On_Hours          0x0032   075   075   000    Old_age   Always       
> -       18940
>  10 Spin_Retry_Count        0x0013   100   100   051    Pre-fail  Always       
> -       0
>  11 Calibration_Retry_Count 0x0013   100   100   051    Pre-fail  Always       
> -       0
>  12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       
> -       120
> 196 Reallocated_Event_Count 0x0032   198   198   000    Old_age   Always       
> -       2
> 197 Current_Pending_Sector  0x0012   200   200   000    Old_age   Always       
> -       0
> 198 Offline_Uncorrectable   0x0012   200   200   000    Old_age   Always       
> -       0
> 199 UDMA_CRC_Error_Count    0x000a   200   253   000    Old_age   Always       
> -       0
> 200 Multi_Zone_Error_Rate   0x0009   200   200   051    Pre-fail  Offline      
> -       0
> 
> SMART Error Log Version: 1
> No Errors Logged
> 
> 
> On Wednesday 22 June 2005 1:38 pm, Jure Pecar wrote:
> > On Wed, 22 Jun 2005 13:16:33 -0700
> >
> > Harry Mangalam <hjm@tacgi.com> wrote:
> > > Perhaps something like the CDDB, where a you can run an applet on a
> > > cronjob  that will upload your disk's SMART data cache to a remote DB on
> > > a regular  basis, so the millions of disks out there can be profiled. 
> > > Hmmm - sounds  like a good undergrad project. Did someone say Summer of
> > > Code? (http://code.google.com/summerofcode.html)
> >
> > Actually if you look at the storagereview.com, they already have a crude
> > form of this online under "reliability survey". You have to login, then you
> > can enter disk model that you have expirience with and what were those
> > expiriences. There's quite some info already available ...
> 
> -- 
> Cheers, Harry
> Harry J Mangalam - 949 856 2847 (vox; email for fax) - hjm@tacgi.com 
>             <<plain text preferred>>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

-- 
Konstantin Olchanski
Data Acquisition Systems: The Bytes Must Flow!
Email: olchansk-at-triumf-dot-ca
Snail mail: 4004 Wesbrook Mall, TRIUMF, Vancouver, B.C., V6T 2A3, Canada

Re: SMART, was Re: More tales of horror from the linux (HW) raid crypt

[Harry Mangalam]

On Wednesday 22 June 2005 4:15 pm, Konstantin Olchanski wrote:

> But does anybody know what all these SMART parameters *mean* ?!?

Yes, apparently, it's all written down:
http://smartmontools.sourceforge.net/#references


> AFAIK, only a handful of parameters ("temperature", "reallocated sectors",
> and "start stop count") seem to measure something. All the other
> stuff appears to be complete gibberish.

Yup, I agree.  But the information that isn't useful doesn't negate that which 
is.  And the gibberish may indeed be useful once it is interpreted.


> Then for the SMART "status", it is completely useless- I had SMART
> complain about perfectly good disks ("this disk will fail soon!!!"- a few
> 120GB WDs that sometimes stop seeking, probably through overheating)
> and I had completely unusable disks (i.e. develops about 10 unreadable
> sectors per day) that SMART was perfectly happy with.

This I can't address - I've not had this experience - others can speak up 
about this.

> K.O.

(and thanks again for your help on my original problem!)
-- 
Cheers, Harry
Harry J Mangalam - 949 856 2847 (vox; email for fax) - hjm@tacgi.com 
            <<plain text preferred>>

Re: SMART, was Re: More tales of horror from the linux (HW) raid crypt

[Mike Hardy]

Konstantin Olchanski wrote:

> But does anybody know what all these SMART parameters *mean* ?!?
> 
> What is this "Raw_Read_Error_Rate"?
> What is this "Seek_Error_Rate"?

Most of them seem to make sense.
I'd guess raw read error rate is the number of read errors corrected by
the firmware in some way (recalibration, ecc, or something).
Seek error rate is probably a calibration thing. The last time I googled
one, I got a pretty good answer.

This was the site I saw:
http://www.ariolic.com/activesmart/smart-attributes/soft-read-error-rate.html

> Then for the SMART "status", it is completely useless- I had SMART
> complain about perfectly good disks ("this disk will fail soon!!!"- a few
> 120GB WDs that sometimes stop seeking, probably through overheating)
> and I had completely unusable disks (i.e. develops about 10 unreadable
> sectors per day) that SMART was perfectly happy with.

Well, I guess anecdotal evidence is just that. At least for the mostly
Maxtor drives I run (which do fail from time to time, alas), the SMART
monitoring has been dead on. Every time. If there's a bad block, SMART
sees it. Right before the drive goes belly-up mysteriously, I'll get
lots of soft read error rate triggers or TA increase count triggers etc.
Its gotten to where if its not a bad block, I should just about hook it
up with an automated dealtime.com query to just buy a drive.

I imagine most folks are somewhere in between. I'm never going to run a
drive without smartd watching it again though, that's for sure.

-Mike

Re: More tales of horror from the linux (HW) raid crypt

[Jon Lewis]

On Wed, 22 Jun 2005, Harry Mangalam wrote:

> I personally buy IBM (now Hitachi I think) and Seagate and have always avoided
> WD as being cheap and cheesy.  5 years ago when I was a sysadmin, I was
> replacing WD at probably 5 times the rate of anything else.  It was probably

Everyone has their own personal HD brand horror stories.  I've seen
multiple IBM DeathStar drives suddenly self destruct.  I've had multiple
defective brand new Maxtor SATA drives and Maxtor send an RMA replacement
(for a PATA) drive that after being stored for a few months turned out to
be dead when I tried to put it in service.

I try to base my drive purchases on warranty length now, on the assumption
that if they don't want to stand behind their product (<3 year warranty),
I don't want to rely on it either.

----------------------------------------------------------------------
 Jon Lewis                   |  I route
 Senior Network Engineer     |  therefore you are
 Atlantic Net                |
_________ http://www.lewis.org/~jlewis/pgp for PGP public key_________

Re: More tales of horror from the linux (HW) raid crypt

[bdameron]

Quoting Ming Zhang <mingz@ele.uri.edu>:

> will this 24 port card itself will be a bottleneck?
> 
> ming

Since the card is PCI-X the only bottleneck on it might be the Processor since 
it is shared with all 24 ports. But I do not know for sure without testing it. 
I personally am going to stick with the new 16 port version. Which is a PCI-
Express card and has twice the CPU power. Since there are so many spindles it 
should be pretty darn fast. And remember that even tho the drives are 150MBps 
they realistically only do about 25-30MBps. 

Brad Dameron
SeaTab Software
www.seatab.com 

Re: More tales of horror from the linux (HW) raid crypt

[bdameron]

There is no U320 SCSI bus. This is a SATA card. Each drive has it's own 
150MBps port. The PCI-X speed of course limits it to 1.06GB per second. I 
doubt the drives will come close to that.

Brad Dameron
SeaTab Software
www.seatab.com


Quoting John Madden <jmadden@ivytech.edu>:

> > will this 24 port card itself will be a bottleneck?
> 
> Yes.  The U320 SCSI bus on which it resides will be if nothing else.
> 
> John
> 

Re: More tales of horror from the linux (HW) raid crypt

[bdameron]

I read this article some time ago. Note that they were testing the older Areca 
card with it's first firmware version. The new PCIe card has twice the CPU 
power and has worked out their initial bugs. The performance is enourmously 
improved. However the price is quit a lot higher than most. 

Brad Dameron
SeaTab Software
www.seatab.com


Quoting Harry Mangalam <hjm@tacgi.com>:

> Also a good point - I was just discussing this in another thread.
> Areca makes a number of SATA RAID controllers - some are SCSI-SATA, some are
> 
> PCI-X-SATA, some are PCIe-SATA.  
> 
> Areca supplied this URL to a relatively long review of recent SATA and SCSI 
> controllers - it's worth a read.  And surprisingly, the SATA controllers do 
> pretty well against the SCSI controllers.
> 
> http://www.tweakers.net/reviews/557/1
> 
> hjm
> 
> On Wednesday 22 June 2005 3:00 pm, Ming Zhang wrote:
> > will this 24 port card itself will be a bottleneck?
> >
> > ming
> >
> 
> -- 
> Cheers, Harry
> Harry J Mangalam - 949 856 2847 (vox; email for fax) - hjm@tacgi.com 
>             <<plain text preferred>>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Re: More tales of horror from the linux (HW) raid crypt

[Ming Zhang]

[-- Attachment #1: Type: text/plain, Size: 1147 bytes --]

On Wed, 2005-06-22 at 17:14 -0700, bdameron@pivotlink.com wrote:
> Quoting Ming Zhang <mingz@ele.uri.edu>:
> 
> > will this 24 port card itself will be a bottleneck?
> > 
> > ming
> 
> Since the card is PCI-X the only bottleneck on it might be the Processor since 
> it is shared with all 24 ports. But I do not know for sure without testing it. 
> I personally am going to stick with the new 16 port version. Which is a PCI-
> Express card and has twice the CPU power. Since there are so many spindles it 
> should be pretty darn fast. And remember that even tho the drives are 150MBps 
> they realistically only do about 25-30MBps. 

the problem here is taht each HD can stably deliver 25-30MBps while the
PCI-x will not arrive that high if have 16 or 24 ports. i do not have a
chance to try out though. those bus at most arrive 70-80% the claimed
peak # :P



> 
> Brad Dameron
> SeaTab Software
> www.seatab.com 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

RE: More tales of horror from the linux (HW) raid crypt

[Guy]

> -----Original Message-----
> From: linux-raid-owner@vger.kernel.org [mailto:linux-raid-
> owner@vger.kernel.org] On Behalf Of Ming Zhang
> Sent: Wednesday, June 22, 2005 8:49 PM
> To: bdameron@pivotlink.com
> Cc: linux-raid@vger.kernel.org
> Subject: Re: More tales of horror from the linux (HW) raid crypt
> 
> On Wed, 2005-06-22 at 17:14 -0700, bdameron@pivotlink.com wrote:
> > Quoting Ming Zhang <mingz@ele.uri.edu>:
> >
> > > will this 24 port card itself will be a bottleneck?
> > >
> > > ming
> >
> > Since the card is PCI-X the only bottleneck on it might be the Processor
> since
> > it is shared with all 24 ports. But I do not know for sure without
> testing it.
> > I personally am going to stick with the new 16 port version. Which is a
> PCI-
> > Express card and has twice the CPU power. Since there are so many
> spindles it
> > should be pretty darn fast. And remember that even tho the drives are
> 150MBps
> > they realistically only do about 25-30MBps.
> 
> the problem here is taht each HD can stably deliver 25-30MBps while the
> PCI-x will not arrive that high if have 16 or 24 ports. i do not have a
> chance to try out though. those bus at most arrive 70-80% the claimed
> peak # :P

Maybe my math is wrong...
But 24 disks at 30 MB/s is 720 MB/s, that is about 68.2% of the PCI-X
bandwidth of 1056 MB/s.

Also, 30 MB/s assumes sequential disk access.  That does not occur in the
real world.  Only during testing.  IMO

Guy

> 
> 
> 
> >
> > Brad Dameron
> > SeaTab Software
> > www.seatab.com
> > -
> > To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: More tales of horror from the linux (HW) raid crypt

[Ming Zhang]

[-- Attachment #1: Type: text/plain, Size: 1457 bytes --]

On Wed, 2005-06-22 at 23:05 -0400, Guy wrote:
> > -----Original Message-----
> > >
> > > > will this 24 port card itself will be a bottleneck?
> > > >
> > > > ming
> > >
> > > Since the card is PCI-X the only bottleneck on it might be the Processor
> > since
> > > it is shared with all 24 ports. But I do not know for sure without
> > testing it.
> > > I personally am going to stick with the new 16 port version. Which is a
> > PCI-
> > > Express card and has twice the CPU power. Since there are so many
> > spindles it
> > > should be pretty darn fast. And remember that even tho the drives are
> > 150MBps
> > > they realistically only do about 25-30MBps.
> > 
> > the problem here is taht each HD can stably deliver 25-30MBps while the
> > PCI-x will not arrive that high if have 16 or 24 ports. i do not have a
> > chance to try out though. those bus at most arrive 70-80% the claimed
> > peak # :P
> 
> Maybe my math is wrong...
> But 24 disks at 30 MB/s is 720 MB/s, that is about 68.2% of the PCI-X
> bandwidth of 1056 MB/s.
yes, u math is better.

> 
> Also, 30 MB/s assumes sequential disk access.  That does not occur in the
> real world.  Only during testing.  IMO
yes, only during test. but what if people build raid5 base on it, this
is probably what people do. and then a disk fail? then a full disk
sequential access becomes normal. and disk fail in 24 disk is not so
uncommon.

> 
> Guy
> 
> > 


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

RE: More tales of horror from the linux (HW) raid crypt

[Guy]

> -----Original Message-----
> From: Ming Zhang [mailto:mingz@ele.uri.edu]
> Sent: Thursday, June 23, 2005 8:32 AM
> To: Guy
> Cc: bdameron@pivotlink.com; linux-raid@vger.kernel.org
> Subject: RE: More tales of horror from the linux (HW) raid crypt
> 
> On Wed, 2005-06-22 at 23:05 -0400, Guy wrote:
> > > -----Original Message-----
> > > >
> > > > > will this 24 port card itself will be a bottleneck?
> > > > >
> > > > > ming
> > > >
> > > > Since the card is PCI-X the only bottleneck on it might be the
> Processor
> > > since
> > > > it is shared with all 24 ports. But I do not know for sure without
> > > testing it.
> > > > I personally am going to stick with the new 16 port version. Which
> is a
> > > PCI-
> > > > Express card and has twice the CPU power. Since there are so many
> > > spindles it
> > > > should be pretty darn fast. And remember that even tho the drives
> are
> > > 150MBps
> > > > they realistically only do about 25-30MBps.
> > >
> > > the problem here is taht each HD can stably deliver 25-30MBps while
> the
> > > PCI-x will not arrive that high if have 16 or 24 ports. i do not have
> a
> > > chance to try out though. those bus at most arrive 70-80% the claimed
> > > peak # :P
> >
> > Maybe my math is wrong...
> > But 24 disks at 30 MB/s is 720 MB/s, that is about 68.2% of the PCI-X
> > bandwidth of 1056 MB/s.
> yes, u math is better.
> 
> >
> > Also, 30 MB/s assumes sequential disk access.  That does not occur in
> the
> > real world.  Only during testing.  IMO
> yes, only during test. but what if people build raid5 base on it, this
> is probably what people do. and then a disk fail? then a full disk
> sequential access becomes normal. and disk fail in 24 disk is not so
> uncommon.
But, this is hardware RAID.  A re-sync would not affect the PCI bus.  All
disk i/o related to re-building the array would be internal to the card.
However, even if it were a software RAID card, the PCI-X would be at 68.2%
load, so it should not be a problem.  If my math is correct!  :)

Also, a single RAID5 array on 24 disks would be high risk of a double
failure.  I think I would build 2 RAID5 arrays of 12 disks each.  Or 2 RAID5
arrays of 11 disks each, with 2 spares.

> 
> >
> > Guy
> >
> > >

Re: More tales of horror from the linux (HW) raid crypt

[Andy Smith]

[-- Attachment #1: Type: text/plain, Size: 352 bytes --]

On Thu, Jun 23, 2005 at 09:03:35AM -0400, Guy wrote:
> Also, a single RAID5 array on 24 disks would be high risk of a double
> failure.  I think I would build 2 RAID5 arrays of 12 disks each.  Or 2 RAID5
> arrays of 11 disks each, with 2 spares.

Just out of interest, why wouldn't you do RAID 10 of 12 pairs?
Because of getting only 50% capacity, or?

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

RE: More tales of horror from the linux (HW) raid crypt

[Ming Zhang]

[-- Attachment #1: Type: text/plain, Size: 2683 bytes --]

On Thu, 2005-06-23 at 09:03 -0400, Guy wrote:
> 
> > -----Original Message-----
> > From: Ming Zhang [mailto:mingz@ele.uri.edu]
> > Sent: Thursday, June 23, 2005 8:32 AM
> > To: Guy
> > Cc: bdameron@pivotlink.com; linux-raid@vger.kernel.org
> > Subject: RE: More tales of horror from the linux (HW) raid crypt
> > 
> > On Wed, 2005-06-22 at 23:05 -0400, Guy wrote:
> > > > -----Original Message-----
> > > > >
> > > > > > will this 24 port card itself will be a bottleneck?
> > > > > >
> > > > > > ming
> > > > >
> > > > > Since the card is PCI-X the only bottleneck on it might be the
> > Processor
> > > > since
> > > > > it is shared with all 24 ports. But I do not know for sure without
> > > > testing it.
> > > > > I personally am going to stick with the new 16 port version. Which
> > is a
> > > > PCI-
> > > > > Express card and has twice the CPU power. Since there are so many
> > > > spindles it
> > > > > should be pretty darn fast. And remember that even tho the drives
> > are
> > > > 150MBps
> > > > > they realistically only do about 25-30MBps.
> > > >
> > > > the problem here is taht each HD can stably deliver 25-30MBps while
> > the
> > > > PCI-x will not arrive that high if have 16 or 24 ports. i do not have
> > a
> > > > chance to try out though. those bus at most arrive 70-80% the claimed
> > > > peak # :P
> > >
> > > Maybe my math is wrong...
> > > But 24 disks at 30 MB/s is 720 MB/s, that is about 68.2% of the PCI-X
> > > bandwidth of 1056 MB/s.
> > yes, u math is better.
> > 
> > >
> > > Also, 30 MB/s assumes sequential disk access.  That does not occur in
> > the
> > > real world.  Only during testing.  IMO
> > yes, only during test. but what if people build raid5 base on it, this
> > is probably what people do. and then a disk fail? then a full disk
> > sequential access becomes normal. and disk fail in 24 disk is not so
> > uncommon.
> But, this is hardware RAID.  A re-sync would not affect the PCI bus.  All
> disk i/o related to re-building the array would be internal to the card.
> However, even if it were a software RAID card, the PCI-X would be at 68.2%
> load, so it should not be a problem.  If my math is correct!  :)
> 
> Also, a single RAID5 array on 24 disks would be high risk of a double
> failure.  I think I would build 2 RAID5 arrays of 12 disks each.  Or 2 RAID5
> arrays of 11 disks each, with 2 spares.
> 
or build with raid6. but anyway, once u a disk, u will need to have a
rsync. with MD, u might try the fr5.

no money, otherwise buy one and try. :P

i think i need a 4U unit, dual amd64...

ming

> > 
> > >
> > > Guy
> > >
> > > >
> 
> 

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]