3TB drives failure rate

3TB drives failure rate

[Rainer Fügenstein]

when trying to upgrade my raid5 with 4 Western digital caviar green
3TB drives [WDC WD30EZRX-00MMMB0] (3 brandnew, 1 about 4months old),
the "old" drive and one of the brand new ones failed with
unrecoverable read errors and about 70 reallocated sectors each. the
failures already occured during the initial resync after creating the
raid.

until now I was very fond of WD caviar green drives, but after this
50% failure rate I'm not very eager to restore data from the backup.

what is your experience with 3TB drives, WD and others?

(low power drives appreciated, performance is not an issue)

tnx.

Re: 3TB drives failure rate

[Mathias Burén]

On 28 October 2012 12:15, Rainer Fügenstein <rfu@oudeis.org> wrote:
>
> when trying to upgrade my raid5 with 4 Western digital caviar green
> 3TB drives [WDC WD30EZRX-00MMMB0] (3 brandnew, 1 about 4months old),
> the "old" drive and one of the brand new ones failed with
> unrecoverable read errors and about 70 reallocated sectors each. the
> failures already occured during the initial resync after creating the
> raid.
>
> until now I was very fond of WD caviar green drives, but after this
> 50% failure rate I'm not very eager to restore data from the backup.
>
> what is your experience with 3TB drives, WD and others?
>
> (low power drives appreciated, performance is not an issue)
>
> tnx.
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Just a few days earlier, I posted on this list, with 2 WD20EARS dives
failing in my RAID6 array. It doesn't matter much if you have them
spinning all the time or not, with age and stress (rebuilds) they
just... die, often.

If you REALLY care about the data, go for enterprise drives, or at
least stay away from green / eco drives.

Mathias
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: 3TB drives failure rate

[John Robinson]

On 28/10/2012 12:19, Mathias Burén wrote:
> On 28 October 2012 12:15, Rainer Fügenstein <rfu@oudeis.org> wrote:
[...]
>> until now I was very fond of WD caviar green drives, but after this
>> 50% failure rate I'm not very eager to restore data from the backup.
>>
>> what is your experience with 3TB drives, WD and others?
[...]
> If you REALLY care about the data, go for enterprise drives, or at
> least stay away from green / eco drives.

I'm about to trial some WD Red 3TB drives, which are theoretically 
quasi-enterprise, slow-spinning, low-power for 24x7 use in NAS RAID 
applications and have usable short-timeout SCT/ERC setting.

I'll let you know how I get on with them.

Cheers,

John.

-- 
John Robinson, yuiop IT services
0131 557 9577 / 07771 784 058
46/12 Broughton Road, Edinburgh EH7 4EE
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: 3TB drives failure rate

[Michael Tokarev]

On 28.10.2012 16:15, Rainer Fügenstein wrote:
> 
> when trying to upgrade my raid5 with 4 Western digital caviar green
> 3TB drives [WDC WD30EZRX-00MMMB0] (3 brandnew, 1 about 4months old),
> the "old" drive and one of the brand new ones failed with
> unrecoverable read errors and about 70 reallocated sectors each. the
> failures already occured during the initial resync after creating the
> raid.
> 
> until now I was very fond of WD caviar green drives, but after this
> 50% failure rate I'm not very eager to restore data from the backup.
> 
> what is your experience with 3TB drives, WD and others?

This is not about 3TB, but about older 2Tb WD EARS.  I've 3 of them
in a raid array which runs 24x7 for about 2 years.  Here's one of
them:

Model Family:     Western Digital Caviar Green (Adv. Format)
Device Model:     WDC WD20EARS-00MVWB0
Firmware Version: 50.0AB50
...
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       83
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  9 Power_On_Hours          0x0032   077   077   000    Old_age   Always       -       16994
194 Temperature_Celsius     0x0022   107   099   000    Old_age   Always       -       43
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   200   200   000    Old_age   Offline      -       0

Other 2 are very similar.

I've several more drives of the same family in other
systems, about 10 in total, some are older than these.
Had no single failure so far.  SMART selftests are run
on a regular basis (about once every 2 months).

I can only guess this depends purely on luck.  I've
seen reports about these drives failing often, yet
others (like me) report these are rock solid.  Go
figure.

/mjt
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: 3TB drives failure rate

[Ed W]

On 28/10/2012 12:15, Rainer Fügenstein wrote:
> when trying to upgrade my raid5 with 4 Western digital caviar green
> 3TB drives [WDC WD30EZRX-00MMMB0] (3 brandnew, 1 about 4months old),
> the "old" drive and one of the brand new ones failed with
> unrecoverable read errors and about 70 reallocated sectors each. the
> failures already occured during the initial resync after creating the
> raid.
>
> until now I was very fond of WD caviar green drives, but after this
> 50% failure rate I'm not very eager to restore data from the backup.
>
> what is your experience with 3TB drives, WD and others?
>
> (low power drives appreciated, performance is not an issue)
>

I think there is clearly serial correlation in drive failures and this 
tends to cause people to have brand love/hate stories.

I bought 9x Samsung 2TB green things about 2 years back  (to go in an 8x 
NAS + 1 spare).  I think I had to return 4 almost immediately due to 
either out of box reallocation warning, or that appeared within 2 
weeks.  Probably if I hadn't been looking I wouldn't have noticed these 
warnings and then been one of those groaning about Samsung when probably 
they all expired within a few weeks of each other.  The RMA'd drives 
have all been fine and the whole array seems ok some years later (tested 
weekly).  Note that I think I got 2x drives from a different supplier 
(hence different batch), so that implies something like 4 out of 7 in a 
given batch were "worrying", but the next 4 from a new batch showed no 
obvious problems

I think this fits with the idea that the spinning disk failure curve has 
a bump in the first few weeks, then flat until some years later when it 
peaks again...

My conclusion:
- RAID6 for data that is highly valuable (and performance is acceptable)
- Thrash the drives initially for some weeks before you accept them into 
production.
- Although highly debated, I believe that failures are likely to be 
correlated in time, when one drive goes there is a high probability of 
loosing others in the next 24 hours. Take precautions as you see fit, eg 
regular backups, hot/warm spares, etc
- Green consumer drives likely are satisfactorarily reliable for most 
uses, caveat that you accept they will fail catastrophically eventually 
(just like your enterprise drive will).  We can debate the relative life 
of each, but it's almost certainly just a linear factor...

Good luck

Ed W

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: 3TB drives failure rate

[Joe Landman]

On 10/28/2012 12:47 PM, Ed W wrote:

> - Green consumer drives likely are satisfactorarily reliable for most
> uses, caveat that you accept they will fail catastrophically eventually
> (just like your enterprise drive will).  We can debate the relative life
> of each, but it's almost certainly just a linear factor...

Our experience is that they aren't acceptable for RAIDs in most cases, 
unless you turn on TLER, and turn off the green functions at minimum. 
In which case, what advantage do they have other than price?

Part of the issue is them falling out of RAIDs.  Part of the the issue 
are the occasional hiccups on coming out of sleep mode, which for most 
desktops isn't a problem, but DEFINITELY an issue for RAIDs.

Another (sometimes significant) issue is that we've been noticing 
partial coverage (e.g. missing functions) in some of the pages available 
to sdparm with the desktop/consumer drives.  This is very annoying, 
especially if you are building a RAID.  Most egregious on SSDs, but 
we've seen one spinning rust device that did the same.

There's not a huge difference in pricing between the two types, and your 
time is valuable.  I'd argue for the lower time cost as part of a longer 
term lower TCO.  We've built systems with both types of drives, but 
based upon the experiences with desktop/consumer drives, we'd have to 
advise avoiding this route if possible.  The upfront money you might 
save will be spent more than likely on your time/efforts to repair 
hiccups in the system later.



-- 
Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics Inc.
email: landman@scalableinformatics.com
web  : http://scalableinformatics.com
        http://scalableinformatics.com/sicluster
phone: +1 734 786 8423 x121
fax  : +1 866 888 3112
cell : +1 734 612 4615

Re: 3TB drives failure rate

[joystick]

On 10/28/12 18:05, Joe Landman wrote:
> On 10/28/2012 12:47 PM, Ed W wrote:
>
>> - Green consumer drives likely are satisfactorarily reliable for most
>> uses, caveat that you accept they will fail catastrophically eventually
>> (just like your enterprise drive will).  We can debate the relative life
>> of each, but it's almost certainly just a linear factor...
>
> Our experience is that they aren't acceptable for RAIDs in most cases, 
> unless you turn on TLER, and turn off the green functions at minimum. 
> In which case, what advantage do they have other than price?

Do you have a magic way to turn on TLER on green drives and would like 
to share it with us?
I can give you money
smartctl scterc did not work there, last time I checked

> Part of the issue is them falling out of RAIDs.  Part of the the issue 
> are the occasional hiccups on coming out of sleep mode, which for most 
> desktops isn't a problem, but DEFINITELY an issue for RAIDs.

This thing of "drives falling out of RAID" I have heard many times, but 
really don't know what people are talking about.
How could a drive fall out of a RAID? A RAID is nothing special, it's 
just read/write commands given to a drive by a process called MD.
If the drive drops out of RAID it means it would have dropped out of a 
normal computer doing normal I/O
Please explain

>
> Another (sometimes significant) issue is that we've been noticing 
> partial coverage (e.g. missing functions) in some of the pages 
> available to sdparm with the desktop/consumer drives.  This is very 
> annoying, especially if you are building a RAID.  Most egregious on 
> SSDs, but we've seen one spinning rust device that did the same.

These sdparm missing feature also would interest me,
but actually less than a response to the other 2 topics above

Re: 3TB drives failure rate

[Miles Fidelman]

joystick wrote:
>
> This thing of "drives falling out of RAID" I have heard many times, 
> but really don't know what people are talking about.
> How could a drive fall out of a RAID? A RAID is nothing special, it's 
> just read/write commands given to a drive by a process called MD.
> If the drive drops out of RAID it means it would have dropped out of a 
> normal computer doing normal I/O
> Please explain

The RAID erroneously thinks that a drive has failed, and drops it from 
the array.


-- 
In theory, there is no difference between theory and practice.
In practice, there is.   .... Yogi Berra

Re: 3TB drives failure rate

[joystick]

On 10/28/12 23:24, Miles Fidelman wrote:
> joystick wrote:
>>
>> This thing of "drives falling out of RAID" I have heard many times, 
>> but really don't know what people are talking about.
>> How could a drive fall out of a RAID? A RAID is nothing special, it's 
>> just read/write commands given to a drive by a process called MD.
>> If the drive drops out of RAID it means it would have dropped out of 
>> a normal computer doing normal I/O
>> Please explain
>
> The RAID erroneously thinks that a drive has failed, and drops it from 
> the array.

Due to missing ERC/TLER and a bad sector, or another reason I am not 
aware of?

And if it is "another reason", would you please explain how is that 
different from what happens in a desktop? How come they are acceptable 
for use in a desktop but they drop out of RAID? Why is RAID so special?

Re: 3TB drives failure rate

[Miles Fidelman]

joystick wrote:
> On 10/28/12 23:24, Miles Fidelman wrote:
>> joystick wrote:
>>>
>>> This thing of "drives falling out of RAID" I have heard many times, 
>>> but really don't know what people are talking about.
>>> How could a drive fall out of a RAID? A RAID is nothing special, 
>>> it's just read/write commands given to a drive by a process called MD.
>>> If the drive drops out of RAID it means it would have dropped out of 
>>> a normal computer doing normal I/O
>>> Please explain
>>
>> The RAID erroneously thinks that a drive has failed, and drops it 
>> from the array.
>
> Due to missing ERC/TLER and a bad sector, or another reason I am not 
> aware of?
>
> And if it is "another reason", would you please explain how is that 
> different from what happens in a desktop? How come they are acceptable 
> for use in a desktop but they drop out of RAID? Why is RAID so special?

Two separate issues.

The comments about "dropping out of raid" had to do with drives that are 
slow to come out of sleep mode - causing hiccups when the RAID 
hardware/software simply doesn't see the drive, and drops it.

The more problematic issue, as far as RAID is concerned is that:

- "desktop" drives are designed to to make every attempt to recover data 
- hence the reported examples of multiple minutes to complete a read 
cycle -- if you only have one copy of your data, that may be an 
acceptable tradeoff, but...

- "enterprise" drives are designed with the assumption that they're part 
of a RAID array - if they can't read the data, they simply give up and 
leave it to the RAID software/hardware to read the data off another drive

If you use the wrong kind of drive in a RAID array, you can find 
yourself waiting minutes, while a failing drive tries to read data, when 
what you want is to drop that drive and read the data off a good drive.




-- 
In theory, there is no difference between theory and practice.
In practice, there is.   .... Yogi Berra

Re: 3TB drives failure rate

[Roman Mamedov]

[-- Attachment #1: Type: text/plain, Size: 1145 bytes --]

On Sun, 28 Oct 2012 20:09:06 -0400
Miles Fidelman <mfidelman@meetinghouse.net> wrote:

> Two separate issues.

> The comments about "dropping out of raid" had to do with drives that are 
> slow to come out of sleep mode - causing hiccups when the RAID 
> hardware/software simply doesn't see the drive, and drops it.

There are no drives in good working order that would come out of sleep mode SO
slowly, that the Linux kernel ATA subsystem would even give up trying and
return an I/O error from it (and it's only after that point, when this begins
to become mdraid's concern).

I have yet to see even any first sign of "SATA frozen" due to drive sleep
mode, let alone to imagine this last through all the port resets and speed
step-downs the SATA driver will attempt.

So the "sleep" issue is not relevant with Linux software RAID, and if you're
still concerned that it might be, you can just reconfigure your drives so they
don't enter that sleep mode. 

-- 
With respect,
Roman

~~~~~~~~~~~~~~~~~~~~~~~~~~~
"Stallman had a printer,
with code he could not see.
So he began to tinker,
and set the software free."

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: 3TB drives failure rate

[David Brown]

On 29/10/2012 05:29, Roman Mamedov wrote:
> On Sun, 28 Oct 2012 20:09:06 -0400
> Miles Fidelman <mfidelman@meetinghouse.net> wrote:
>
>> Two separate issues.
>
>> The comments about "dropping out of raid" had to do with drives that are
>> slow to come out of sleep mode - causing hiccups when the RAID
>> hardware/software simply doesn't see the drive, and drops it.
>
> There are no drives in good working order that would come out of sleep mode SO
> slowly, that the Linux kernel ATA subsystem would even give up trying and
> return an I/O error from it (and it's only after that point, when this begins
> to become mdraid's concern).
>
> I have yet to see even any first sign of "SATA frozen" due to drive sleep
> mode, let alone to imagine this last through all the port resets and speed
> step-downs the SATA driver will attempt.
>
> So the "sleep" issue is not relevant with Linux software RAID, and if you're
> still concerned that it might be, you can just reconfigure your drives so they
> don't enter that sleep mode.
>

The same applies to the long retry times of "desktop" drives - Linux 
software raid has no problem with them.  Some (perhaps "many" or "all" - 
I don't have the experience with hardware raid cards to say) hardware 
raid cards see long read retries as a timeout on the disk, and will drop 
the whole disk from the array.

Linux md raid will wait for the data to come in, and use it if it is 
valid.  If the disk returns an error, the md layer will re-create the 
data from the other disks, then re-write the bad block.  The disk will 
then re-locate the bad block to one of its spare blocks, and everything 
should be fine.  (If the write also fails, the drive gets kicked out.)

So with software raid, there are no problems using desktop drives of any 
sort in your array (assuming, of course, you don't have physical issues 
such as heat generation, vibration, support contracts, etc., that might 
otherwise make you prefer "raid" or "enterprise" drives).

Re: 3TB drives failure rate

[Phil Turmel]

On 10/29/2012 03:54 AM, David Brown wrote:
> On 29/10/2012 05:29, Roman Mamedov wrote:
>> On Sun, 28 Oct 2012 20:09:06 -0400
>> Miles Fidelman <mfidelman@meetinghouse.net> wrote:
>>
>>> Two separate issues.
>>
>>> The comments about "dropping out of raid" had to do with drives that are
>>> slow to come out of sleep mode - causing hiccups when the RAID
>>> hardware/software simply doesn't see the drive, and drops it.
>>
>> There are no drives in good working order that would come out of sleep
>> mode SO
>> slowly, that the Linux kernel ATA subsystem would even give up trying and
>> return an I/O error from it (and it's only after that point, when this
>> begins
>> to become mdraid's concern).
>>
>> I have yet to see even any first sign of "SATA frozen" due to drive sleep
>> mode, let alone to imagine this last through all the port resets and
>> speed
>> step-downs the SATA driver will attempt.
>>
>> So the "sleep" issue is not relevant with Linux software RAID, and if
>> you're
>> still concerned that it might be, you can just reconfigure your drives
>> so they
>> don't enter that sleep mode.
>>
> 
> The same applies to the long retry times of "desktop" drives - Linux
> software raid has no problem with them.  Some (perhaps "many" or "all" -
> I don't have the experience with hardware raid cards to say) hardware
> raid cards see long read retries as a timeout on the disk, and will drop
> the whole disk from the array.

Not true.  The default linux controller timeout is 30 seconds.  Drives
that spend longer than the timeout in recovery will be reset.  If they
don't respond to the reset (because they're busy in recovery) when the
raid tries to write the correct data back to them, they will be kicked
out of the array.

Been there, done that, have the tee shirt.

> Linux md raid will wait for the data to come in, and use it if it is
> valid.  If the disk returns an error, the md layer will re-create the
> data from the other disks, then re-write the bad block.  The disk will
> then re-locate the bad block to one of its spare blocks, and everything
> should be fine.  (If the write also fails, the drive gets kicked out.)

Precisely, except for the wait.  It won't wait that long unless you
change the default driver timeout.  The Seagate drives that did this to
me were kicked of the array because they were still stuck on recovery
when the write was commanded.

The drives were (and still are) just fine.  They had UREs that needed to
be rewritten.  When I later wiped the drives, they remained
relocation-free.  They are now in solo duty as off-site backups.

> So with software raid, there are no problems using desktop drives of any
> sort in your array (assuming, of course, you don't have physical issues
> such as heat generation, vibration, support contracts, etc., that might
> otherwise make you prefer "raid" or "enterprise" drives).

Simply not true, in my experience.  You *must* set ERC shorter than the
timeout, or set the driver timeout longer than the drive's worst-case
recovery time.  The defaults for desktop drives are *not* suitable for
linux software raid.

Phil

3TB drives failure rate (summary)

[Rainer Fügenstein]

tnx for your input; apart from the detailed discussion the topic may
be summarised like this:

- better buy WD red drives next time (will do!)

- other manufacturers drives fail too, not only a WD issue

- drives from one (bad) batch are more likely to fail

current status:

the raid5 assembled of just 3 WD 3TB green drives works fine so far,
even under load. no bad sectors so far (knocking on wood). will extend
it to 4 drives as soon as the replacement drive arrives.

the drives are installed in the same chassis and bays as all the
previous drives, therefore it is unlikely that the failures were
caused by overheating/vibrations etc.

never had any (serious) problems with 750GB and 1.5TB WD green drives
before.

tnx & cu

Re: 3TB drives failure rate (summary)

[Phil Turmel]

On 10/30/2012 07:54 PM, Rainer Fügenstein wrote:
> 
> tnx for your input; apart from the detailed discussion the topic may
> be summarised like this:
> 
> - better buy WD red drives next time (will do!)
> 
> - other manufacturers drives fail too, not only a WD issue
> 
> - drives from one (bad) batch are more likely to fail
> 
> current status:
> 
> the raid5 assembled of just 3 WD 3TB green drives works fine so far,
> even under load. no bad sectors so far (knocking on wood). will extend
> it to 4 drives as soon as the replacement drive arrives.
> 
> the drives are installed in the same chassis and bays as all the
> previous drives, therefore it is unlikely that the failures were
> caused by overheating/vibrations etc.
> 
> never had any (serious) problems with 750GB and 1.5TB WD green drives
> before.

I strongly encourage you to run "smartctl -l scterc /dev/sdX" for each
of your drives.  For any drive that warns that it doesn't support SCT
ERC, set the controller device timeout to 180 like so:

echo 180 >/sys/block/sdX/device/timeout

If the report says read or write ERC is disabled, run "smartctl -l
scterc,70,70 /dev/sdX" to set it to 7.0 seconds.

You then set up a boot-time script to do these adjustments at every restart.

Phil
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: 3TB drives failure rate (summary)

[Miles Fidelman]

Phil Turmel wrote:
> I strongly encourage you to run "smartctl -l scterc /dev/sdX" for each
> of your drives.  For any drive that warns that it doesn't support SCT
> ERC, set the controller device timeout to 180 like so:
>
> echo 180 >/sys/block/sdX/device/timeout
>
> If the report says read or write ERC is disabled, run "smartctl -l
> scterc,70,70 /dev/sdX" to set it to 7.0 seconds.
>
> You then set up a boot-time script to do these adjustments at every restart.
>

Sounds like a very bad idea if your drive is part of a RAID array.



-- 
In theory, there is no difference between theory and practice.
In practice, there is.   .... Yogi Berra

Re: 3TB drives failure rate (summary)

[John Robinson]

On 01/11/2012 15:13, Miles Fidelman wrote:
> Phil Turmel wrote:
>> I strongly encourage you to run "smartctl -l scterc /dev/sdX" for each
>> of your drives.  For any drive that warns that it doesn't support SCT
>> ERC, set the controller device timeout to 180 like so:
>>
>> echo 180 >/sys/block/sdX/device/timeout
>>
>> If the report says read or write ERC is disabled, run "smartctl -l
>> scterc,70,70 /dev/sdX" to set it to 7.0 seconds.
>>
>> You then set up a boot-time script to do these adjustments at every
>> restart.
>
> Sounds like a very bad idea if your drive is part of a RAID array.

Either of these - the scterc if you can, the device timeout if you can't 
- are an excellent idea if you are using a desktop drive as part of a md 
RAID array. Why do you think otherwise?

Cheers,

John.

Re: 3TB drives failure rate (summary)

[Miles Fidelman]

John Robinson wrote:
> On 01/11/2012 15:13, Miles Fidelman wrote:
>> Phil Turmel wrote:
>>> I strongly encourage you to run "smartctl -l scterc /dev/sdX" for each
>>> of your drives.  For any drive that warns that it doesn't support SCT
>>> ERC, set the controller device timeout to 180 like so:
>>>
>>> echo 180 >/sys/block/sdX/device/timeout
>>>
>>> If the report says read or write ERC is disabled, run "smartctl -l
>>> scterc,70,70 /dev/sdX" to set it to 7.0 seconds.
>>>
>>> You then set up a boot-time script to do these adjustments at every
>>> restart.
>>
>> Sounds like a very bad idea if your drive is part of a RAID array.
>
> Either of these - the scterc if you can, the device timeout if you 
> can't - are an excellent idea if you are using a desktop drive as part 
> of a md RAID array. Why do you think otherwise?

If I'm seeing that kind of device timeout, I want it out of my array, 
and I want an alert telling me to replace it.  Otherwise my server's 
performance goes to s*&t.






-- 
In theory, there is no difference between theory and practice.
In practice, there is.   .... Yogi Berra

Re: 3TB drives failure rate (summary)

[John Robinson]

On 01/11/2012 15:39, Miles Fidelman wrote:
> John Robinson wrote:
>> On 01/11/2012 15:13, Miles Fidelman wrote:
>>> Phil Turmel wrote:
>>>> I strongly encourage you to run "smartctl -l scterc /dev/sdX" for each
>>>> of your drives.  For any drive that warns that it doesn't support SCT
>>>> ERC, set the controller device timeout to 180 like so:
>>>>
>>>> echo 180 >/sys/block/sdX/device/timeout
>>>>
>>>> If the report says read or write ERC is disabled, run "smartctl -l
>>>> scterc,70,70 /dev/sdX" to set it to 7.0 seconds.
>>>>
>>>> You then set up a boot-time script to do these adjustments at every
>>>> restart.
>>>
>>> Sounds like a very bad idea if your drive is part of a RAID array.
>>
>> Either of these - the scterc if you can, the device timeout if you
>> can't - are an excellent idea if you are using a desktop drive as part
>> of a md RAID array. Why do you think otherwise?
>
> If I'm seeing that kind of device timeout, I want it out of my array,
> and I want an alert telling me to replace it.  Otherwise my server's
> performance goes to s*&t.

You might not want that kind of device timeout, but as you've made clear 
earlier in this thread, you wouldn't use desktop drives in md RAID, so 
this doesn't really apply to you.

Anyone using desktop drives which don't support SCT ERC in md RAID is 
liable to see long timeouts on the simplest bad sector, and they 
probably prefer to keep the drive in the array and have the sector 
rewritten after reconstruction than have the drive failed out of the array.

Cheers,

John.

Re: 3TB drives failure rate (summary)

[Miles Fidelman]

John Robinson wrote:
> On 01/11/2012 15:39, Miles Fidelman wrote:
>> John Robinson wrote:
>>> On 01/11/2012 15:13, Miles Fidelman wrote:
>>>> Phil Turmel wrote:
>>>>> I strongly encourage you to run "smartctl -l scterc /dev/sdX" for 
>>>>> each
>>>>> of your drives.  For any drive that warns that it doesn't support SCT
>>>>> ERC, set the controller device timeout to 180 like so:
>>>>>
>>>>> echo 180 >/sys/block/sdX/device/timeout
>>>>>
>>>>> If the report says read or write ERC is disabled, run "smartctl -l
>>>>> scterc,70,70 /dev/sdX" to set it to 7.0 seconds.
>>>>>
>>>>> You then set up a boot-time script to do these adjustments at every
>>>>> restart.
>>>>
>>>> Sounds like a very bad idea if your drive is part of a RAID array.
>>>
>>> Either of these - the scterc if you can, the device timeout if you
>>> can't - are an excellent idea if you are using a desktop drive as part
>>> of a md RAID array. Why do you think otherwise?
>>
>> If I'm seeing that kind of device timeout, I want it out of my array,
>> and I want an alert telling me to replace it.  Otherwise my server's
>> performance goes to s*&t.
>
> You might not want that kind of device timeout, but as you've made 
> clear earlier in this thread, you wouldn't use desktop drives in md 
> RAID, so this doesn't really apply to you.

Except for the time I learned this the hard way.



-- 
In theory, there is no difference between theory and practice.
In practice, there is.   .... Yogi Berra

Re: 3TB drives failure rate (summary)

[Adam Goryachev]

On 31/10/12 23:35, Phil Turmel wrote:
> I strongly encourage you to run "smartctl -l scterc /dev/sdX" for each
> of your drives.  For any drive that warns that it doesn't support SCT
> ERC, set the controller device timeout to 180 like so:
>
> echo 180 >/sys/block/sdX/device/timeout
>
> If the report says read or write ERC is disabled, run "smartctl -l
> scterc,70,70 /dev/sdX" to set it to 7.0 seconds.
>
> You then set up a boot-time script to do these adjustments at every restart.

I'm using some SSD's in a RAID, but smartctl -l scterc /dev/sdX says it
doesn't support it.

Do SSD's not support it because it is not relevant (ie, either the data
is there or not, we always know immediately, never any "retry" or whatever?)

Or, should I be concerned about this?

Thanks,
Adam

-- 
Adam Goryachev
Website Managers
www.websitemanagers.com.au

Re: 3TB drives failure rate (summary)

[Roy Sigurd Karlsbakk]

> I'm using some SSD's in a RAID, but smartctl -l scterc /dev/sdX says
> it
> doesn't support it.
> 
> Do SSD's not support it because it is not relevant (ie, either the
> data
> is there or not, we always know immediately, never any "retry" or
> whatever?)
> 
> Or, should I be concerned about this?

My C300s support SCTERC, defaulting to 4 seconds timeout. How the SSD actually handles errors, is probably implementation specific.

Vennlige hilsener / Best regards

roy
--
Roy Sigurd Karlsbakk
(+47) 98013356
roy@karlsbakk.net
http://blogg.karlsbakk.net/
GPG Public key: http://karlsbakk.net/roysigurdkarlsbakk.pubkey.txt
--
I all pedagogikk er det essensielt at pensum presenteres intelligibelt. Det er et elementært imperativ for alle pedagoger å unngå eksessiv anvendelse av idiomer med xenotyp etymologi. I de fleste tilfeller eksisterer adekvate og relevante synonymer på norsk.
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: 3TB drives failure rate (summary)

[Wolfgang Denk]

Dear Adam Goryachev,

In message <511144A7.4050208@websitemanagers.com.au> you wrote:
>
> I'm using some SSD's in a RAID, but smartctl -l scterc /dev/sdX says it
> doesn't support it.
> 
> Do SSD's not support it because it is not relevant (ie, either the data
> is there or not, we always know immediately, never any "retry" or whatever?)

It depends on the type of SSD, I think.

On an Intel 320 Series SSD (SSDSA2CW120G3) I see this:

# smartctl -l scterc /dev/sdg
smartctl 6.0 2012-10-10 r3643 [x86_64-linux-3.6.10-2.fc17.x86_64] (local build)
Copyright (C) 2002-12, Bruce Allen, Christian Franke, www.smartmontools.org

SCT Error Recovery Control:
           Read:    100 (10.0 seconds)
          Write:    100 (10.0 seconds)


Best regards,

Wolfgang Denk

-- 
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd@denx.de
 The software required `Windows 95 or better', so I installed Linux.

Re: 3TB drives failure rate

[Miles Fidelman]

Roman Mamedov wrote:
> On Sun, 28 Oct 2012 20:09:06 -0400
> Miles Fidelman <mfidelman@meetinghouse.net> wrote:
>
>> Two separate issues.
>> The comments about "dropping out of raid" had to do with drives that are
>> slow to come out of sleep mode - causing hiccups when the RAID
>> hardware/software simply doesn't see the drive, and drops it.
> There are no drives in good working order that would come out of sleep mode SO
> slowly, that the Linux kernel ATA subsystem would even give up trying and
> return an I/O error from it (and it's only after that point, when this begins
> to become mdraid's concern).

I was just pointing out that the "dropping out of raid" sub-thread was 
started by a comment about sleep mode causing hiccups, which is a 
separate issue from that of RAID arrays getting f*&ked by long read 
delays from failing consumer drives.

I run servers, I wouldn't be crazy enough to use drives that go to sleep 
(not that they'd have the opportunity).

-- 
In theory, there is no difference between theory and practice.
In practice, there is.   .... Yogi Berra

Re: 3TB drives failure rate

[Chris Murphy]

On Oct 28, 2012, at 6:15 AM, Rainer Fügenstein <rfu@oudeis.org> wrote:

> 
> when trying to upgrade my raid5 with 4 Western digital caviar green
> 3TB drives [WDC WD30EZRX-00MMMB0] (3 brandnew, 1 about 4months old),
> the "old" drive and one of the brand new ones failed with
> unrecoverable read errors and about 70 reallocated sectors each. the
> failures already occured during the initial resync after creating the
> raid.
> 
> until now I was very fond of WD caviar green drives, but after this
> 50% failure rate I'm not very eager to restore data from the backup.

Sounds like they are cooking or vibrating themselves to death?

But also, longer term, you realize Greens are explicitly stated by WDC as not for RAID 5? And RAID 5 implies 24x7 and they're also not a 24x7 drive? The early failure isn't what you asked for, but you've got drives that aren't really applicable for your use case.


> what is your experience with 3TB drives, WD and others?
> 
> (low power drives appreciated, performance is not an issue)

The WDC Reds are what (it seems) the buyers of Greens thought they were buying. Take a look at those, you can use them in RAID5 and 24x7 and they have a 3 year warranty.

And for a non-enterprise disk the Hitachi Deskstar for that same size and warranty range, and I *think* it's a 24x7 drive also but I could be wrong.

Chris Murphy

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: 3TB drives failure rate

[Roman Mamedov]

[-- Attachment #1: Type: text/plain, Size: 710 bytes --]

On Sun, 28 Oct 2012 13:50:41 -0600
Chris Murphy <lists@colorremedies.com> wrote:

> not for RAID 5

> RAID 5 implies 24x7 

> they're also not a 24x7 drive

Wrong on pretty much all points. Or perhaps you are just way too easily
misguided by marketing b/s.

> I *think* it's a 24x7 drive also but I could be wrong.

There is no such thing as a "24x7" drive or not. Given adequate cooling,
powering down and spinning up are an order of magnitude much more stressful
and wearing out operation for a drive than just staying on.

-- 
With respect,
Roman

~~~~~~~~~~~~~~~~~~~~~~~~~~~
"Stallman had a printer,
with code he could not see.
So he began to tinker,
and set the software free."

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: 3TB drives failure rate

[Chris Murphy]

On Oct 28, 2012, at 1:59 PM, Roman Mamedov <rm@romanrm.ru> wrote:

> On Sun, 28 Oct 2012 13:50:41 -0600
> Chris Murphy <lists@colorremedies.com> wrote:
> 
>> not for RAID 5
> 
>> RAID 5 implies 24x7 
> 
>> they're also not a 24x7 drive
> 
> Wrong on pretty much all points. Or perhaps you are just way too easily
> misguided by marketing b/s.

I read their own published specs. Doesn't matter whether that piece is used predominantly by marketing or not, it is a defacto contract that cannot substantially depart from the legalized warranty verbiage.

> 
>> I *think* it's a 24x7 drive also but I could be wrong.
> 
> There is no such thing as a "24x7" drive or not. Given adequate cooling,
> powering down and spinning up are an order of magnitude much more stressful
> and wearing out operation for a drive than just staying on.

Yeah that's more of a marketing term for sure, but in determining the drive warranty, how much it's doing the various things it does is a significant factor. It's not at all the same thing to take a 2.5" laptop drive and run it 24x7 without it ever spinning down. Just like it's not the at all the same thing to take an RE drive and have it spin up and down 15 times a day… like a laptop drive.


Chris Murphy--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: 3TB drives failure rate

[Roman Mamedov]

[-- Attachment #1: Type: text/plain, Size: 1333 bytes --]

On Sun, 28 Oct 2012 14:10:00 -0600
Chris Murphy <lists@colorremedies.com> wrote:

> >> not for RAID 5
> > 
> >> RAID 5 implies 24x7 
> > 
> >> they're also not a 24x7 drive
> > 
> > Wrong on pretty much all points. Or perhaps you are just way too easily
> > misguided by marketing b/s.
> 
> I read their own published specs. Doesn't matter whether that piece is used predominantly by marketing or not, it is a defacto contract that cannot substantially depart from the legalized warranty verbiage.

I could not find any specs from WD where it would say that this particular
drive should be powered on for no more than X hours a day, and not 24x7.
Really, just downloaded the PDF and rechecked.

The closest thing I could find is [1], which after cutting out all the
"we really really want to sell you this enterprise drive for 1.5x as much"
boils down to describing the problem that dumb "hardware RAID" controllers
have with drives without TLER, an issue irrelevant to mdadm users. So,
anything else?

[1]
http://wdc.custhelp.com/app/answers/detail/a_id/1397/~/difference-between-desktop-edition-(wd-blue,-wd-green-and-wd-black)-and-raid

-- 
With respect,
Roman

~~~~~~~~~~~~~~~~~~~~~~~~~~~
"Stallman had a printer,
with code he could not see.
So he began to tinker,
and set the software free."

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: 3TB drives failure rate

[Chris Murphy]

On Oct 28, 2012, at 2:16 PM, Roman Mamedov <rm@romanrm.ru> wrote:

> On Sun, 28 Oct 2012 14:10:00 -0600
> Chris Murphy <lists@colorremedies.com> wrote:
> 
>>>> not for RAID 5
>>> 
>>>> RAID 5 implies 24x7 
>>> 
>>>> they're also not a 24x7 drive
>>> 
>>> Wrong on pretty much all points. Or perhaps you are just way too easily
>>> misguided by marketing b/s.
>> 
>> I read their own published specs. Doesn't matter whether that piece is used predominantly by marketing or not, it is a defacto contract that cannot substantially depart from the legalized warranty verbiage.
> 
> I could not find any specs from WD where it would say that this particular
> drive should be powered on for no more than X hours a day, and not 24x7.

http://www.wdc.com/wdproducts/library/SpecSheet/ENG/2879-771442.pdf

Oh no you're right, it's by inference. Green is a desktop drive. It's neither designed for nor marketed for RAID applications other than consumer RAID (which they consider 1 and 0.) And the Green spec sheet does say that.

http://www.wdc.com/wdproducts/library/SpecSheet/ENG/2879-701229.pdf


> The closest thing I could find is [1], which after cutting out all the
> "we really really want to sell you this enterprise drive for 1.5x as much"

It's typical to do an upsell in all markets. Usually the upsell is a better product too, even if the markup is also a bit higher. It's not like you generally get products that are marked up for no reason at all. Eventually people figure this out, that you're a big fat liar, and then won't buy anything from you, even your cheapest product. (Unless of course you have some weirdly misguided blood loyalty thing going on.)

And on that note, I think WDC has done themselves a disservice by not doing a better job differentiating their products, that they decided to yank configurable SCT ERC in consumer drives, and therein created, very conveniently, space for a new drive, red, that previously didn't exist. Clever and annoying. But they are being rewarded so far. Red sales appear to be through the roof.


> boils down to describing the problem that dumb "hardware RAID" controllers
> have with drives without TLER, an issue irrelevant to mdadm users. So,
> anything else?

That's not true. A drive that can sit there like a bump on the log for 2 minutes before it issues an actual read failure on sector(s) means mdadm is likewise waiting, and everything above it is waiting. That's a long hang. I'd not be surprised to see users go for hard reset with such a system hang after 45 seconds let alone 2 minutes.

Ideally what you'd get is a quick first error recovery with a clean normally operating array. As soon as the first drive fails, the system would set the remaining drives to a slightly longer error recovery time so that you don't get nearly as quick error recovery on the remaining drives - ask them to try a little harder before they error out. If you get another read error, there is no mirror or parity to rebuild from. Best to try a little longer in such a degraded state.

Anyway, the idea mdadm users can't benefit from shorter ERC is untrue. They certainly can. But the open question is why would they be getting such long error recovery times in the first place? 7 seconds is a long time. 2 minutes is a WTF moment. Has the user been doing scrubs and smart tests at all?


Chris Murphy

Re: 3TB drives failure rate

[Roman Mamedov]

[-- Attachment #1: Type: text/plain, Size: 1520 bytes --]

On Sun, 28 Oct 2012 14:34:49 -0600
Chris Murphy <lists@colorremedies.com> wrote:

> A drive that can sit there like a bump on the log for 2 minutes before it issues an actual read failure on sector(s) means mdadm is likewise waiting, and everything above it is waiting. That's a long hang. I'd not be surprised to see users go for hard reset with such a system hang after 45 seconds let alone 2 minutes.

Which is not different from what you would get (the same hang) in a non-RAID
environment on the same drive, so I don't see how this would be specifically a
RAID-related problem.

> Anyway, the idea mdadm users can't benefit from shorter ERC is untrue. They certainly can. But the open question is why would they be getting such long error recovery times in the first place? 7 seconds is a long time.

One thing is "benefit", i.e. just a comfort issue, to prevent pauses when a
drive starts failing (but really... does that happen very often? and during
those times do you really care so that there are no pauses for error recovery,
or maybe you just want to replace the drive and still have your data safe?),
and another thing is a complete RAID failure that you'd get with a
hardware RAID controller simply because something is not within 7 seconds,
which is the whole source of that vendor-supported myth of "non-RAID drives".

-- 
With respect,
Roman

~~~~~~~~~~~~~~~~~~~~~~~~~~~
"Stallman had a printer,
with code he could not see.
So he began to tinker,
and set the software free."

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: 3TB drives failure rate

[Chris Murphy]

On Oct 28, 2012, at 2:49 PM, Roman Mamedov <rm@romanrm.ru> wrote:

> On Sun, 28 Oct 2012 14:34:49 -0600
> Chris Murphy <lists@colorremedies.com> wrote:
> 
>> A drive that can sit there like a bump on the log for 2 minutes before it issues an actual read failure on sector(s) means mdadm is likewise waiting, and everything above it is waiting. That's a long hang. I'd not be surprised to see users go for hard reset with such a system hang after 45 seconds let alone 2 minutes.
> 
> Which is not different from what you would get (the same hang) in a non-RAID
> environment on the same drive, so I don't see how this would be specifically a
> RAID-related problem.

The difference is what you depend on running on that RAID. If it's someone playing movies or games, who cares. If you're a small business running a database or web site off this hardware and everything crawls for two minutes, or implodes because of that delay, now it's just bad design.


> 
>> Anyway, the idea mdadm users can't benefit from shorter ERC is untrue. They certainly can. But the open question is why would they be getting such long error recovery times in the first place? 7 seconds is a long time.
> 
> One thing is "benefit", i.e. just a comfort issue, to prevent pauses when a
> drive starts failing (but really… does that happen very often?

Yes. And obviously with a drive that delays this by 2 minutes it's going to be an increasing problem. Any wonder why "crazy" home users with these drives refer to determinalistic contraptions as "wow the computer is really slow lately" and then some geek blames it on fragmentation. It would not surprise me if these drives are in deep recovery for critical sectors.

If this were untrue, there'd be no need for short recovery times. As it turns out it's a problem for any remotely serious RAID usage.


> and during
> those times do you really care so that there are no pauses for error recovery,
> or maybe you just want to replace the drive and still have your data safe?),
> and another thing is a complete RAID failure that you'd get with a
> hardware RAID controller simply because something is not within 7 seconds,
> which is the whole source of that vendor-supported myth of "non-RAID drives".

Yeah a controller that totally fails a drive for a delayed recovery of one to a few sectors? Bad sysadmin. He picked the wrong drive or he picked the wrong drive timeout in the controller. Either way, it's a mismatch and he's a bad sysadmin if it's important data and not stolen movies off the internet.

Chris Murphy

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: 3TB drives failure rate

[Miles Fidelman]

Chris Murphy wrote:
> On Oct 28, 2012, at 2:49 PM, Roman Mamedov <rm@romanrm.ru> wrote:
>
>> On Sun, 28 Oct 2012 14:34:49 -0600
>> Chris Murphy <lists@colorremedies.com> wrote:
>>
>>> A drive that can sit there like a bump on the log for 2 minutes before it issues an actual read failure on sector(s) means mdadm is likewise waiting, and everything above it is waiting. That's a long hang. I'd not be surprised to see users go for hard reset with such a system hang after 45 seconds let alone 2 minutes.
>> Which is not different from what you would get (the same hang) in a non-RAID
>> environment on the same drive, so I don't see how this would be specifically a
>> RAID-related problem.
> The difference is what you depend on running on that RAID. If it's someone playing movies or games, who cares. If you're a small business running a database or web site off this hardware and everything crawls for two minutes, or implodes because of that delay, now it's just bad design.

Exactly.  The whole point of RAID is to keep things going despite a 
drive failure, kind of problematic if the RAID hardware/software doesn't 
get the message to drop a failing drive from the array.



-- 
In theory, there is no difference between theory and practice.
In practice, there is.   .... Yogi Berra

Re: 3TB drives failure rate

[Chris Murphy]

On Oct 28, 2012, at 2:34 PM, Chris Murphy <lists@colorremedies.com> wrote:

> 
> That's not true. A drive that can sit there like a bump on the log for 2 minutes before it issues an actual read failure on sector(s) means mdadm is likewise waiting, and everything above it is waiting. That's a long hang. I'd not be surprised to see users go for hard reset with such a system hang after 45 seconds let alone 2 minutes.
> 
> Ideally what you'd get is a quick first error recovery with a clean normally operating array. As soon as the first drive fails, the system would set the remaining drives to a slightly longer error recovery time so that you don't get nearly as quick error recovery on the remaining drives - ask them to try a little harder before they error out. If you get another read error, there is no mirror or parity to rebuild from. Best to try a little longer in such a degraded state.

In fact the long error recovery of consumer drives *prevents* automatic correction by md. At least it significantly delays the correction.

That drive, if it can recover a sector in 30 seconds (let alone 2 minutes) instead of failing it, will not be corrected; md won't get an alternate from mirror or from parity, and won't overwrite that obvious flakey sector. So instead you get a propensity for bad sector accumulation, even in the case of regular check scrubs.

For any serious use I just wouldn't use the Greens, without very non-consumer like scrubs, extended smart tests, and cycling out drives so they could be ATA Enhance Secure Erase nuked say once a year or maybe more often. And a rigorous backup. With that kind of expertise and dedication should come a better budget for a better drive.


Chris Murphy

Re: 3TB drives failure rate

[Chris Murphy]

On Oct 28, 2012, at 2:50 PM, Chris Murphy <lists@colorremedies.com> wrote:

> For any serious use I just wouldn't use the Greens, without very non-consumer like scrubs, extended smart tests, and cycling out drives so they could be ATA Enhance Secure Erase nuked say once a year or maybe more often. And a rigorous backup. With that kind of expertise and dedication should come a better budget for a better drive.

On 2nd thought, I'd consider the greens something like a "hostile witness" in legal jargon. Relegate them to only ZFS or btrfs (or ReFS maybe, unclear) for "raid" like pooling or redundancy. The drives themselves are inherently suspect so a suspicious grand inquisitor of a file system seems like an appropriate match, to constantly 2nd guess them.

But still, once a drive is asked to retrieve an LBA, so long as the drive eventually reports it back correctly, the file system won't correct that sector merely for a delay, even if it is up to 2 minutes or whatever it is. So, filesystem choice doesn't really solve the delay problem. You just have to obliterate the disk periodically with zeros or secure erase.


Chris Murphy

Re: 3TB drives failure rate

[Roman Mamedov]

[-- Attachment #1: Type: text/plain, Size: 2207 bytes --]

On Sun, 28 Oct 2012 15:07:53 -0600
Chris Murphy <lists@colorremedies.com> wrote:

> 
> On Oct 28, 2012, at 2:50 PM, Chris Murphy <lists@colorremedies.com> wrote:
> 
> > For any serious use I just wouldn't use the Greens, without very non-consumer like scrubs, extended smart tests, and cycling out drives so they could be ATA Enhance Secure Erase nuked say once a year or maybe more often. And a rigorous backup. With that kind of expertise and dedication should come a better budget for a better drive.
> 
> On 2nd thought, I'd consider the greens something like a "hostile witness" in legal jargon. Relegate them to only ZFS or btrfs (or ReFS maybe, unclear) for "raid" like pooling or redundancy. The drives themselves are inherently suspect so a suspicious grand inquisitor of a file system seems like an appropriate match, to constantly 2nd guess them.

Now you are moving into unfounded superstitions territory.

You seem to imply that a Green drive would return just plain bad data in some
failure condition (else why all the checksumming FSes?), and not an IO Error.
I don't think anything of this sort has been demonstrated so far, and while
this could happen due to bad RAM/chipset/controller/bus/cache/etc, this would
have nothing to do specifically with "Greenness" of a drive, nor any
particular model would be inherently more prone to that.

> But still, once a drive is asked to retrieve an LBA, so long as the drive eventually reports it back correctly, the file system won't correct that sector merely for a delay, even if it is up to 2 minutes or whatever it is. So, filesystem choice doesn't really solve the delay problem. You just have to obliterate the disk periodically with zeros or secure erase.

I do not think there is a state in modern HDDs that there would be a sector
which consistently takes 30-120 seconds to read. Those are either unreadable at
all, or readable after a delay -- and then already remapped by the HDD into the
reserved zone, so the delay is not there the next time.

-- 
With respect,
Roman

~~~~~~~~~~~~~~~~~~~~~~~~~~~
"Stallman had a printer,
with code he could not see.
So he began to tinker,
and set the software free."

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: 3TB drives failure rate

[Mikael Abrahamsson]

On Mon, 29 Oct 2012, Roman Mamedov wrote:

> I do not think there is a state in modern HDDs that there would be a 
> sector which consistently takes 30-120 seconds to read. Those are either 
> unreadable at all, or readable after a delay -- and then already 
> remapped by the HDD into the reserved zone, so the delay is not there 
> the next time.

I have seen drives with abysimal read speed with constant hiccups when 
reading. When reading your text below I believe you're implying that a 
drive will re-write or re-map a sector that took 5 seconds to read. I wish 
this was true, but from what I've seen that is just not happening. If you 
have evidence that contradicts this, do share.

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: 3TB drives failure rate

[Miles Fidelman]

Roman Mamedov wrote:
> I do not think there is a state in modern HDDs that there would be a 
> sector which consistently takes 30-120 seconds to read. Those are 
> either unreadable at all, or readable after a delay -- and then 
> already remapped by the HDD into the reserved zone, so the delay is 
> not there the next time. 

Umm... yes.  This is a common near-failure mode with WD disks, as I 
learned the hard way when I discovered that I had a server that had been 
built with desktop drives rather than enterprise drives.  Took quite 
some time to figure out why my server was slowing WAY down. I still kind 
of wonder why md doesn't consider exceptionally long read times as a 
reason to drop a drive from a RAID array.

Miles Fidelman

-- 
In theory, there is no difference between theory and practice.
In practice, there is.   .... Yogi Berra

Re: 3TB drives failure rate

[Chris Murphy]

On Oct 28, 2012, at 3:45 PM, Miles Fidelman <mfidelman@meetinghouse.net> wrote:

> Roman Mamedov wrote:
>> I do not think there is a state in modern HDDs that there would be a sector which consistently takes 30-120 seconds to read. Those are either unreadable at all, or readable after a delay -- and then already remapped by the HDD into the reserved zone, so the delay is not there the next time. 
> 
> Umm... yes.  This is a common near-failure mode with WD disks, as I learned the hard way when I discovered that I had a server that had been built with desktop drives rather than enterprise drives.  Took quite some time to figure out why my server was slowing WAY down. I still kind of wonder why md doesn't consider exceptionally long read times as a reason to drop a drive from a RAID array.

Drives should be more reliable than this and if they aren't, it's the wrong drive for the task. I'm pretty sure that's the XFS dev position on this too: this sort of data integrity question is drive and application domain, not really kernel or filesystem domain.

And better than dropping a drive from an array, which significantly increases your risk of contact with any subsequent error the system encounters, is to contend with the error. Otherwise you're just exchanging the location of the problem from a hardware controller prematurely dropping multiple drives leading to a collapsed array, to md raid doing the same thing and that's not an improvement worthy of a feature. But it may not be so easy to put md into a 2 minute degraded mode, restoring the array to useful operation, while waiting for that drive to get a grip. Once it has returned, it's anywhere from seconds to 2 minutes behind the array state. So some way to quickly bring it back up to speed would be needed, or you'd have a resync delay on your hands. And that sounds to ignorant me fairl
 y non-trivial.

Honestly I think there are better ways to manage this. Don't use this class of drives for this purpose is one. Another is you take exception measures to make sure they're really working well: I'm not sure if the SATA spec defines transient vs persistent read/write failure unambiguously. Or if each manufacturer's firmware can play loosey goosey with what exactly is a persistent vs transient fail. If it's loose, even an ATA Secure Erase could mean you get "transient" and thus acceptable fail on some sectors than some other firmware (maker, model or even version) would mark as persistent.


On Oct 28, 2012, at 3:59 PM, joystick <joystick@shiftmail.org> wrote:

> Suppose that in one disk you are hitting a bad area where all sectors are unreadable: that's a 256-sectors sequence of 7 seconds waits, that means HALF AN HOUR wait!
> it's total nonsense

That's a lot of bad sectors to have happen all of a sudden. I think I'd be concerned about that, even though the delay is the immediate crisis. 

> I want the SCSI command to be aborted, device RESET, bus RESET or whatever (without the drive dropping out of the controller if possible), then an error to be returned to MD so that it starts the sector rewrite and goes on immediately. Do you think this would be possible or it would puzzle the drive?

My vague recollection of slides comparing SATA and SAS was that once SATA is in an error condition it's non-responsive until it's reconstructed the sector(s) or fails them. If failure, anything in the command queue is flushed, which is not the case for SAS. Maybe this is better dealt elsewhere than directly with the drive, like with AHCI if it could be informed "hey, could you just pretend this drive over here has been hot disconnected? thanks".


Chris Murphy

Re: 3TB drives failure rate

[Chris Murphy]

On Oct 28, 2012, at 3:18 PM, Roman Mamedov <rm@romanrm.ru> wrote:

> Now you are moving into unfounded superstitions territory.

Umm, wholesale trusting data on drives is superstition.

> 
> You seem to imply that a Green drive would return just plain bad data in some
> failure condition (else why all the checksumming FSes?), and not an IO Error.

It certainly has a LOT longer to produce a result other than a read error. And we know ECC doesn't always correct for error correctly. So statistically there is a much larger window for bogus error to be returned that the drive thinks it has legitimately reconstructed. Absolutely.

> I don't think anything of this sort has been demonstrated so far, and while
> this could happen due to bad RAM/chipset/controller/bus/cache/etc, this would
> have nothing to do specifically with "Greenness" of a drive, nor any
> particular model would be inherently more prone to that.

It's a particular consumer drive that happens to not have a disable feature for ERC. The consumer Hitachi deskstars do, last I encountered them. So I don't mean to pick on just the green drives, so you're right that this isn't about the greenness of the drive. It is the particular SCT ERC behavior, and even more to that it's the ECC. Some implementations are better than others.

> 
>> But still, once a drive is asked to retrieve an LBA, so long as the drive eventually reports it back correctly, the file system won't correct that sector merely for a delay, even if it is up to 2 minutes or whatever it is. So, filesystem choice doesn't really solve the delay problem. You just have to obliterate the disk periodically with zeros or secure erase.
> 
> I do not think there is a state in modern HDDs that there would be a sector
> which consistently takes 30-120 seconds to read. Those are either unreadable at
> all, or readable after a delay -- and then already remapped by the HDD into the
> reserved zone, so the delay is not there the next time.

Could be. Seems unclear. And these behaviors change between firmware revisions so there are all kinds of variables changing constantly.


Chris Murphy

Re: 3TB drives failure rate

[joystick]

On 10/28/12 21:34, Chris Murphy wrote:
> Anyway, the idea mdadm users can't benefit from shorter ERC is untrue. 
> They certainly can. But the open question is why would they be getting 
> such long error recovery times in the first place? 7 seconds is a long 
> time. 2 minutes is a WTF moment. 

Totally agreed that 7 seconds is a way way too long time already.
Suppose the user is reading a 5MB file in a 5 disks array, that's 
(approximately) 1MB from each disk.
Suppose that in one disk you are hitting a bad area where all sectors 
are unreadable: that's a 256-sectors sequence of 7 seconds waits, that 
means HALF AN HOUR wait!
it's total nonsense

I have tried to set ERC to lower values than 7 on Hitachi drives, and 
maybe AFAIR also on WD RE, but none of the two allowed values lower 
than, IIRC, 6.0 seconds . Which is strange because the smart command 
wants 2 digits, expressed in deciseconds, so I would expect to be able 
to set the ERC to 0.1 seconds which is definitely not possible.

Any comments would be appreciated.

What Linux needs to address this feature imho is the ability to 
configure the failure actions. If the drive does not respond within 1 
second, I want the SCSI command to be aborted, device RESET, bus RESET 
or whatever (without the drive dropping out of the controller if 
possible), then an error to be returned to MD so that it starts the 
sector rewrite and goes on immediately. Do you think this would be 
possible or it would puzzle the drive?

Re: 3TB drives failure rate

[Phil Turmel]

On 10/28/2012 05:59 PM, joystick wrote:
> I have tried to set ERC to lower values than 7 on Hitachi drives, and
> maybe AFAIR also on WD RE, but none of the two allowed values lower
> than, IIRC, 6.0 seconds . Which is strange because the smart command
> wants 2 digits, expressed in deciseconds, so I would expect to be able
> to set the ERC to 0.1 seconds which is definitely not possible.
> 
> Any comments would be appreciated.
> 
> What Linux needs to address this feature imho is the ability to
> configure the failure actions. If the drive does not respond within 1
> second, I want the SCSI command to be aborted, device RESET, bus RESET
> or whatever (without the drive dropping out of the controller if
> possible), then an error to be returned to MD so that it starts the
> sector rewrite and goes on immediately. Do you think this would be
> possible or it would puzzle the drive?

The timeouts are attributes of the controller queues, and default to 30
seconds.  However, if the timeout is shorter than the drive's worst-case
error recovery time, the drive typically gets kicked out of the array.
The drives I've suffered with on this have ignored the link reset and
were still unresponsive when MD tried to rewrite the sector(s) involved
(from the other drives in the array).

In my experience, that's the real problem with green/consumer drives--if
you choose to use them, you *must* set the queue timeouts to ~180
seconds.  And if your digital video recorder app can't buffer data that
long, you're screwed.

Phil

Re: 3TB drives failure rate

[joystick]

On 10/28/12 23:10, Phil Turmel wrote:
> The drives I've suffered with on this have ignored the link reset and
> were still unresponsive when MD tried to rewrite the sector(s) involved
> (from the other drives in the array).

Very interesting

I would like to see this behaviour to make some experiments.
I would like to simulate an unreadable sector...
I see --make-bad-sector in hdparm, does that simulate an unreadable 
sector faithfully?
(e.g. will it even be recorded in SMART current_pending_sector / 
reallocated_sector_ct ?)

Also, how do I trigger a link reset manually?

Thank you

Re: 3TB drives failure rate

[Phil Turmel]

On 10/28/2012 08:12 PM, joystick wrote:
> On 10/28/12 23:10, Phil Turmel wrote:
>> The drives I've suffered with on this have ignored the link reset and
>> were still unresponsive when MD tried to rewrite the sector(s) involved
>> (from the other drives in the array).
> 
> Very interesting
> 
> I would like to see this behaviour to make some experiments.
> I would like to simulate an unreadable sector...
> I see --make-bad-sector in hdparm, does that simulate an unreadable
> sector faithfully?

I've never tried this, but the description of that option suggests that
it does.

> (e.g. will it even be recorded in SMART current_pending_sector /
> reallocated_sector_ct ?)

Pending, I would presume.  Which should go away when you write back over it.

> Also, how do I trigger a link reset manually?

That I don't know.  But you should get one when you try to read back
that bad sector, if the drive timeout is longer than the driver timeout.

Phil

Re: 3TB drives failure rate

[Chris Murphy]

On Oct 28, 2012, at 6:12 PM, joystick <joystick@shiftmail.org> wrote:

> On 10/28/12 23:10, Phil Turmel wrote:
>> The drives I've suffered with on this have ignored the link reset and
>> were still unresponsive when MD tried to rewrite the sector(s) involved
>> (from the other drives in the array).
> 
> Very interesting
> 
> I would like to see this behaviour to make some experiments.
> I would like to simulate an unreadable sector...
> I see --make-bad-sector in hdparm, does that simulate an unreadable sector faithfully?

In this (simulated) case, it knows the answer: bad sector. In a real scenario it doesn't know. So with hdparm's bad sector you're probably going to get an immediate read failure. And hence no hang.

Chris Murphy

Re: 3TB drives failure rate

[Mikael Abrahamsson]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 972 bytes --]

On Sun, 28 Oct 2012, Rainer Fügenstein wrote:

> what is your experience with 3TB drives, WD and others?

I have some experience with the 2TB Green drives, the ones I bought right 
after launch had a very high failure rate, the ones I got as replacement 
and purchased 6-12 months after launch, have fared a lot better.

I am however replacing them now with WD Red drives (only one so far, had 
my first failure in a year a few months back). The raid resync went very 
smoothly, I had constant 130 megabyte/s write speed to that 2TB Red drive 
during the resync.

As for people still talking RAID5 for these sized drives, I don't 
understand anyone not running RAID6. The rated read error rate on the 
drives are frightening (10^-14), so with single drive failure and doing 
resync, it's not uncommon to get read errors. RAID6 is definitely 
recommended.

<www.zdnet.com/blog/storage/why-raid-5-stops-working-in-2009/162>

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: 3TB drives failure rate

[Peter Kieser]

[-- Attachment #1: Type: text/plain, Size: 1280 bytes --]

I've had interesting cases where WD Green Drives (specifically WD15EARS, 
WD15EADS) don't necessarily "fail" but suddenly have very slow 
random/read write performance which slows down the rest of the array. 
Removing the drive, and replacing it with a (new) drive usually resolves 
the issue. I have since started to avoid "green" drives due to this issue.

-Peter

On 2012-10-28 5:15 AM, Rainer Fügenstein wrote:
> when trying to upgrade my raid5 with 4 Western digital caviar green
> 3TB drives [WDC WD30EZRX-00MMMB0] (3 brandnew, 1 about 4months old),
> the "old" drive and one of the brand new ones failed with
> unrecoverable read errors and about 70 reallocated sectors each. the
> failures already occured during the initial resync after creating the
> raid.
>
> until now I was very fond of WD caviar green drives, but after this
> 50% failure rate I'm not very eager to restore data from the backup.
>
> what is your experience with 3TB drives, WD and others?
>
> (low power drives appreciated, performance is not an issue)
>
> tnx.
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html



[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4504 bytes --]