4-disk raid5 with 2 disks going bad: best way to proceed?

4-disk raid5 with 2 disks going bad: best way to proceed?

[rob pfile]

Hi all,

any collective wisdom on what to do here? i've got a 4-disk raid5, and the most recent checkarray showed several bad blocks caused by uncorrectable read errors on two of the disks in the array. both disks in question show 0 reallocated sectors, but one looks like this:

197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   200   200   000    Old_age   Offline      -       16
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0008   200   200   000    Old_age   Offline      -       16

and the other like this:

197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       14
198 Offline_Uncorrectable   0x0030   200   200   000    Old_age   Offline      -       3
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0008   200   200   000    Old_age   Offline      -       11

i'm a bit worried about failing one of these disks for fear that the other might give uncorrectable read errors during the rebuild. if i *had* to choose one, should i choose the one with all the pending sectors, or the one with all the uncorrectable sectors?

does it makes sense to do a smartctl -t offline scan on one or both of these disks first?

i guess i could take the array offline, clone one of the disks with dd, and then swap the clone in. but... is there a way to clone one disk in the array using mdadm? in other words, is there a way to construct a clean copy of one of the disks even if there are raid-correctable read errors?

i do have backups, so perhaps it will not kill me if the array dies, but i'd like to tread carefully and try and get out of this mess without nuking everything.

thanks for any advice,

rob

Re: 4-disk raid5 with 2 disks going bad: best way to proceed?

[Roberto Spadim]

i'm not a master expert with linux raid, but... stop array, add 2 new
disks and make a backup (dd) of two bad disks, start array with the
new disks (must be same size)
(i don't know if raid5 have spare disks and how they work, maybe
there's a 'online' solution without stopping array)
maybe others guys here could help you better, but this one works :)

2011/4/6 rob pfile <rpfile@gmail.com>:
> Hi all,
>
> any collective wisdom on what to do here? i've got a 4-disk raid5, and the most recent checkarray showed several bad blocks caused by uncorrectable read errors on two of the disks in the array. both disks in question show 0 reallocated sectors, but one looks like this:
>
> 197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
> 198 Offline_Uncorrectable   0x0030   200   200   000    Old_age   Offline      -       16
> 199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
> 200 Multi_Zone_Error_Rate   0x0008   200   200   000    Old_age   Offline      -       16
>
> and the other like this:
>
> 197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       14
> 198 Offline_Uncorrectable   0x0030   200   200   000    Old_age   Offline      -       3
> 199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
> 200 Multi_Zone_Error_Rate   0x0008   200   200   000    Old_age   Offline      -       11
>
> i'm a bit worried about failing one of these disks for fear that the other might give uncorrectable read errors during the rebuild. if i *had* to choose one, should i choose the one with all the pending sectors, or the one with all the uncorrectable sectors?
>
> does it makes sense to do a smartctl -t offline scan on one or both of these disks first?
>
> i guess i could take the array offline, clone one of the disks with dd, and then swap the clone in. but... is there a way to clone one disk in the array using mdadm? in other words, is there a way to construct a clean copy of one of the disks even if there are raid-correctable read errors?
>
> i do have backups, so perhaps it will not kill me if the array dies, but i'd like to tread carefully and try and get out of this mess without nuking everything.
>
> thanks for any advice,
>
> rob
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>



-- 
Roberto Spadim
Spadim Technology / SPAEmpresarial
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: 4-disk raid5 with 2 disks going bad: best way to proceed?

[Mikael Abrahamsson]

On Wed, 6 Apr 2011, rob pfile wrote:

> Hi all,
>
> any collective wisdom on what to do here? i've got a 4-disk raid5, and 
> the most recent checkarray showed several bad blocks caused by 
> uncorrectable read errors on two of the disks in the array. both disks 
> in question show 0 reallocated sectors, but one looks like this:

Generally I'd recommend a "repair" as it would try to read all, if it 
can't read it properly, it'd recalculate from parity and as long as that 
write succeeded, you'd be golden.

To be safe, stop the array, dd_rescue the two bad drives, start the array 
again with the originals in the array, don't mount the filesystem, 
issue repair and see what happens.

This is one reason why I nowadays always run RAID6, then you can fail a 
drive and still have parity for read errors...

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: 4-disk raid5 with 2 disks going bad: best way to proceed?

[Nagilum]

Hmm, I think you could:
  - add a disk,
  - reshape your raid5 to raid6 while using the layout where the raid5  
disks aren't really changed
  - let it finish syncing
  - reshape the raid6 to raid5 taking one of the bad disks out

I'm sure Neil will let us know if this is nonsense. ;)
Alex.

----- Message from rpfile@gmail.com ---------
     Date: Wed, 6 Apr 2011 18:45:15 -0700
     From: rob pfile <rpfile@gmail.com>
  Subject: 4-disk raid5 with 2 disks going bad: best way to proceed?
       To: linux-raid@vger.kernel.org

> is there a way to construct a clean copy of one of the disks even if  
> there are raid-correctable read errors?

----- End message from rpfile@gmail.com -----



========================================================================
#    _  __          _ __     http://www.nagilum.org/ \n icq://69646724 #
#   / |/ /__ ____ _(_) /_ ____ _  nagilum@nagilum.org \n +491776461165 #
#  /    / _ `/ _ `/ / / // /  ' \  Amiga (68k/PPC): AOS/NetBSD/Linux   #
# /_/|_/\_,_/\_, /_/_/\_,_/_/_/_/   Mac (PPC): MacOS-X / NetBSD /Linux #
#           /___/     x86: FreeBSD/Linux/Solaris/Win2k  ARM9: EPOC EV6 #
========================================================================


----------------------------------------------------------------
cakebox.homeunix.net - all the machine one needs..

Re: 4-disk raid5 with 2 disks going bad: best way to proceed?

[rob pfile]

On Apr 7, 2011, at 12:21 AM, Mikael Abrahamsson wrote:

> On Wed, 6 Apr 2011, rob pfile wrote:
> 
>> Hi all,
>> 
>> any collective wisdom on what to do here? i've got a 4-disk raid5, and the most recent checkarray showed several bad blocks caused by uncorrectable read errors on two of the disks in the array. both disks in question show 0 reallocated sectors, but one looks like this:
> 
> Generally I'd recommend a "repair" as it would try to read all, if it can't read it properly, it'd recalculate from parity and as long as that write succeeded, you'd be golden.
> 
> To be safe, stop the array, dd_rescue the two bad drives, start the array again with the originals in the array, don't mount the filesystem, issue repair and see what happens.
> 
> This is one reason why I nowadays always run RAID6, then you can fail a drive and still have parity for read errors...
> 
> 

thanks for your reply.

from reading a similar thread, (http://www.spinics.net/lists/raid/msg31779.html) it's stated that the "repair" command will rebuild the parity if it is thought to be wrong.  i don't think i want to risk writing parity blocks that are probably now correct and could become corrupted because of a bad data read messing up the parity... it's almost like i want something inbetween "check" and "repair" where when the disk gives a hard error on a sector, the data block containing that sector is reconstructed from the parity and then immediately written back to the disk. or does "check" already do that? i was guessing that it did not, or else the drives would probably show a few reallocated sectors.

by "as long as that write succeeded you'd be golden", do you mean that re-writing the block would either reallocate the bad sector, or that perhaps the write would just succeed on the same physical sector, thus cleaning it up somehow? i think reallocating the sector would be preferable but i guess we don't have too much control over what the disk does.

i guess i should clone these disks as you suggest. if dd_rescue runs without error then i suppose i can just put in the replacement disks and forget about it. if not, i could try the repair. i assume that if the repair goes horribly wrong, i could just put the clones into the array. but as above i'd worry that if corrupt parity got rewritten during the repair of the original disks that perhaps the clones would no longer match the corrupt parity. in that case i'd have to run "repair" again after putting the clones in, i assume.


yeah, i should probably look into raid6.

thanks,

rob

Re: 4-disk raid5 with 2 disks going bad: best way to proceed?

[NeilBrown]

On Thu, 07 Apr 2011 22:13:34 +0200 Nagilum <nagilum@nagilum.org> wrote:

> Hmm, I think you could:
>   - add a disk,
>   - reshape your raid5 to raid6 while using the layout where the raid5  
> disks aren't really changed

That would be
  mdadm --grow /dev/md0 --level=6 --parity=preserve
(I think).

>   - let it finish syncing
>   - reshape the raid6 to raid5 taking one of the bad disks out

I don't think that makes sense. (what command would you use exactly?)
It would be better to remove the bad device and replace it with another good
device (which means more dollars of course...)

> 
> I'm sure Neil will let us know if this is nonsense. ;)

Only half ;-)

NeilBrown


> Alex.
> 
> ----- Message from rpfile@gmail.com ---------
>      Date: Wed, 6 Apr 2011 18:45:15 -0700
>      From: rob pfile <rpfile@gmail.com>
>   Subject: 4-disk raid5 with 2 disks going bad: best way to proceed?
>        To: linux-raid@vger.kernel.org
> 
> > is there a way to construct a clean copy of one of the disks even if  
> > there are raid-correctable read errors?
> 
> ----- End message from rpfile@gmail.com -----
> 
> 
> 
> ========================================================================
> #    _  __          _ __     http://www.nagilum.org/ \n icq://69646724 #
> #   / |/ /__ ____ _(_) /_ ____ _  nagilum@nagilum.org \n +491776461165 #
> #  /    / _ `/ _ `/ / / // /  ' \  Amiga (68k/PPC): AOS/NetBSD/Linux   #
> # /_/|_/\_,_/\_, /_/_/\_,_/_/_/_/   Mac (PPC): MacOS-X / NetBSD /Linux #
> #           /___/     x86: FreeBSD/Linux/Solaris/Win2k  ARM9: EPOC EV6 #
> ========================================================================
> 
> 
> ----------------------------------------------------------------
> cakebox.homeunix.net - all the machine one needs..
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: 4-disk raid5 with 2 disks going bad: best way to proceed?

[NeilBrown]

On Thu, 7 Apr 2011 15:15:09 -0700 rob pfile <rpfile@gmail.com> wrote:

> 
> On Apr 7, 2011, at 12:21 AM, Mikael Abrahamsson wrote:
> 
> > On Wed, 6 Apr 2011, rob pfile wrote:
> > 
> >> Hi all,
> >> 
> >> any collective wisdom on what to do here? i've got a 4-disk raid5, and the most recent checkarray showed several bad blocks caused by uncorrectable read errors on two of the disks in the array. both disks in question show 0 reallocated sectors, but one looks like this:
> > 
> > Generally I'd recommend a "repair" as it would try to read all, if it can't read it properly, it'd recalculate from parity and as long as that write succeeded, you'd be golden.
> > 
> > To be safe, stop the array, dd_rescue the two bad drives, start the array again with the originals in the array, don't mount the filesystem, issue repair and see what happens.
> > 
> > This is one reason why I nowadays always run RAID6, then you can fail a drive and still have parity for read errors...
> > 
> > 
> 
> thanks for your reply.
> 
> from reading a similar thread, (http://www.spinics.net/lists/raid/msg31779.html) it's stated that the "repair" command will rebuild the parity if it is thought to be wrong.  i don't think i want to risk writing parity blocks that are probably now correct and could become corrupted because of a bad data read messing up the parity... it's almost like i want something inbetween "check" and "repair" where when the disk gives a hard error on a sector, the data block containing that sector is reconstructed from the parity and then immediately written back to the disk. or does "check" already do that? i was guessing that it did not, or else the drives would probably show a few reallocated sectors.

When a device gives a hard read error md/raid always calculates the correct
data from other devices (Assuming that parity is correct) and writes it out.
It does this for check and for repair and for normal IO.

I am no expert on SMART however if there are no reallocated sectors then
maybe what happened is that whenever md wrote to a bad sector, the drive
determined that the media there was still usable and wrote the data there.
But that is just a guess.

> 
> by "as long as that write succeeded you'd be golden", do you mean that re-writing the block would either reallocate the bad sector, or that perhaps the write would just succeed on the same physical sector, thus cleaning it up somehow? i think reallocating the sector would be preferable but i guess we don't have too much control over what the disk does.
> 
> i guess i should clone these disks as you suggest. if dd_rescue runs without error then i suppose i can just put in the replacement disks and forget about it. if not, i could try the repair. i assume that if the repair goes horribly wrong, i could just put the clones into the array. but as above i'd worry that if corrupt parity got rewritten during the repair of the original disks that perhaps the clones would no longer match the corrupt parity. in that case i'd have to run "repair" again after putting the clones in, i assume.

I would probably be using 'check' rather than 'repair'.  The key is to read
all the drive so as to find any bad block and correct them.  Check does this.
If check reports errors in mismatch_cnt, then it might be appropriate to run
'repair'.  If any data has been corrupt it is already to late to do anything
about it.

NeilBrown


> 
> 
> yeah, i should probably look into raid6.
> 
> thanks,
> 
> rob
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: 4-disk raid5 with 2 disks going bad: best way to proceed?

[Nagilum]

----- Message from neilb@suse.de ---------
     From: NeilBrown <neilb@suse.de>
  Subject: Re: 4-disk raid5 with 2 disks going bad: best way to proceed?


> On Thu, 07 Apr 2011 22:13:34 +0200 Nagilum <nagilum@nagilum.org> wrote:
>
>> Hmm, I think you could:
>>   - add a disk,
>>   - reshape your raid5 to raid6 while using the layout where the raid5
>> disks aren't really changed
>
> That would be
>   mdadm --grow /dev/md0 --level=6 --parity=preserve
> (I think).
>
>>   - let it finish syncing
>>   - reshape the raid6 to raid5 taking one of the bad disks out
>
> I don't think that makes sense. (what command would you use exactly?)

I was thinking one fails one of the bad disks and then does something like:
mdadm --grow /dev/md0 --level=5 --raid-devices=4  
--backup-file=/root/backup-md4

But obviously I've never tried that and I would play around with  
losetup and some files for practice and to see if that works before  
starting.


----- End message from neilb@suse.de -----



========================================================================
#    _  __          _ __     http://www.nagilum.org/ \n icq://69646724 #
#   / |/ /__ ____ _(_) /_ ____ _  nagilum@nagilum.org \n +491776461165 #
#  /    / _ `/ _ `/ / / // /  ' \  Amiga (68k/PPC): AOS/NetBSD/Linux   #
# /_/|_/\_,_/\_, /_/_/\_,_/_/_/_/   Mac (PPC): MacOS-X / NetBSD /Linux #
#           /___/     x86: FreeBSD/Linux/Solaris/Win2k  ARM9: EPOC EV6 #
========================================================================


----------------------------------------------------------------
cakebox.homeunix.net - all the machine one needs..

Re: 4-disk raid5 with 2 disks going bad: best way to proceed?

[rob pfile]

On Apr 8, 2011, at 5:10 AM, NeilBrown wrote:

> When a device gives a hard read error md/raid always calculates the correct
> data from other devices (Assuming that parity is correct) and writes it out.
> It does this for check and for repair and for normal IO.
> 
> I am no expert on SMART however if there are no reallocated sectors then
> maybe what happened is that whenever md wrote to a bad sector, the drive
> determined that the media there was still usable and wrote the data there.
> But that is just a guess.
> 

excellent, thanks. this probably explains why the errors seem to be migrating around the disk - perhaps the ones that are getting corrected/rewritten are effectively being 'refreshed' and sectors that are sitting around unread for a while are rotting. 

i ran another check, and this time saw several read errors on one of the bad disks, but not on the other. strangely, i did not see raid correcting these sectors. i wonder if this means they eventually returned good data. does md/raid try more than once in the face of a hard error before correcting from parity?

given that i have backups, i decided to just fail out the disk that was still giving errors. the raid is rebuilding now, and in 20h or so i'll know if i'm out of the woods. i think i'll probably also replace the other flaky disk, even though it's SMART status now looks pretty clean.

===

update: bad news... a *different* drive in the array threw a read error while rebuilding. i wonder if my controller or enclosure might be bad, since 3 out of 4 disks have exhibited random problems (and no reallocated sectors).

anyway, now the raid is up, but in a degraded state. what i want to do is bring the array back to a clean state with 3 drives and restart the rebuild onto the new disk and see what happens. how do i get the "failed" disk back into "active sync" state? mdadm --manage /dev/md1 --add /dev/sdi1 returns "device busy", and --assemble won't take --assume-clean as an argument. do i have to stop and re-start the array? a little scared to do that.

thanks,

rob