Fwd: Maximizing failed disk replacement on a RAID5 array

Fwd: Maximizing failed disk replacement on a RAID5 array

[Durval Menezes]

Hello folks,

A few days ago, the smartd daemon running on my Lucid system at home
(kernel 2.6.32-32-generic, mdadm 2.6.7.1) has started warning me about
a few (less than 50 so far) offline uncorrectable and other errors on
one of my 1.5TB HDs three-disk RAID5 array. This failing HD is still
online (ie, hasn't been kicked off the array), at least for now.

I have another disk ready for replacement, and I'm trying to determine
the safer (not necessarily the simpler) way of proceeding.

I understand that, if I do it the "standard" way (ie, power down the
system, remove the failing disk, add the replacement disk, then boot
up and use "mdadm --add" to add the new disk to the array) I run the
risk of running into unreadable sectors on one of the other two disks,
and then my RAID5 is kaput.

What I would really like to do is to be able to add the new HD to the
array WITHOUT removing the failing HD, somehow sync it with the rest,
and THEN remove the failing HD: that way, an eventual failed read from
one of the two other HDs could possibly be satisfied from the failing
HD (unless EXACTLY that same sector is also unreadable on it, which I
find unlikely), and so avoid losing the whole array in the above case.

So far, the only way I've been able to figure to do that would be to
convert the  array from RAID5 to RAID6, add the new disk, wait for the
array to sync, remove the failing disk, and then convert the array
back from RAID6 to RAID5 (and I'm not really sure that this is a good
idea, or even doable).

So, folks, what do you say? Is there a better way? Any gotchas in the
RAID5->RAID6->RAID6 approach?

Thanks,
--
   Durval Menezes.
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Maximizing failed disk replacement on a RAID5 array

[Drew]

> I understand that, if I do it the "standard" way (ie, power down the
> system, remove the failing disk, add the replacement disk, then boot
> up and use "mdadm --add" to add the new disk to the array) I run the
> risk of running into unreadable sectors on one of the other two disks,
> and then my RAID5 is kaput.
>
> What I would really like to do is to be able to add the new HD to the
> array WITHOUT removing the failing HD, somehow sync it with the rest,
> and THEN remove the failing HD: that way, an eventual failed read from
> one of the two other HDs could possibly be satisfied from the failing
> HD (unless EXACTLY that same sector is also unreadable on it, which I
> find unlikely), and so avoid losing the whole array in the above case.

A reshape from RAID5 -> RAID6 -> RAID5 will hammer your disks so if
either of the other two are ready to die, this will most likely tip
them over the edge.

A far simpler way would be to take the array offline, dd (or
dd_rescue) the old drive's contents onto the new disk, pull the old
disk, and restart the array with the new drive in it's place. With
luck you won't need a resync *and* you're not hammering the other two
drives in the process.


-- 
Drew

"Nothing in life is to be feared. It is only to be understood."
--Marie Curie

"This started out as a hobby and spun horribly out of control."
-Unknown

Re: Maximizing failed disk replacement on a RAID5 array

[Brad Campbell]

On 06/06/11 23:02, Drew wrote:
>> I understand that, if I do it the "standard" way (ie, power down the
>> system, remove the failing disk, add the replacement disk, then boot
>> up and use "mdadm --add" to add the new disk to the array) I run the
>> risk of running into unreadable sectors on one of the other two disks,
>> and then my RAID5 is kaput.
>>
>> What I would really like to do is to be able to add the new HD to the
>> array WITHOUT removing the failing HD, somehow sync it with the rest,
>> and THEN remove the failing HD: that way, an eventual failed read from
>> one of the two other HDs could possibly be satisfied from the failing
>> HD (unless EXACTLY that same sector is also unreadable on it, which I
>> find unlikely), and so avoid losing the whole array in the above case.
> A reshape from RAID5 ->  RAID6 ->  RAID5 will hammer your disks so if
> either of the other two are ready to die, this will most likely tip
> them over the edge.
>
> A far simpler way would be to take the array offline, dd (or
> dd_rescue) the old drive's contents onto the new disk, pull the old
> disk, and restart the array with the new drive in it's place. With
> luck you won't need a resync *and* you're not hammering the other two
> drives in the process.
<afterthought>
Bear with me, I've had a few scotches and this might not be as coherent as it might be, but I think 
I spot a very, very fatal flaw in your plan.
</afterthought>

I thought this initially also, except it blows up in the scenario where the dud sectors are data and 
not parity.

If you do it the way you suggest and choose dd_rescue in place of dd, dodgy data from the dud 
sectors will be replicated as kosher sectors on the replacement disk (or zero, or random or whatever)

If you execute a "repair" first, it will strike the dud sectors, see they are toast, re-calculate 
them from parity and write them back forcing a reallocation.

You can then replicate the failing disk using "dd", *not* dd_rescue. If dd fails due to a read error 
then you know that part of your data is likely to be toast on the replaced disk, and you can go 
about making provisions for a backup/restore operation using the original disk (which will likely 
succeed as the data read from the array will be re-built from parity where required).

dd_rescue is a blessing and a curse. It's _very_ good at getting you access to data that you have no 
backup of, and you have no other way of getting back. On the other hand, it will happily go and 
replicate whatever trash it happens to get back from the source disk, or skip those sectors and 
leave you with an incomplete copy that will leave no trace of it being incomplete until you find 
chunks missing (like superblocks or your formula for a zero cost petroleum replacement).

If your array works, but has a badly failing drive you are far better to buy some cheap 2TB disks 
and back it up, then restore it onto a re-created array than chance losing chunks of data by using a 
dd_rescue'd clone disk.

Now, if I'm off the wall and missing something blindingly obvious feel free to thump me with a clue 
bat (it would not be the first time).

I've lost 2 arrays recently. 8TB to a dodgy controller (thanks SIL), and 2TB to complete idiocy on 
my part, so I know the sting of lost or corrupted data.

Brad

Re: Maximizing failed disk replacement on a RAID5 array

[Drew]

> Now, if I'm off the wall and missing something blindingly obvious feel free
> to thump me with a clue bat (it would not be the first time).
>
> I've lost 2 arrays recently. 8TB to a dodgy controller (thanks SIL), and 2TB
> to complete idiocy on my part, so I know the sting of lost or corrupted
> data.

I think you've covered the process in more detail, including pitfalls,
then I have. :-) Only catch is where would you find a cheap 2-3TB
drive right now?

I also know the sting of mixing stupidity and dd. ;-) A friend was
helping me do some complex rework with dd on one of my disks. Being
the n00b I followed his instructions exactly, and him being the expert
(and assuming I wasn't the n00b I was back then) didn't double check
my work. Net result was I backed the MBR/Partition Table up using dd,
but did so to a partition on the drive we were working on. There may
have been some alcohol involved (I was in University), the revised
data we inserted failed, and next thing you know I'm running Partition
Magic (the gnu tools circa 2005 failed to detect anything) to try and
recover the partition table. No backups obviously. ;-)


-- 
Drew

"Nothing in life is to be feared. It is only to be understood."
--Marie Curie

Re: Maximizing failed disk replacement on a RAID5 array

[Brad Campbell]

On 06/06/11 23:37, Drew wrote:
>> Now, if I'm off the wall and missing something blindingly obvious feel free
>> to thump me with a clue bat (it would not be the first time).
>>
>> I've lost 2 arrays recently. 8TB to a dodgy controller (thanks SIL), and 2TB
>> to complete idiocy on my part, so I know the sting of lost or corrupted
>> data.
> I think you've covered the process in more detail, including pitfalls,
> then I have. :-) Only catch is where would you find a cheap 2-3TB
> drive right now?

I bought 10 recently for about $90 each. It's all relative, but I consider ~$45 / TB cheap.

> I also know the sting of mixing stupidity and dd. ;-) A friend was
> helping me do some complex rework with dd on one of my disks. Being
> the n00b I followed his instructions exactly, and him being the expert
> (and assuming I wasn't the n00b I was back then) didn't double check
> my work. Net result was I backed the MBR/Partition Table up using dd,
> but did so to a partition on the drive we were working on. There may
> have been some alcohol involved (I was in University), the revised
> data we inserted failed, and next thing you know I'm running Partition
> Magic (the gnu tools circa 2005 failed to detect anything) to try and
> recover the partition table. No backups obviously. ;-)

Similar to my

dd if=/dev/zero of=/dev/sdb bs=1M count=100

except instead of the target disk, it was to a raid array member that was currently active. To its 
credit, ext3 and fsck managed to give me most of my data back, even if I had to spend months 
intermittently sorting/renaming inode numbers from lost+found into files and directories.

I'd like to claim Alcohol as a mitigating factor (hell, it gets people off charges in our court 
system all the time) but unfortunately I was just stupid.

Re: Maximizing failed disk replacement on a RAID5 array

[Durval Menezes]

Hello Brad, Drew,

Thanks for reminding me of the hammering a RAID level conversion would  cause.
This is certainly a major  reason to avoid the RAID5->RAID6->RAID5 route.

The "repair" has been running here for a few days already, with the
server online, and ought to finish in 24 more hours. So far (thanks to
the automatic rewrite relocation) the number of  uncorrectable sectors
being reported by SMART has dropped from 40 to 20 , so it seems the
repair is  doing its job. Lets just hope the disk has enough  spare
sectors  to remap all the bad sectors; if it does, a simple "dd "from
the bad disk to  its replacement ought to  do the job  (as you have
indicated).

On the other hand, as this "dd" has to be done with the array offline,
it will entail in some downtime (although not as much as having to
restore the whole array from backups).... not ideal, but not too bad
either.

In case worst comes to worst, I have an up-to-date offline backup of
the contents of the whole array, so if something really bad happens, I
have something to restore from.

It would be great to have a
"duplicate-this-bad-old-disk-into-this-shiny-new-disk"  functionality,
as it would enable  an almost-no-downtime disk replacement with
minimum  risk, but it seems we can't have everything... :-0 Maybe it's
something for the wishlist?

About mishaps with "dd", I think everyone  who ever dealt with a
system  (not just Linux)  on the level we do has sometime gone through
something similar... the last time I remember doing this was many
years ago, before  Linux existed, when me and a few friends spent a
wonderful night installing  William Jolitz ' then-new 386/BSD  on a HD
 (a process which *required*  dd)  and trashing its Windows partitions
(which contained the only copy of the graduation thesis of one of us,
due in a few days).

Thanks for all the help,
--
   Durval Menezes.

On Mon, Jun 6, 2011 at 12:54 PM, Brad Campbell <brad@fnarfbargle.com> wrote:
>
> On 06/06/11 23:37, Drew wrote:
>>>
>>> Now, if I'm off the wall and missing something blindingly obvious feel free
>>> to thump me with a clue bat (it would not be the first time).
>>>
>>> I've lost 2 arrays recently. 8TB to a dodgy controller (thanks SIL), and 2TB
>>> to complete idiocy on my part, so I know the sting of lost or corrupted
>>> data.
>>
>> I think you've covered the process in more detail, including pitfalls,
>> then I have. :-) Only catch is where would you find a cheap 2-3TB
>> drive right now?
>
> I bought 10 recently for about $90 each. It's all relative, but I consider ~$45 / TB cheap.
>
>> I also know the sting of mixing stupidity and dd. ;-) A friend was
>> helping me do some complex rework with dd on one of my disks. Being
>> the n00b I followed his instructions exactly, and him being the expert
>> (and assuming I wasn't the n00b I was back then) didn't double check
>> my work. Net result was I backed the MBR/Partition Table up using dd,
>> but did so to a partition on the drive we were working on. There may
>> have been some alcohol involved (I was in University), the revised
>> data we inserted failed, and next thing you know I'm running Partition
>> Magic (the gnu tools circa 2005 failed to detect anything) to try and
>> recover the partition table. No backups obviously. ;-)
>
> Similar to my
>
> dd if=/dev/zero of=/dev/sdb bs=1M count=100
>
> except instead of the target disk, it was to a raid array member that was currently active. To its credit, ext3 and fsck managed to give me most of my data back, even if I had to spend months intermittently sorting/renaming inode numbers from lost+found into files and directories.
>
> I'd like to claim Alcohol as a mitigating factor (hell, it gets people off charges in our court system all the time) but unfortunately I was just stupid.
>
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Maximizing failed disk replacement on a RAID5 array

[Durval Menezes]

Hello Folks,

Just finished the "repair". It completed OK, and over SMART the HD now
shows a "Reallocated_Sector_Ct" of 291 (which shows that many bad
sectors have been remapped), but it's also still reporting 4
"Current_Pending_Sector" and 4 "Offline_Uncorrectable"... which I
think means exactly the same thing, ie, that there are 4 "active"
(from the HD perspective) sectors on the drive still detected as bad
and not remapped.

I've been thinking about exactly what that means, and I think that
these 4 sectors are either A) outside the RAID partition (not very
probable as this partition occupies more than 99.99% of the disk,
leaving just a small, less than 105MB area at the beginning), or B)
some kind of metadata or unused space that hasn't been read and
rewritten by the "repair" I've just completed. I've just done a "dd
bs=1024k count=105 </dev/DISK >/dev/null" to account for the
hyphotesys A), and come out empty: no errors, and the drive still
shows 4 bad, unmapped sectors on SMART.

So, by elimination, it must be either case B) above, or a bug in the
linux md code (which prevents it from hitting every needed block on
the disk), or a bug in SMART (which makes it report inexistent bad
sectors). I've just started running a "smart long test" on the disk
(which will try to read all of its sectors, reporting the first error
by LBA) and see what happens. If it shows no errors, I will know it's
a SMART bug. If it shows errors, it must be in a unused/metadata block
or a bug in linux md.

Either way, my plan is then to try a  plain "dd" (no "dd_repair", at
least not now) of this failing disk to a new one; if it goes by
without any errors, I will know it's a bug in SMART. if it hits any
errors, I will have the first errors position (from "dd" point of
view) and then I will try and dump that specific sector with dd_repair
and examine it.

I will keep you posted.

Cheers,
-- 
   Durval Menezes.


On Mon, Jun 6, 2011 at 3:06 PM, Durval Menezes <durval.menezes@gmail.com> wrote:
> Hello Brad, Drew,
>
> Thanks for reminding me of the hammering a RAID level conversion would  cause.
> This is certainly a major  reason to avoid the RAID5->RAID6->RAID5 route.
>
> The "repair" has been running here for a few days already, with the
> server online, and ought to finish in 24 more hours. So far (thanks to
> the automatic rewrite relocation) the number of  uncorrectable sectors
> being reported by SMART has dropped from 40 to 20 , so it seems the
> repair is  doing its job. Lets just hope the disk has enough  spare
> sectors  to remap all the bad sectors; if it does, a simple "dd "from
> the bad disk to  its replacement ought to  do the job  (as you have
> indicated).
>
> On the other hand, as this "dd" has to be done with the array offline,
> it will entail in some downtime (although not as much as having to
> restore the whole array from backups).... not ideal, but not too bad
> either.
>
> In case worst comes to worst, I have an up-to-date offline backup of
> the contents of the whole array, so if something really bad happens, I
> have something to restore from.
>
> It would be great to have a
> "duplicate-this-bad-old-disk-into-this-shiny-new-disk"  functionality,
> as it would enable  an almost-no-downtime disk replacement with
> minimum  risk, but it seems we can't have everything... :-0 Maybe it's
> something for the wishlist?
>
> About mishaps with "dd", I think everyone  who ever dealt with a
> system  (not just Linux)  on the level we do has sometime gone through
> something similar... the last time I remember doing this was many
> years ago, before  Linux existed, when me and a few friends spent a
> wonderful night installing  William Jolitz ' then-new 386/BSD  on a HD
>  (a process which *required*  dd)  and trashing its Windows partitions
> (which contained the only copy of the graduation thesis of one of us,
> due in a few days).
>
> Thanks for all the help,
> --
>    Durval Menezes.
>
> On Mon, Jun 6, 2011 at 12:54 PM, Brad Campbell <brad@fnarfbargle.com> wrote:
>>
>> On 06/06/11 23:37, Drew wrote:
>>>>
>>>> Now, if I'm off the wall and missing something blindingly obvious feel free
>>>> to thump me with a clue bat (it would not be the first time).
>>>>
>>>> I've lost 2 arrays recently. 8TB to a dodgy controller (thanks SIL), and 2TB
>>>> to complete idiocy on my part, so I know the sting of lost or corrupted
>>>> data.
>>>
>>> I think you've covered the process in more detail, including pitfalls,
>>> then I have. :-) Only catch is where would you find a cheap 2-3TB
>>> drive right now?
>>
>> I bought 10 recently for about $90 each. It's all relative, but I consider ~$45 / TB cheap.
>>
>>> I also know the sting of mixing stupidity and dd. ;-) A friend was
>>> helping me do some complex rework with dd on one of my disks. Being
>>> the n00b I followed his instructions exactly, and him being the expert
>>> (and assuming I wasn't the n00b I was back then) didn't double check
>>> my work. Net result was I backed the MBR/Partition Table up using dd,
>>> but did so to a partition on the drive we were working on. There may
>>> have been some alcohol involved (I was in University), the revised
>>> data we inserted failed, and next thing you know I'm running Partition
>>> Magic (the gnu tools circa 2005 failed to detect anything) to try and
>>> recover the partition table. No backups obviously. ;-)
>>
>> Similar to my
>>
>> dd if=/dev/zero of=/dev/sdb bs=1M count=100
>>
>> except instead of the target disk, it was to a raid array member that was currently active. To its credit, ext3 and fsck managed to give me most of my data back, even if I had to spend months intermittently sorting/renaming inode numbers from lost+found into files and directories.
>>
>> I'd like to claim Alcohol as a mitigating factor (hell, it gets people off charges in our court system all the time) but unfortunately I was just stupid.
>>
>
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Maximizing failed disk replacement on a RAID5 array

[Brad Campbell]

On 07/06/11 13:03, Durval Menezes wrote:
> Hello Folks,
>
> Just finished the "repair". It completed OK, and over SMART the HD now
> shows a "Reallocated_Sector_Ct" of 291 (which shows that many bad
> sectors have been remapped), but it's also still reporting 4
> "Current_Pending_Sector" and 4 "Offline_Uncorrectable"... which I
> think means exactly the same thing, ie, that there are 4 "active"
> (from the HD perspective) sectors on the drive still detected as bad
> and not remapped.
>
> I've been thinking about exactly what that means, and I think that
> these 4 sectors are either A) outside the RAID partition (not very
> probable as this partition occupies more than 99.99% of the disk,
> leaving just a small, less than 105MB area at the beginning), or B)
> some kind of metadata or unused space that hasn't been read and
> rewritten by the "repair" I've just completed. I've just done a "dd
> bs=1024k count=105</dev/DISK>/dev/null" to account for the
> hyphotesys A), and come out empty: no errors, and the drive still
> shows 4 bad, unmapped sectors on SMART.
>
> So, by elimination, it must be either case B) above, or a bug in the
> linux md code (which prevents it from hitting every needed block on
> the disk), or a bug in SMART (which makes it report inexistent bad
>
Try running a SMART long test smartctl -t long and it will tell you whether the sectors are really 
bad or not.
I've had instances where the firmware still thought that some previously pending sectors were still 
pending until I forced a test, at which time the drive came to its senses and they went away.

I believe if you wait until the drive gets around to doing its periodic offline data collection 
you'll see the same thing, but a long test is nice as it will give you an actual block number for 
the first failure (if you have one)

Re: Maximizing failed disk replacement on a RAID5 array

[John Robinson]

On 06/06/2011 19:06, Durval Menezes wrote:
[...]
> It would be great to have a
> "duplicate-this-bad-old-disk-into-this-shiny-new-disk"  functionality,
> as it would enable  an almost-no-downtime disk replacement with
> minimum  risk, but it seems we can't have everything... :-0 Maybe it's
> something for the wishlist?

It's already on the wishlist, described as a hot replace.

Cheers,

John.

Re: Maximizing failed disk replacement on a RAID5 array

[Durval Menezes]

Hello,

On Tue, Jun 7, 2011 at 2:35 AM, Brad Campbell <brad@fnarfbargle.com> wrote:
> On 07/06/11 13:03, Durval Menezes wrote:
>>
>> Hello Folks,
>>
>> Just finished the "repair". It completed OK, and over SMART the HD now
>> shows a "Reallocated_Sector_Ct" of 291 (which shows that many bad
>> sectors have been remapped), but it's also still reporting 4
>> "Current_Pending_Sector" and 4 "Offline_Uncorrectable"... which I
>> think means exactly the same thing, ie, that there are 4 "active"
>> (from the HD perspective) sectors on the drive still detected as bad
>> and not remapped.
>>
>> I've been thinking about exactly what that means, and I think that
>> these 4 sectors are either A) outside the RAID partition (not very
>> probable as this partition occupies more than 99.99% of the disk,
>> leaving just a small, less than 105MB area at the beginning), or B)
>> some kind of metadata or unused space that hasn't been read and
>> rewritten by the "repair" I've just completed. I've just done a "dd
>> bs=1024k count=105</dev/DISK>/dev/null" to account for the
>> hyphotesys A), and come out empty: no errors, and the drive still
>> shows 4 bad, unmapped sectors on SMART.
>>
>> So, by elimination, it must be either case B) above, or a bug in the
>> linux md code (which prevents it from hitting every needed block on
>> the disk), or a bug in SMART (which makes it report inexistent bad
>>
> Try running a SMART long test smartctl -t long and it will tell you whether
> the sectors are really bad or not.
> I've had instances where the firmware still thought that some previously
> pending sectors were still pending until I forced a test, at which time the
> drive came to its senses and they went away.
>
> I believe if you wait until the drive gets around to doing its periodic
> offline data collection you'll see the same thing, but a long test is nice
> as it will give you an actual block number for the first failure (if you
> have one)

I did it (smartctl -a long) and it completed (registering an error at
the very end of the disk):

     SMART Self-test log structure revision number 1
     Num  Test_Description    Status                  Remaining
LifeTime(hours)  LBA_of_first_error
     # 1  Extended offline    Completed: read failure       10%
9942           2930273794

The SMART Attributes table still shows 4 pending/uncorrectable sectors:

    197 Current_Pending_Sector  0x0012   100   100   000    Old_age
Always       -
           4
    198 Offline_Uncorrectable   0x0010   100   100   000    Old_age
Offline      -
           4

Converting the above LBA to a block number, I find 2930273794/2=
1465136897; as this is a 1.5TB HD,
this first error (there are possibly 3 more) is right at the final
35GB of the media, so it's inside (near the
end) of the RAID partition:

     fdisk -l /dev/sdc
         Disk /dev/sdc: 1500.3 GB, 1500301910016 bytes
          255 heads, 63 sectors/track, 182401 cylinders
          Units = cylinders of 16065 * 512 = 8225280 bytes
          Sector size (logical/physical): 512 bytes / 512 bytes
          I/O size (minimum/optimal): 512 bytes / 512 bytes
          Disk identifier: 0x6be6057c
             Device Boot      Start         End      Blocks   Id  System
          /dev/sdc1               1           1        8001    4  FAT16 <32M
          /dev/sdc2   *           2          14      104422+  83  Linux
          /dev/sdc3              15      182401  1465023577+  fd
Linux raid autodetect

Confirming that this block is indeed returning read errors:

    dd count=1 bs=1024 skip=1465136897 if=/dev/sdc of=/dev/null
        [long delay]
        dd: reading `/dev/sdc': Input/output error
        0+0 records in
        0+0 records out
        0 bytes (0 B) copied, 45.1076 s, 0.0 kB/s

Examining one sector before:

    dd count=1 bs=1024 skip=146513686 if=/dev/sdc | hexdump -C
        00000000  92 e1 b4 d4 c6 cd 0f 33  db 7c ff a9 be c1 c1 8e
|.......3.|......|
        00000010  71 35 fc 55 16 c4 36 ef  59 10 db 20 22 f4 57 99
|q5.U..6.Y.. ".W.|
        00000020  31 61 2b 24 e0 98 3c 94  4b 8a 17 93 23 aa e9 96
|1a+$..<.K...#...|
        00000030  b0 47 7b 8f 12 c6 52 42  99 0d 72 b4 51 02 5a 8e
|.G{...RB..r.Q.Z.|
        00000040  c6 5a ac 86 0b a5 74 9b  13 e7 87 7a db 94 e2 7f
|.Z....t....z....|
        00000050  c6 42 75 ba 53 bf 7f 20  fc 9c ad 4b 8f 3c 85 64
|.Bu.S.. ...K.<.d|
        00000060  3a b0 ac 41 6e 41 fb 95  03 70 24 7e 2e d5 df 8a
|:..AnA...p$~....|
        00000070  f9 dc d1 7d 4a 1e e1 93  9d 39 18 83 6c 9f 9f 79
|...}J....9..l..y|
        00000080  53 a3 d1 fb 7f c6 bd 44  8d 0c 40 06 0a 92 f9 7e
|S......D..@....~|
        00000090  0c 0e 87 43 66 9d fc 12  2b 0d 7a 34 ba 84 cb 73
|...Cf...+.z4...s|
        000000a0  47 3b a4 fa c9 50 d9 96  f9 50 a2 60 17 eb 7c c8
|G;...P...P.`..|.|
        000000b0  42 76 59 d0 1e 06 10 a8  3b 89 74 8d b4 04 83 88
|BvY.....;.t.....|
        000000c0  d7 9d 3c 82 cf 8f 7d 6e  a2 b6 bf 56 06 c0 aa 7c
|..<...}n...V...||
        000000d0  7d 39 ae 0a 67 48 28 b5  07 fd fc ae 49 e4 7a 08
|}9..gH(.....I.z.|
        000000e0  8a 37 94 e0 d3 d7 f0 f4  4c 49 3a ed b7 f4 84 95
|.7......LI:.....|
        000000f0  3f 0a 4f 6c 47 62 1a f4  70 ca 14 8a 52 6d 4c 1e
|?.OlGb..p...RmL.|
        00000100  da 0c 29 17 c1 a4 e1 5c  cb 43 e0 01 45 9c 72 7f
|..)....\.C..E.r.|
        00000110  78 b8 19 3f dd 35 c5 50  ff 9b 42 fb 0b d8 61 5a
|x..?.5.P..B...aZ|
        00000120  24 2b ae c9 45 e6 e5 e9  04 00 93 bb 53 c0 fd d6
|$+..E.......S...|
        00000130  9c ab 69 98 50 f0 5e 98  0d 0b b3 dc cb cb d0 7d
|..i.P.^........}|
        00000140  21 70 68 e8 fb 3c 55 fd  2d c6 6c 25 86 dd 9a 4a
|!ph..<U.-.l%...J|
        00000150  fc e2 24 a9 fb 9a 6b be  d5 e2 3b e9 a0 b1 61 ad
|..$...k...;...a.|
        00000160  1f 9a c8 31 86 91 c6 1f  86 9e 17 35 25 7e 77 42
|...1.......5%~wB|
        00000170  37 86 b2 17 08 8e c4 cf  4e e2 64 7d 83 11 05 1e
|7.......N.d}....|
        00000180  6b c1 e7 5d 0f e2 c9 f9  0a 0a b1 2b 83 a1 2a a4
|k..].......+..*.|
        00000190  1d f8 a6 13 2f e9 45 bb  b7 e2 71 e9 69 ad 3c 47
|..../.E...q.i.<G|
        000001a0  3f fa 39 7f 1e 93 0e d2  89 09 dc d2 b3 3b f8 6f
|?.9..........;.o|
        000001b0  21 21 72 b6 9e 9d 42 79  fb 78 3c 02 85 7b 1f 4f
|!!r...By.x<..{.O|
        000001c0  8b 3c 26 62 8a 58 38 a7  48 31 b9 e2 0c 0d 41 d6
|.<&b.X8.H1....A.|
        000001d0  8f 43 95 f0 1f 52 3e 0e  55 8d c0 93 f7 e3 c8 79
|.C...R>.U......y|
        000001e0  a2 bc 51 72 87 3c 16 c3  d0 f3 57 a8 e4 48 51 32
|..Qr.<....W..HQ2|
        000001f0  00 99 3e 0e 88 a3 fa e3  00 a4 c2 cb 28 7a a1 00
|..>.........(z..|
        00000200  a0 b4 1b 6d c4 2a 15 75  a3 f0 24 47 5a d6 54 74
|...m.*.u..$GZ.Tt|
        00000210  d0 ad e4 92 b1 99 5d 7a  62 47 b9 54 8f 9e 15 ca
|......]zbG.T....|
        00000220  65 09 9e d0 d3 61 51 93  88 4a 46 1e 5c 15 07 ef
|e....aQ..JF.\...|
        00000230  b0 92 fa a7 e7 3d e5 36  20 67 d2 24 b7 59 ae f4
|.....=.6 g.$.Y..|
        00000240  7c 26 57 90 e1 69 b5 f3  b4 1b 8e e6 07 2e 46 84
||&W..i........F.|
        1+0 records in
        1+0 records out
        1024 bytes (1.0 kB) copied, 5.0224e-05 s, 20.4 MB/s

Looking at one sector after the error returns similar results.

So, I don't know about you, but the above seems pretty much like data
to me (although it could also be parity).

So I have two questions:

1) can I simply skip over these sectors (using dd_rescue or multiple
dd invocations) when off-line copying the old disk to the new one,
trusting the RAID5 to reconstruct the data correctly from the other 2
disks? Or is it better to simply do the recover the "traditional" way
(ie, "fail" the old disk, "add" the new one, and run the risk of a
possible bad sector on one of the two remaining old disks ruining the
show completely and forcing me to recover from backups [I *do* have
up-to-date backups on this array])?

2) Is there a formula, a program or anything that can tell me exactly
what is located at the above sector (ie, whether it's RAID parity or a
data sector)?

Thanks,
-- 
   Durval Menezes.




Ditto, one sector after:



So, when I "dd" this partition to a new one, I think



>
>

Re: Maximizing failed disk replacement on a RAID5 array

[Brad Campbell]

On 08/06/11 14:58, Durval Menezes wrote:

> 1) can I simply skip over these sectors (using dd_rescue or multiple
> dd invocations) when off-line copying the old disk to the new one,
> trusting the RAID5 to reconstruct the data correctly from the other 2

Noooooooooooo. As we stated early on, it you do that md will have no 
idea that the data missing is actually missing as the drive won't return 
a read error.

does a repair take long on your machine? I find that a few repair runs 
generally gets me enough re-writes to clear the dud sectors and allow an 
offline clone.

If your dd of the old disk to the new disk aborts with an error, do 
_not_ under any circumstances (well, unless you have really good 
backups) do a dd_rescue and just swap the disks.

Re: Maximizing failed disk replacement on a RAID5 array

[Durval Menezes]

Hello Brad,

On Wed, Jun 8, 2011 at 4:32 AM, Brad Campbell <brad@fnarfbargle.com> wrote:
> On 08/06/11 14:58, Durval Menezes wrote:
>
>> 1) can I simply skip over these sectors (using dd_rescue or multiple
>> dd invocations) when off-line copying the old disk to the new one,
>> trusting the RAID5 to reconstruct the data correctly from the other 2
>
> Noooooooooooo. As we stated early on, it you do that md will have no idea
> that the data missing is actually missing as the drive won't return a read
> error.

Even if a "repair" (echo "repair" >/sys/block/md1/md/sync_status,
checking progress with "cat /proc/mdstat" and completion with "tail -f
/var/log/messages | grep md" ) finishes with no errors?

> does a repair take long on your machine? I find that a few repair runs
> generally gets me enough re-writes to clear the dud sectors and allow an
> offline clone.

I'm sorry if I did not make myself clear; I've already run both a
"repair" on the RAID  (see above) and a "smart -t long" on the
particular disk... I had about 40 bad sectors before, and now have
just 4, but these 4 sectors persist as being marked in error... I
think the "RAID repair" didn't touch them.

Cheers,
-- 
  Durval.

Re: Maximizing failed disk replacement on a RAID5 array

[Brad Campbell]

On 08/06/11 15:47, Durval Menezes wrote:

> I'm sorry if I did not make myself clear; I've already run both a
> "repair" on the RAID  (see above) and a "smart -t long" on the
> particular disk... I had about 40 bad sectors before, and now have
> just 4, but these 4 sectors persist as being marked in error... I
> think the "RAID repair" didn't touch them.

Apologies, I obviously missed that fact.

I think your best course of action in this case is to test both the 
other drives with SMART long checks and fail/replace the faulty one.

I've never had md not report a repaired sector when performing a repair 
operation.

I'll just re-iterate, if you take the bad sectors away without a good 
copy of the data on them, md won't know it is supposed to reconstruct 
those missing sectors.

Hrm.. *or*, and this is a big *or* you could use hdparm to create 
correctable bad sectors on the copy at the appropriate LBA's, and md 
should do the right thing as it will get read errors from those, which 
will go away when they are re-written.

I'd not thought of that before, but it should do the trick.

Re: Maximizing failed disk replacement on a RAID5 array

[John Robinson]

On 07/06/2011 09:52, John Robinson wrote:
> On 06/06/2011 19:06, Durval Menezes wrote:
> [...]
>> It would be great to have a
>> "duplicate-this-bad-old-disk-into-this-shiny-new-disk" functionality,
>> as it would enable an almost-no-downtime disk replacement with
>> minimum risk, but it seems we can't have everything... :-0 Maybe it's
>> something for the wishlist?
>
> It's already on the wishlist, described as a hot replace.

Actually I've been thinking about this. I think I'd rather the hot 
replace functionality did a normal rebuild from the still-good drives, 
and only if it came across a read error from those would it attempt to 
refer to the contents of the known-to-be-failing drive (and then also 
attempt to repair the read error on the supposedly-still-good drive that 
gave a read error, as already happens).

My rationale for this is as follows: if we want to hot-replace a drive 
that's known to be failing, we should trust it less than the remaining 
still-good drives, and treat it with kid gloves. It may be suffering 
from bit-rot. We'd rather not hit all the bad sectors on the failing 
drive, because each time we do that we send the drive into 7 seconds (or 
more, for cheap drives without TLER) of re-reading, plus any Linux-level 
re-reading there might be. Further, making the known-to-be-failing drive 
work extra hard (doing the equivalent of dd'ing from it while also still 
using it to serve its contents as an array member) might make it die 
completely before we've finished.

What will this do for rebuild time? Well, I don't think it'll be any 
slower. On the one hand, you'd think that copying from one drive to 
another would be faster than a rebuild, because you're only reading 1 
drive instead of N-1, but on the other, your array is going to run 
slowly (pretty much degraded speed) anyway because you're keeping one 
drive in constant use reading from it, and you risk it becoming much, 
much slower if you do run in to hundreds or thousands of read errors on 
the failing drive.

So overall I think hot-replace should be a normal replace with a 
possible second source of data/parity.

Thoughts?

Yes, I know, -ENOPATCH

Cheers,

John.

Re: Maximizing failed disk replacement on a RAID5 array

[Durval Menezes]

Hello John,

On Fri, Jun 10, 2011 at 7:25 AM, John Robinson
<john.robinson@anonymous.org.uk> wrote:
> On 07/06/2011 09:52, John Robinson wrote:
>>
>> On 06/06/2011 19:06, Durval Menezes wrote:
>> [...]
>>>
>>> It would be great to have a
>>> "duplicate-this-bad-old-disk-into-this-shiny-new-disk" functionality,
>>> as it would enable an almost-no-downtime disk replacement with
>>> minimum risk, but it seems we can't have everything... :-0 Maybe it's
>>> something for the wishlist?
>>
>> It's already on the wishlist, described as a hot replace.
>
> Actually I've been thinking about this. I think I'd rather the hot replace
> functionality did a normal rebuild from the still-good drives, and only if
> it came across a read error from those would it attempt to refer to the
> contents of the known-to-be-failing drive (and then also attempt to repair
> the read error on the supposedly-still-good drive that gave a read error, as
> already happens).

This looks like a very good idea. The old (failing) drive would be
kept "on reserve", ready to be accessed for eventual failed sectors on
the other old (good) drives...

> My rationale for this is as follows: if we want to hot-replace a drive
> that's known to be failing, we should trust it less than the remaining
> still-good drives, and treat it with kid gloves. It may be suffering from
> bit-rot. We'd rather not hit all the bad sectors on the failing drive,
> because each time we do that we send the drive into 7 seconds (or more, for
> cheap drives without TLER) of re-reading, plus any Linux-level re-reading
> there might be. Further, making the known-to-be-failing drive work extra
> hard (doing the equivalent of dd'ing from it while also still using it to
> serve its contents as an array member) might make it die completely before
> we've finished.

I agree completely.

> What will this do for rebuild time? Well, I don't think it'll be any slower.

I think it will actually be faster.

> On the one hand, you'd think that copying from one drive to another would be
> faster than a rebuild, because you're only reading 1 drive instead of N-1,
> but on the other, your array is going to run slowly (pretty much degraded
> speed) anyway because you're keeping one drive in constant use reading from
> it, and you risk it becoming much, much slower if you do run in to hundreds
> or thousands of read errors on the failing drive.
>
> So overall I think hot-replace should be a normal replace with a possible
> second source of data/parity.

Your reasoning sounds good to me.

> Thoughts?

Only sadness that it's not implemented yet... :-)

> Yes, I know, -ENOPATCH

Exactly :-)

Cheers,
-- 
   Durval Menezes.


>
> Cheers,
>
> John.
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Maximizing failed disk replacement on a RAID5 array

[Durval Menezes]

Hello Folks,

On Wed, Jun 8, 2011 at 10:21 AM, Brad Campbell
<lists2009@fnarfbargle.com> wrote:
>
> Best of luck, and let us know how you get on.

Just finished the process here. To summarize, seems I've got my array
back in a stable state.

What I did:

1) Got a good backup of all the data in the array (using "tar") to
   removable HDs, verified it (using md5sum), and then stored these
   HDs safely offline;

2) Unmounted the filesystem in the array;

3) inserted the replacement disk on a USB dock, partitioned it,
   then added it to the array ("mdadm --add");
    -> Verified (via "mdadm --detail") that the replacement disk was
       listed on the array as a "spare";

4) failed the bad disk in the array ("mdadm --fail")
   -> At that point, the array immediatelly started to resync into the
      replacement disk;

5) Monitored the resync process via "cat /proc/mdstat": it took
   roughly 11 hours (I guess because transfer speed to the replacement
   disk was limited by the USB ~40MB/s speed limit), but it signaled
   no errors;

6) Verified that the array was really synced ("mdadm --detail") and
   that there were indeed no errors during the resync (less
   /var/log/messages);

7) removed the bad disk logically from the array ("mdadm --remove");

8) shut down the machine (init 0);

9) removed the bad disk physically from the machine, ejected the
   replacement disk from the USB dock, and then installed the
   replacement disk inside the machine;

10) turned the system on: the OS booted, assembled the array and
    mounted the filesystem in it with no issues;

11) checked (using "md5sum -c" on the md5sum files generated during
    pass#1 above) that all that ON THE ARRAY was indeed correct, so
    in the end I didn't need to restore anything from backup.

Thanks for all the help, folks, and I pray we have the "hot-replace"
functionality implemented soon... it will make for much sounder sleep
the next time one of my disks fails... :-)

Cheers,
--
  Durval Menezes.
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Maximizing failed disk replacement on a RAID5 array

[Durval Menezes]

Hello Brad, folks,

On Wed, Jun 8, 2011 at 4:57 AM, Brad Campbell <lists2009@fnarfbargle.com> wrote:
> On 08/06/11 15:47, Durval Menezes wrote:
>
>> I'm sorry if I did not make myself clear; I've already run both a
>> "repair" on the RAID  (see above) and a "smart -t long" on the
>> particular disk... I had about 40 bad sectors before, and now have
>> just 4, but these 4 sectors persist as being marked in error... I
>> think the "RAID repair" didn't touch them.
[...]
> I've never had md not report a repaired sector when performing a repair
> operation.

Just to keep you posted: after I finished replacing that failing disk,
I wrote zeroes to all its sectors (dd bs=16065b </dev/zero >/dev/DISK)
and then checked SMART again.

Guess what? As expected, the 4 sectors that were being reported in
both "Current_Pending_Sector" and "Offline_Uncorrectable" just went
away... both counters now report a big round  "0".

So, it seems my hypothesis that these sectors were not being touched
by the md "repair" was indeed correct: once they were written to, they
ended up being automatically remapped (or at least cleaned out of the
above counters) by the HD firmware. Also, as the array resync reported
no errors whatsoever, it seems these sectors were simply not being
used by md.

Anyone has a good explanation to that? Inquiring minds want to know... :-)

Cheers,
-- 
   Durval Menezes.
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html