How to force rewrite of a smart detected bad block with raid5: checkarray?

How to force rewrite of a smart detected bad block with raid5: checkarray?

[Marc MERLIN]

I couldn't quite find the answer to this with a search.

Smart reported a bad block (pending).
I confirmed it with:

hdrecover /dev/sdk
(...)
Sector 3907017780 (99%) ETR: 0 seconds
Error at sector 3907029168
Attempting to pounce on it...
Attempt 1 from sector   3219409027: FAILED
Attempt 2 from sector   1546846730: FAILED
(...)
Attempt 19 from sector   3837517549: FAILED
Attempt 20 from sector   2281104567: FAILED
Couldn't recover sector 3907029168
The data for this sector could not be recovered. However, destroying the
contents of this sector (ie writing zeros to it) should cause the hard disk
to reallocate it making the drive useable again
Do you really want to destroy the data in sector 3907029168?     [ (y)es / (n)o / (a)ll / (q)uit ]:^C

Now, hdrecover had no idea that /dev/sdk is part of a raid5, so I'm not going to
use it to rewrite the bad block.

I know I can remove /dev/sdk1 from my array and put it back in, but that's
unsafe because if during a rebuild another block from another drive happens
to be bad, I'm then left with a double raid failure.

So, I tried this: /usr/share/mdadm/checkarray md7

md7 : active raid5 sdi1[0] sdh1[5] sdl1[3] sdk1[2] sdj1[1]
      7814045696 blocks super 1.2 level 5, 512k chunk, algorithm 2 [5/5] [UUUUU]
      [>....................]  check =  0.0% (447616/1953511424) finish=28435.6min speed=1144K/sec

But I couldn't quite find documentation on what the kernel does in this case.

Does this actually do what I want? Try all the blocks on each drive and if
one block is unreadable, it forces a rewrite on the bad drive using the n-1
other ones to recreate it?

Thanks,
Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems & security ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/  

Re: How to force rewrite of a smart detected bad block with raid5: checkarray?

[Mikael Abrahamsson]

On Tue, 18 Jan 2011, Marc MERLIN wrote:

> Does this actually do what I want? Try all the blocks on each drive and 
> if one block is unreadable, it forces a rewrite on the bad drive using 
> the n-1 other ones to recreate it?

You want to use the "repair" command. This has been discussed numerous 
times and discussions regarding it can be found in the archives.

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: How to force rewrite of a smart detected bad block with raid5: checkarray?

[NeilBrown]

On Wed, 19 Jan 2011 08:42:53 +0100 (CET) Mikael Abrahamsson
<swmike@swm.pp.se> wrote:

> On Tue, 18 Jan 2011, Marc MERLIN wrote:
> 
> > Does this actually do what I want? Try all the blocks on each drive and 
> > if one block is unreadable, it forces a rewrite on the bad drive using 
> > the n-1 other ones to recreate it?
> 
> You want to use the "repair" command. This has been discussed numerous 
> times and discussions regarding it can be found in the archives.
> 

No, he actually wants the 'check' command, though 'repair' would work too.

All you need to do is get md/raid5 to try reading the bad block.  Once it does
that it will get a read error and automagically try to correct it.

You could do the math, figure out where in the array the bad block is and use
e.g. dd to just read that block.  That could work, but if the block is a
parity block, then it probably won't.

'check'  (i.e. echo check > /sys/block/mdXX/md/sync_action) will cause
md/raid5 to read all blocks on all devices, thus auto-repairing any
unreadable blocks.

'repair' will also do that but if it find that the parity looks wrong, it
will try to correct the parity.  It is generally best to only use 'repair'
when you have good reason to believe that you need it.


If you were really keen, you could 
  cd /sys/block/mdXX/md
  echo 3907029168 > sync_min
  echo 3907029170 > sync_max
  echo check > sync_action

and it will just sync that stripe.
Though you need to adjust for any Data Offset if using 1.1 or 1.2 metadata,
and you might need to round both numbers of to the next page or maybe next
chunk, I'm not sure.

NeilBrown

Re: How to force rewrite of a smart detected bad block with raid5: checkarray?

[Marc MERLIN]

On Wed, Jan 19, 2011 at 08:41:15PM +1100, NeilBrown wrote:
> All you need to do is get md/raid5 to try reading the bad block.  Once it does
> that it will get a read error and automagically try to correct it.
 
So, if I get this right, raid5 only reads n-1 drives. Unless I'm unlucky
enough to have the bad disk be the parity stripe, just reading the file with
a bad stripe by luck would cause the kernel to recompute parity on the read
error and re-write the bad block?
(I also read in the online docs that raid4 actually reads all the blocks,
including parity, which is a bit slower, but would actually guarantee that
all blocks are read, and parity is still consistent at ready time?)


But back to your point: check, which I had started, will indeed do what I
was hoping it would, thanks.

> If you were really keen, you could 
>   cd /sys/block/mdXX/md
>   echo 3907029168 > sync_min
>   echo 3907029170 > sync_max
>   echo check > sync_action

I stopped the full check, and tried:
 
gargamel:/sys/block/md7/md# cat sync_min
244188936
gargamel:/sys/block/md7/md# cat sync_max
max
gargamel:/sys/block/md7/md# echo 3907029168 > sync_min
bash: echo: write error: Invalid argument

Any idea what went wrong here?

gargamel:/sys/block/md7/md# mdadm --detail /dev/md7
/dev/md7:
        Version : 1.02
  Creation Time : Thu Mar 25 20:15:00 2010
     Raid Level : raid5
     Array Size : 7814045696 (7452.05 GiB 8001.58 GB)
  Used Dev Size : 1953511424 (1863.01 GiB 2000.40 GB)
   Raid Devices : 5
  Total Devices : 5
    Persistence : Superblock is persistent

    Update Time : Wed Jan 19 09:27:57 2011
          State : clean
 Active Devices : 5
Working Devices : 5
 Failed Devices : 0
  Spare Devices : 0
         
         Layout : left-symmetric
     Chunk Size : 512K
           
           Name : gargamel.svh.merlins.org:7  (local to host gargamel.svh.merlins.org)
           UUID : 5884576b:0e402a5d:8629093c:ec020760
         Events : 28714

    Number   Major   Minor   RaidDevice State
       0       8      129        0      active sync   /dev/sdi1
       1       8      145        1      active sync   /dev/sdj1
       2       8      161        2      active sync   /dev/sdk1
       3       8      177        3      active sync   /dev/sdl1
       5       8      113        4      active sync   /dev/sdh1


As for docs, a bit of googling before posting didn't help. I since then
found the new README.checkarray in my /usr/share/doc (debian), so that helps
although it doesn't talk about check vs repair.

Also, I didn't find anything about sync_action, check, and repair in the
mdadm man page (a pointer to
https://raid.wiki.kernel.org/index.php/RAID_Administration
would me useful).
Actually the above page still says that you can't check just a range of blocks.

Is there more up to date documentation that I should be reading somewhere?

Thanks for your answer,
Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems & security ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/  

Re: How to force rewrite of a smart detected bad block with raid5: checkarray?

[NeilBrown]

On Wed, 19 Jan 2011 09:31:50 -0800 Marc MERLIN <marc@merlins.org> wrote:

> On Wed, Jan 19, 2011 at 08:41:15PM +1100, NeilBrown wrote:
> > All you need to do is get md/raid5 to try reading the bad block.  Once it does
> > that it will get a read error and automagically try to correct it.
>  
> So, if I get this right, raid5 only reads n-1 drives. Unless I'm unlucky
> enough to have the bad disk be the parity stripe, just reading the file with
> a bad stripe by luck would cause the kernel to recompute parity on the read
> error and re-write the bad block?
> (I also read in the online docs that raid4 actually reads all the blocks,
> including parity, which is a bit slower, but would actually guarantee that
> all blocks are read, and parity is still consistent at ready time?)
> 
> 
> But back to your point: check, which I had started, will indeed do what I
> was hoping it would, thanks.
> 
> > If you were really keen, you could 
> >   cd /sys/block/mdXX/md
> >   echo 3907029168 > sync_min
> >   echo 3907029170 > sync_max
> >   echo check > sync_action
> 
> I stopped the full check, and tried:
>  
> gargamel:/sys/block/md7/md# cat sync_min
> 244188936
> gargamel:/sys/block/md7/md# cat sync_max
> max
> gargamel:/sys/block/md7/md# echo 3907029168 > sync_min
> bash: echo: write error: Invalid argument
> 
> Any idea what went wrong here?

Like I suggest, rounding might be necessary.
I just checked the code and this number does need to be a multiple of the
chunk size, which is 1024 sectors for you.
so 
   echo 3907028992 > sync_min

should do the trick.

NeilBrown

Re: How to force rewrite of a smart detected bad block with raid5: checkarray?

[Marc MERLIN]

On Thu, Jan 20, 2011 at 07:58:42AM +1100, NeilBrown wrote:
> Like I suggest, rounding might be necessary.
> I just checked the code and this number does need to be a multiple of the
> chunk size, which is 1024 sectors for you.
> so 
>    echo 3907028992 > sync_min
> 
> should do the trick.

I misunderstood, I thought you were worried that I wouldn't check the right
blocks, not "rounding or it won't be accepted by the kernel" :)

This worked fine, thanks for your help.

Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems & security ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/  

Re: How to force rewrite of a smart detected bad block with raid5: checkarray?

[Richard Scobie]

Marc MERLIN wrote:

> So, I went back and read man md, and md.txt in the kernel Documentation
> tree, but I could not find documentation on this:
>   echo 3907029168 > sync_min                                                                                                
>   echo 3907029170 > sync_max 
> 
> and as per my other post, it didn't work for me on 2.6.36
> (echo: write error: Invalid argument)

sync_max is mentioned in md.txt for 2.6.29.1 (currently running here):

   This is a number of sectors at which point a resync/recovery
      process will pause.  When a resync is active, the value can
      only ever be increased, never decreased.  The value of 'max'
      effectively disables the limit.

There is no mention of sync_min.

Regards,

Richard

Re: How to force rewrite of a smart detected bad block with raid5: checkarray?

[Marc MERLIN]

On Thu, Jan 20, 2011 at 07:01:33AM +1100, NeilBrown wrote:
> > > The kernel source, Documentation/md.txt.
> > 
> > Ah, yes of course. Didn't think about looking there, thanks.
> 
> "man md" is also an appropriate place to look.

Ah yes. Thanks for that too.

> > Mmmh, so I was curious as to how repair, when reading all the blocks of a
> > stripe with no read errors, and finding a parity mismatch, would know which
> > block was corrupted and needs to be rewritten.
> 
> It doesn't repair the data - that would be impossible.  It repairs the

that's what I thought.

> redundancy information which is all that raid really knows about.
> i.e. if it finds an inconsistency it re-writes the parity block.

Right, so it has only chance out of n to fix the right drive. Better than
nothing though.

> It isn't often useful.  But if you parity blocks are wrong somehow, then it
> can be useful.  It will not recover data that you have already lost, but it
> could make it less likely to lose more data.
 
Fair enough.

> md currently treats RAID6 just the same way as RAID5 - parity is re-written.
> It is possible that more could be done, but it isn't completely clear that it
> should - and it certainly isn't high on my priority list.

Understood.

So, I went back and read man md, and md.txt in the kernel Documentation
tree, but I could not find documentation on this:
  echo 3907029168 > sync_min                                                                                                
  echo 3907029170 > sync_max 

and as per my other post, it didn't work for me on 2.6.36
(echo: write error: Invalid argument)

Any suggestions?

Thanks,
Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems & security ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/  

Re: How to force rewrite of a smart detected bad block with raid5: checkarray?

[NeilBrown]

On Wed, 19 Jan 2011 10:49:14 -0800 Marc MERLIN <marc@merlins.org> wrote:

> On Thu, Jan 20, 2011 at 07:36:12AM +1300, Richard Scobie wrote:
> > Marc Merlin wrote:
> > 
> > > Also, I didn't find anything about sync_action, check, and repair in
> > > the mdadm man page (a pointer to
> > > https://raid.wiki.kernel.org/index.php/RAID_Administration
> > > would me useful).
> > > Actually the above page still says that you can't check just a range
> > > of blocks.
> > 
> > > Is there more up to date documentation that I should be reading
> > > somewhere?
> > 
> > The kernel source, Documentation/md.txt.
> 
> Ah, yes of course. Didn't think about looking there, thanks.

"man md" is also an appropriate place to look.

> 
> Mmmh, so I was curious as to how repair, when reading all the blocks of a
> stripe with no read errors, and finding a parity mismatch, would know which
> block was corrupted and needs to be rewritten.

It doesn't repair the data - that would be impossible.  It repairs the
redundancy information which is all that raid really knows about.
i.e. if it finds an inconsistency it re-writes the parity block.

> Given that I don't see how it can figure that out, I'm not even sure when
> repair would be useful for raid5 when there are no underlying media errors
> returned (raid6 obviously has redundant info and it's possible there).

It isn't often useful.  But if you parity blocks are wrong somehow, then it
can be useful.  It will not recover data that you have already lost, but it
could make it less likely to lose more data.

md currently treats RAID6 just the same way as RAID5 - parity is re-written.
It is possible that more could be done, but it isn't completely clear that it
should - and it certainly isn't high on my priority list.

NeilBrown


> 
> Not that I need repair in my case (check indeed does the right thing), I'm
> just not sure when repair would be useful.
> 
> Anyway, thanks again for the pointer.
> 
> Cheers,
> Marc

Re: How to force rewrite of a smart detected bad block with raid5: checkarray?

[Marc MERLIN]

On Thu, Jan 20, 2011 at 07:36:12AM +1300, Richard Scobie wrote:
> Marc Merlin wrote:
> 
> > Also, I didn't find anything about sync_action, check, and repair in
> > the mdadm man page (a pointer to
> > https://raid.wiki.kernel.org/index.php/RAID_Administration
> > would me useful).
> > Actually the above page still says that you can't check just a range
> > of blocks.
> 
> > Is there more up to date documentation that I should be reading
> > somewhere?
> 
> The kernel source, Documentation/md.txt.

Ah, yes of course. Didn't think about looking there, thanks.

Mmmh, so I was curious as to how repair, when reading all the blocks of a
stripe with no read errors, and finding a parity mismatch, would know which
block was corrupted and needs to be rewritten.
Given that I don't see how it can figure that out, I'm not even sure when
repair would be useful for raid5 when there are no underlying media errors
returned (raid6 obviously has redundant info and it's possible there).

Not that I need repair in my case (check indeed does the right thing), I'm
just not sure when repair would be useful.

Anyway, thanks again for the pointer.

Cheers,
Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems & security ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/  

Re: How to force rewrite of a smart detected bad block with raid5: checkarray?

[Richard Scobie]

Marc Merlin wrote:

 > Also, I didn't find anything about sync_action, check, and repair in
 > the mdadm man page (a pointer to
 > https://raid.wiki.kernel.org/index.php/RAID_Administration
 > would me useful).
 > Actually the above page still says that you can't check just a range
 > of blocks.

 > Is there more up to date documentation that I should be reading
 > somewhere?

The kernel source, Documentation/md.txt.

Regards,

Richard