Reshape stalled at first badblock location (was: RAID 5 --assemble doesn't recognize all overlays as component devices)

Reshape stalled at first badblock location (was: RAID 5 --assemble doesn't recognize all overlays as component devices)

[George Rapp]

On Sat, Feb 11, 2017 at 7:32 PM, George Rapp <george.rapp@gmail.com> wrote:
> Previous thread: http://marc.info/?l=linux-raid&m=148564798430138&w=2
> -- to summarize, while adding two drives to a RAID 5 array, one of the
> existing RAID 5 component drives failed, causing the reshape progress
> to stall at 77.5%. I removed the previous thread from this message to
> conserve space -- before resolving that situation, another problem has
> arisen.
>
> We have cloned and replaced the failed /dev/sdg with "ddrescue --force
> -r3 -n /dev/sdh /dev/sde c/sdh-sde-recovery.log"; copied in below, or
> viewable via https://app.box.com/v/sdh-sde-recovery . The failing
> device was removed from the server, and the RAID component partition
> on the cloned drive is now /dev/sdg4.

[previous thread snipped - after stepping through the code under gdb,
I realized that "mdadm --assemble --force" was needed.]

# uname -a
Linux localhost 4.3.4-200.fc22.x86_64 #1 SMP Mon Jan 25 13:37:15 UTC
2016 x86_64 x86_64 x86_64 GNU/Linux
# mdadm --version
mdadm - v3.3.4 - 3rd August 2015

As previously mentioned, the device that originally failed was cloned
to a new drive. This copy included the bad blocks list from the md
metadata, because I'm showing 23 bad blocks on the clone target drive,
/dev/sdg4:

# mdadm --examine-badblocks /dev/sdg4
Bad-blocks on /dev/sdg4:
          3802454640 for 512 sectors
          3802455664 for 512 sectors
          3802456176 for 512 sectors
          3802456688 for 512 sectors
          3802457200 for 512 sectors
          3802457712 for 512 sectors
          3802458224 for 512 sectors
          3802458736 for 512 sectors
          3802459248 for 512 sectors
          3802459760 for 512 sectors
          3802460272 for 512 sectors
          3802460784 for 512 sectors
          3802461296 for 512 sectors
          3802461808 for 512 sectors
          3802462320 for 512 sectors
          3802462832 for 512 sectors
          3802463344 for 512 sectors
          3802463856 for 512 sectors
          3802464368 for 512 sectors
          3802464880 for 512 sectors
          3802465392 for 512 sectors
          3802465904 for 512 sectors
          3802466416 for 512 sectors

However, when I run the following command to attempt to read each of
the bad blocks, no I/O errors pop up either on the command line or in
/var/log messages:

# for i in $(mdadm --examine-badblocks /dev/sdg4 | grep "512 sectors"
| cut -c11-20) ; do dd bs=512 if=/dev/sdg4 skip=$i count=512 | wc -c;
done

I've truncated the output, but in each case it is similar to this:

512+0 records in
512+0 records out
262144
262144 bytes (262 kB) copied, 0.636762 s, 412 kB/s

Thus, the bad blocks on the failed hard drive are apparently now
readable on the cloned drive.

When I try to assemble the RAID 5 array, though, the process gets
stuck at the location of the first bad block. The assemble command is:

# mdadm --assemble --force /dev/md4
--backup-file=/home/gwr/2017/2017-01/md4_backup__2017-01-25 /dev/sde4
/dev/sdf4 /dev/sdh4 /dev/sdl4 /dev/sdg4 /dev/sdk4 /dev/sdi4 /dev/sdj4
/dev/sdb4 /dev/sdd4
mdadm: accepting backup with timestamp 1485366772 for array with
timestamp 1487624068
mdadm: /dev/md4 has been started with 9 drives (out of 10).

The md4_raid5 process immediately spikes to 100% CPU utilization, and
the reshape stops at 1901225472 KiB (which is exactly half of the
first bad sector value, 3802454640):

# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md4 : active raid5 sde4[0] sdb4[12] sdj4[7] sdi4[8] sdk4[11] sdg4[10]
sdl4[9] sdh4[2] sdf4[1]
      13454923776 blocks super 1.1 level 5, 512k chunk, algorithm 2
[10/9] [UUUUUUUUU_]
      [===================>.]  reshape = 98.9% (1901225472/1922131968)
finish=2780.9min speed=125K/sec

unused devices: <none>

Googling around, I get the impression that resetting the badblocks
list is (a) not supported by the mdadm command; and (b) considered
harmful. However, if the blocks aren't really bad any more, as they
are now readable, does that risk still hold? How can I get this
reshape to proceed?

Updated mdadm --examine output is at
https://app.box.com/v/raid-status-2017-02-20

-- 
George Rapp  (Pataskala, OH) Home: george.rapp -- at -- gmail.com
LinkedIn profile: https://www.linkedin.com/in/georgerapp
Phone: +1 740 936 RAPP (740 936 7277)

Re: Reshape stalled at first badblock location (was: RAID 5 --assemble doesn't recognize all overlays as component devices)

[Tomasz Majchrzak]

On Mon, Feb 20, 2017 at 05:18:46PM -0500, George Rapp wrote:
> On Sat, Feb 11, 2017 at 7:32 PM, George Rapp <george.rapp@gmail.com> wrote:
> > Previous thread: http://marc.info/?l=linux-raid&m=148564798430138&w=2
> > -- to summarize, while adding two drives to a RAID 5 array, one of the
> > existing RAID 5 component drives failed, causing the reshape progress
> > to stall at 77.5%. I removed the previous thread from this message to
> > conserve space -- before resolving that situation, another problem has
> > arisen.
> >
> > We have cloned and replaced the failed /dev/sdg with "ddrescue --force
> > -r3 -n /dev/sdh /dev/sde c/sdh-sde-recovery.log"; copied in below, or
> > viewable via https://app.box.com/v/sdh-sde-recovery . The failing
> > device was removed from the server, and the RAID component partition
> > on the cloned drive is now /dev/sdg4.
> 
> [previous thread snipped - after stepping through the code under gdb,
> I realized that "mdadm --assemble --force" was needed.]
> 
> # uname -a
> Linux localhost 4.3.4-200.fc22.x86_64 #1 SMP Mon Jan 25 13:37:15 UTC
> 2016 x86_64 x86_64 x86_64 GNU/Linux
> # mdadm --version
> mdadm - v3.3.4 - 3rd August 2015
> 
> As previously mentioned, the device that originally failed was cloned
> to a new drive. This copy included the bad blocks list from the md
> metadata, because I'm showing 23 bad blocks on the clone target drive,
> /dev/sdg4:
> 
> # mdadm --examine-badblocks /dev/sdg4
> Bad-blocks on /dev/sdg4:
>           3802454640 for 512 sectors
>           3802455664 for 512 sectors
>           3802456176 for 512 sectors
>           3802456688 for 512 sectors
>           3802457200 for 512 sectors
>           3802457712 for 512 sectors
>           3802458224 for 512 sectors
>           3802458736 for 512 sectors
>           3802459248 for 512 sectors
>           3802459760 for 512 sectors
>           3802460272 for 512 sectors
>           3802460784 for 512 sectors
>           3802461296 for 512 sectors
>           3802461808 for 512 sectors
>           3802462320 for 512 sectors
>           3802462832 for 512 sectors
>           3802463344 for 512 sectors
>           3802463856 for 512 sectors
>           3802464368 for 512 sectors
>           3802464880 for 512 sectors
>           3802465392 for 512 sectors
>           3802465904 for 512 sectors
>           3802466416 for 512 sectors
> 
> However, when I run the following command to attempt to read each of
> the bad blocks, no I/O errors pop up either on the command line or in
> /var/log messages:
> 
> # for i in $(mdadm --examine-badblocks /dev/sdg4 | grep "512 sectors"
> | cut -c11-20) ; do dd bs=512 if=/dev/sdg4 skip=$i count=512 | wc -c;
> done
> 
> I've truncated the output, but in each case it is similar to this:
> 
> 512+0 records in
> 512+0 records out
> 262144
> 262144 bytes (262 kB) copied, 0.636762 s, 412 kB/s
> 
> Thus, the bad blocks on the failed hard drive are apparently now
> readable on the cloned drive.
> 
> When I try to assemble the RAID 5 array, though, the process gets
> stuck at the location of the first bad block. The assemble command is:
> 
> # mdadm --assemble --force /dev/md4
> --backup-file=/home/gwr/2017/2017-01/md4_backup__2017-01-25 /dev/sde4
> /dev/sdf4 /dev/sdh4 /dev/sdl4 /dev/sdg4 /dev/sdk4 /dev/sdi4 /dev/sdj4
> /dev/sdb4 /dev/sdd4
> mdadm: accepting backup with timestamp 1485366772 for array with
> timestamp 1487624068
> mdadm: /dev/md4 has been started with 9 drives (out of 10).
> 
> The md4_raid5 process immediately spikes to 100% CPU utilization, and
> the reshape stops at 1901225472 KiB (which is exactly half of the
> first bad sector value, 3802454640):
> 
> # cat /proc/mdstat
> Personalities : [raid6] [raid5] [raid4]
> md4 : active raid5 sde4[0] sdb4[12] sdj4[7] sdi4[8] sdk4[11] sdg4[10]
> sdl4[9] sdh4[2] sdf4[1]
>       13454923776 blocks super 1.1 level 5, 512k chunk, algorithm 2
> [10/9] [UUUUUUUUU_]
>       [===================>.]  reshape = 98.9% (1901225472/1922131968)
> finish=2780.9min speed=125K/sec
> 
> unused devices: <none>
> 
> Googling around, I get the impression that resetting the badblocks
> list is (a) not supported by the mdadm command; and (b) considered
> harmful. However, if the blocks aren't really bad any more, as they
> are now readable, does that risk still hold? How can I get this
> reshape to proceed?

Indeed, it is not possible to reset badblocks list. This list is the
indication for a driver which blocks it is not allowed to read - the
operation might succeed but the data can be out-of-date. But it still
attempts to write those blocks. If write operation is successful, the block
is removed from bad block list. I think it is the only way to clear the
list.

I guess your reshape is blocked because it cannot read the data from the
disk to restore the data on the other drive. On the other hand it cannot
clear bad block as it's not possible to execute a write on a degraded array
with bad block.

As long as you're sure the data on the disk is valid, I believe clearing
bad block list manually in metadata (no easy way to do it) would allow
reshape to complete.

Tomek

Re: Reshape stalled at first badblock location (was: RAID 5 --assemble doesn't recognize all overlays as component devices)

[Shaohua Li]

On Mon, Feb 20, 2017 at 05:18:46PM -0500, George Rapp wrote:
> On Sat, Feb 11, 2017 at 7:32 PM, George Rapp <george.rapp@gmail.com> wrote:
> > Previous thread: http://marc.info/?l=linux-raid&m=148564798430138&w=2
> > -- to summarize, while adding two drives to a RAID 5 array, one of the
> > existing RAID 5 component drives failed, causing the reshape progress
> > to stall at 77.5%. I removed the previous thread from this message to
> > conserve space -- before resolving that situation, another problem has
> > arisen.
> >
> > We have cloned and replaced the failed /dev/sdg with "ddrescue --force
> > -r3 -n /dev/sdh /dev/sde c/sdh-sde-recovery.log"; copied in below, or
> > viewable via https://app.box.com/v/sdh-sde-recovery . The failing
> > device was removed from the server, and the RAID component partition
> > on the cloned drive is now /dev/sdg4.
> 
> [previous thread snipped - after stepping through the code under gdb,
> I realized that "mdadm --assemble --force" was needed.]
> 
> # uname -a
> Linux localhost 4.3.4-200.fc22.x86_64 #1 SMP Mon Jan 25 13:37:15 UTC
> 2016 x86_64 x86_64 x86_64 GNU/Linux
> # mdadm --version
> mdadm - v3.3.4 - 3rd August 2015
> 
> As previously mentioned, the device that originally failed was cloned
> to a new drive. This copy included the bad blocks list from the md
> metadata, because I'm showing 23 bad blocks on the clone target drive,
> /dev/sdg4:
> 
> # mdadm --examine-badblocks /dev/sdg4
> Bad-blocks on /dev/sdg4:
>           3802454640 for 512 sectors
>           3802455664 for 512 sectors
>           3802456176 for 512 sectors
>           3802456688 for 512 sectors
>           3802457200 for 512 sectors
>           3802457712 for 512 sectors
>           3802458224 for 512 sectors
>           3802458736 for 512 sectors
>           3802459248 for 512 sectors
>           3802459760 for 512 sectors
>           3802460272 for 512 sectors
>           3802460784 for 512 sectors
>           3802461296 for 512 sectors
>           3802461808 for 512 sectors
>           3802462320 for 512 sectors
>           3802462832 for 512 sectors
>           3802463344 for 512 sectors
>           3802463856 for 512 sectors
>           3802464368 for 512 sectors
>           3802464880 for 512 sectors
>           3802465392 for 512 sectors
>           3802465904 for 512 sectors
>           3802466416 for 512 sectors
> 
> However, when I run the following command to attempt to read each of
> the bad blocks, no I/O errors pop up either on the command line or in
> /var/log messages:
> 
> # for i in $(mdadm --examine-badblocks /dev/sdg4 | grep "512 sectors"
> | cut -c11-20) ; do dd bs=512 if=/dev/sdg4 skip=$i count=512 | wc -c;
> done
> 
> I've truncated the output, but in each case it is similar to this:
> 
> 512+0 records in
> 512+0 records out
> 262144
> 262144 bytes (262 kB) copied, 0.636762 s, 412 kB/s
> 
> Thus, the bad blocks on the failed hard drive are apparently now
> readable on the cloned drive.
> 
> When I try to assemble the RAID 5 array, though, the process gets
> stuck at the location of the first bad block. The assemble command is:
> 
> # mdadm --assemble --force /dev/md4
> --backup-file=/home/gwr/2017/2017-01/md4_backup__2017-01-25 /dev/sde4
> /dev/sdf4 /dev/sdh4 /dev/sdl4 /dev/sdg4 /dev/sdk4 /dev/sdi4 /dev/sdj4
> /dev/sdb4 /dev/sdd4
> mdadm: accepting backup with timestamp 1485366772 for array with
> timestamp 1487624068
> mdadm: /dev/md4 has been started with 9 drives (out of 10).
> 
> The md4_raid5 process immediately spikes to 100% CPU utilization, and
> the reshape stops at 1901225472 KiB (which is exactly half of the
> first bad sector value, 3802454640):
> 
> # cat /proc/mdstat
> Personalities : [raid6] [raid5] [raid4]
> md4 : active raid5 sde4[0] sdb4[12] sdj4[7] sdi4[8] sdk4[11] sdg4[10]
> sdl4[9] sdh4[2] sdf4[1]
>       13454923776 blocks super 1.1 level 5, 512k chunk, algorithm 2
> [10/9] [UUUUUUUUU_]
>       [===================>.]  reshape = 98.9% (1901225472/1922131968)
> finish=2780.9min speed=125K/sec
> 
> unused devices: <none>
> 
> Googling around, I get the impression that resetting the badblocks
> list is (a) not supported by the mdadm command; and (b) considered
> harmful. However, if the blocks aren't really bad any more, as they
> are now readable, does that risk still hold? How can I get this
> reshape to proceed?
> 
> Updated mdadm --examine output is at
> https://app.box.com/v/raid-status-2017-02-20

Add Neil and Jes.

Yes, there were similar reports before. When reshape finds nadblocks, the
reshape will do an infinite loop without any progress. I think there are two
things we need to do:

- Make reshape more robust. Maybe reshape should bail out if badblocks found.
- Add an option in mdadm to force reset badblocks

Thanks,
Shaohua

Re: Reshape stalled at first badblock location (was: RAID 5 --assemble doesn't recognize all overlays as component devices)

[George Rapp]

> On Mon, Feb 20, 2017 at 05:18:46PM -0500, George Rapp wrote:
>> On Sat, Feb 11, 2017 at 7:32 PM, George Rapp <george.rapp@gmail.com> wrote:
>> [...snip...]
>>
>> When I try to assemble the RAID 5 array, though, the process gets
>> stuck at the location of the first bad block. The assemble command is:
>>
>> [...snip...]
>>
>> The md4_raid5 process immediately spikes to 100% CPU utilization, and
>> the reshape stops at 1901225472 KiB (which is exactly half of the
>> first bad sector value, 3802454640):
>>
> [...snip...]
On Tue, Feb 21, 2017 at 4:51 AM, Tomasz Majchrzak
<tomasz.majchrzak@intel.com> wrote:
> As long as you're sure the data on the disk is valid, I believe clearing
> bad block list manually in metadata (no easy way to do it) would allow
> reshape to complete.
>
> Tomek
On Tue, Feb 21, 2017 at 12:58 PM, Shaohua Li <shli@kernel.org> wrote:
>
> Add Neil and Jes.
>
> Yes, there were similar reports before. When reshape finds nadblocks, the
> reshape will do an infinite loop without any progress. I think there are two
> things we need to do:
>
> - Make reshape more robust. Maybe reshape should bail out if badblocks found.
> - Add an option in mdadm to force reset badblocks

OK, I examined the structure of the superblock and the badblocks
array. My first attempt was to zero out the bblog_offset and
bblog_size in the md superblock using dd (but that causes the checksum
to be different than the sb_csum in the superblock, and the mdadm
--assemble fails. I didn't want to research how to recalculate the
checksum unless I really, really have to.  8^)

Running mdadm under gdb, I determined that my bblog_offset was 72
sectors from the start of the md superblock), and filled that space
with 0xff characters in my overlay file:

# dd if=/dev/mapper/sdg4 bs=512 count=1 skip=73 of=ffffffff
# dd if=ffffffff of=/dev/mapper/sdg4 bs=512 count=1 seek=72

That convinced mdadm that I have a badblocks list, but it's empty:

# mdadm --examine-badblocks /dev/mapper/sdg4
Bad-blocks on /dev/mapper/sdg4:
#

Once I did that, and restarted the array with my overlay files:

# mdadm --assemble --force /dev/md4
--backup-file=/home/gwr/2017/2017-01/md4_backup__2017-01-25
/dev/mapper/sde4 /dev/mapper/sdf4 /dev/mapper/sdh4 /dev/mapper/sdl4
/dev/mapper/sdg4 /dev/mapper/sdk4 /dev/mapper/sdi4 /dev/mapper/sdj4
/dev/mapper/sdb4
mdadm: accepting backup with timestamp 1485366772 for array with
timestamp 1487645030
mdadm: /dev/md4 has been started with 9 drives (out of 10).
#

The reshape operation got past the two positions where it had frozen
earlier, and didn't throw any obvious errors to /var/log/messages, so
Tomek's suggestion seems to clear the badblocks seems to have worked.
However, this was in the overlay files, not the actual devices.

Before I proceed for real, does clearing the badblocks log and
assembling the array seem like my best option?

-- 
George Rapp  (Pataskala, OH) Home: george.rapp -- at -- gmail.com
LinkedIn profile: https://www.linkedin.com/in/georgerapp
Phone: +1 740 936 RAPP (740 936 7277)

Re: Reshape stalled at first badblock location (was: RAID 5 --assemble doesn't recognize all overlays as component devices)

[Phil Turmel]

On 02/21/2017 08:12 PM, George Rapp wrote:

> Before I proceed for real, does clearing the badblocks log and 
> assembling the array seem like my best option?

Yes.  And consider disabling badblocks on all of your arrays.

The badblocks feature does nothing but guarantee that errors
encountered on a device become uncorrectable, even if the cause
of the error was a transient communications or power problem, not
a true media flaw.  Bad block tracking at the OS level is
appropriate for ancient MFM and RLL devices that lack modern
firmware, or similar low-level devices.  And since the MD raid
bad block tracking feature does *not* provide redirects to
usable spare sectors, the feature is useless for such devices,
too.

MD raid currently does *nothing* when a badblock entry reduces
the redundancy available for a particular sector in an array.
The badblocks feature is incomplete, has no upside for modern
component devices, and should not be used.

Phil

Re: Reshape stalled at first badblock location (was: RAID 5 --assemble doesn't recognize all overlays as component devices)

[George Rapp]

On Wed, Feb 22, 2017 at 11:17 AM Phil Turmel <philip@turmel.org> wrote:
>
> On 02/21/2017 08:12 PM, George Rapp wrote:
>
> > Before I proceed for real, does clearing the badblocks log and
> > assembling the array seem like my best option?
>
> Yes.  And consider disabling badblocks on all of your arrays.
>
> The badblocks feature does nothing but guarantee that errors
> encountered on a device become uncorrectable, even if the cause
> of the error was a transient communications or power problem, not
> a true media flaw.  Bad block tracking at the OS level is
> appropriate for ancient MFM and RLL devices that lack modern
> firmware, or similar low-level devices.  And since the MD raid
> bad block tracking feature does *not* provide redirects to
> usable spare sectors, the feature is useless for such devices,
> too.
>
> MD raid currently does *nothing* when a badblock entry reduces
> the redundancy available for a particular sector in an array.
> The badblocks feature is incomplete, has no upside for modern
> component devices, and should not be used.

OK, thanks. I removed the badblocks list as previously proposed and
reassembled the RAID 5 array, and the reshape completed on 9 drives.
I'm re-adding the tenth drive for redundancy before fsck'ing and
mounting the filesystem to see what we've done.  8^)

Current RAID status uploaded to
https://app.box.com/v/raid-status-2017-02-22-txt for them as are
interested.

Thanks to all for your input and help!

George

Re: Reshape stalled at first badblock location (was: RAID 5 --assemble doesn't recognize all overlays as component devices)

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 916 bytes --]

On Tue, Feb 21 2017, Shaohua Li wrote:

>
> Add Neil and Jes.
>
> Yes, there were similar reports before. When reshape finds nadblocks, the
> reshape will do an infinite loop without any progress. I think there are two
> things we need to do:
>
> - Make reshape more robust. Maybe reshape should bail out if badblocks found.
> - Add an option in mdadm to force reset badblocks

The second of these is already possible
Commit: 6dd16dac4001 ("Add --update=force-no-bbl.")

It isn't documented though, and only works during "assemble", not on an
active array.
Writing to the "bad" blocks should remove the "bad" status.  It would be
nice if mdadm could locate the bad blocks, map them to array blocks,
trigger a limited "resync" if there are any good copies, or write zeros
if there aren't.

And yes; reshape should be more robust...  if only we had a pool of
developers, eager to work on these problems :-)

NeilBrown

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]